Allow FS (fatfs) API to work on ESP8266's flash #1116
Comments
|
Hi Gordon, I have been thinking about storage recently and looking at options. The esp32 has a number of options that could be of use... http://esp-idf.readthedocs.io/en/latest/api/storage/index.html
So the idea is to take a js object - json stringy, compress with heat shrink, and then store the blob against the key.
They have an sd card fat example that uses this. This means that the Lunix implemention of the
|
|
I was looking If this was abstracted the same way as the network library, with a function table it would be more extensible and easier to add new layers |
|
Here is the nvs stuff on the esp32 https://gitter.im/espruino/esp32?at=58c66411dd08b4b859c2c560 At this stage just working with i32 |
|
Yes, some kind of function table could work. I guess it depends whether we have to support different filesystem types with the same build? If not we could do like jsHardware - create a common API and then just compile in the implementation that's needed? Swapping to the function table from that could be easy to do later though. Personally, I don't think adding ESP32-specific stuff is a good idea at the moment - 99% of people with this problem right now are using ESP8266, so ideally we want some solution which handles both. It's fine just to set the block size to 4096b in FatFS - which would solve this? http://elm-chan.org/fsw/ff/en/appnote.html It would require 4k of RAM temporarily to access files, but it's not the end of the world. Or there's so much free flash available it wouldn't be the end of the world to just ignore the other 3500 bytes. I kind of like that approach, because we can do it using the existing Espruino flash APIs. The filesystem could then be supported on any device that Espruino runs on if it has spare pages of flash - not just ESP8266/32. For writing, a simple JS webserver that handles posting data should be fine - maybe as a library. I'd posted some example code on the forum a year or so ago. Also at some point I'll be looking at adding a built-in server to allow console over websocket, and I guess at that point we could look at adding a way to upload files with it. |
|
Here is a js post server that emulates the esp8266 ota server that is in firmware. https://github.com/wilberforce/Espruino/blob/ota/targets/esp32/tests/ota-server.js I was thinking something like this could be extended, so not only handle ota updates, but update files in a storage system. |
|
So it looks like these set of functions would need to be set up to use the inbuild flash libs: https://github.com/espruino/Espruino/blob/master/libs/filesystem/fat_sd/diskio.h#L34-L38 |
|
I think so, yes. It seems like they'd all be pretty simple. Looks like you specify the sector size with ioctl as well: http://elm-chan.org/fsw/ff/en/dioctl.html Part of me wonders whether that could be exposed to JS. It'd be very cool to be able to put a FAT filesystem on external SPI flash. |
|
http://elm-chan.org/fsw/ff/en/appnote.html#port Function Required
Use as template for starting point gotchas:
|
|
It doesn't look like much, however have made a start here: It would be great if we can select as using an SD card or flash as some people are already using the SD card on the ESP32. Perhaps with some sort of initialiser like: perhaps `E.flashFS(0x300000); or do you think the address for the flash should be defaulted? The other possible params would be fs type - fat/fat32 or size of the block of flash to use? |
|
Nice - does it work? I didn't realise there was a Probably for the final version I'd add a new It sounds like a good idea to force people to initialise it, yes. To stick with |
|
No - it doesn't work ;-( I have set up a test harness that attempts to make the file system. It's going to be pretty slow as all of the pages get set to zero as part of the format - only to clear them back to FF before the next write.. It is spi_diskio.c I'm changing at the moment - Ideally want to make it switchable so depending on what call you make, you can connect to flash or an SD card. BTW - what command do you use to format the SD card in linux? I've been trying to make an img, and then flash that and then try to read from it.. `` WARNING: Not enough clusters for a 16 bit FAT! The filesystem will be |
|
That command looks ok to me as far as I know (however I guess fat.fs and fat.fs.img should be the same?). I guess another option would be to implement jshFlash* on Linux (just writing to a file) and then you could run and debug much more easily? |
|
Yay - success (of sorts). Created - The files system can be formatted, and files written. however, it is crashing on file read - which is weird since the folder structures etc can be read fine.
|
|
I don't know - maybe print out exactly what addresses and sizes you're reading and see if anything looks odd about the one that crashes. Good sign though! |
|
I've spent quite a bit of time on this today - it's very frustrating. Reading back the flash: It looks like all the bits are getting put in the write places... however still crashes on file reads on the ESP32 -so it could be be some other issue? I guess I could see it compiles on the esp8266 and if the same problem . Downloading the 1M block and trying to read the img file does not work either - so it could be some sort of corruption with what is written. Anyway, it's a starting point if anyone is keen for the challenge. |
|
I just committed some flash emulation functions to the Linux port - so if you're able to compile your code for linux then you should be able to see how it works on the PC (might need some bodging in jswrap_fs to force it to use the SPI FS functions). It'll be much easier to debug and it should even trap segfaults/etc :) |
|
It's interesting how comments trigger thought processes... following what you said about fs_wrap lead to here... I think this read is the issue... Espruino/libs/filesystem/jswrap_fs.c Line 251 in e120ce1 JsVar *buffer = jswrap_file_read(f, 0x7FFFFFFF); Is this meant to be -1 ? |
|
That read itself should be safe since you don't want the user to be able to segfault by calling |
|
Found it. argg,h. It was a buffer overflow and no safety checks in the ff.c to https://github.com/espruino/Espruino/blob/master/libs/filesystem/fat_sd/ffconf.h#L174-L181 According to the comment - get sector size is implemented so we should be able to leave _MAX_SS 4096 - unless you want it in an ifdef? here is some output anyway... |
|
further good news.... read the flash to fs.img: And on windows opened with |
|
I copied a module onto the SDcard: No sure what the path or syntax is meant to be for this? |
|
@MaBecker |
|
I think it looks in a 'node_modules' folder? That's great news though! Which is statically allocated in jswrap_file.c - so by increasing it you're using 3.5kB of RAM all the time. I guess potentially:
|
|
I have used #ifdef for the sector side, so the other builds are not affected. https://github.com/espruino/Espruino/compare/master...wilberforce:flash_fs?expand=1 The code is getting close to ready to merge.
The Fatfs also supports disks.... |
|
Nice, thanks! Looks to me like It might be an idea to just disable it by default or at least make it read-only? as-is, I imagine just trying to use a module that doesn't exist would cause 1MB of flash memory to get formatted? I'd steer clear of multiple disks - that just sounds like pain (i mean, that's what directories are for?). If we did do that, IMO we should focus on allowing normal SD cards and the flash memory to coexist on separate volumes first as that could be really useful. Please could we move jsDebug to just commented out jsWarns for now? seems like overkill to add a new global debug macro just for this? |
|
I made the jsDebug macro global, so it can be used anywhere - not just with this code. I don't have any gdb debugging, so was using this. I can make it local to these files if required. The I just wanted to make it easy to use, by turning on if you used fs or file... |
|
To Do:
Pull request #1119 |
|
@MaBecker - Working on the ESP8266! For this 4Mb build: Add to the BOARD.py: |
|
Does it work for you on the ESP32, btw? |
|
Pipe read from file piping to http res does not work on esp32 however everything in the comment above works:
With the esp8266 you are battling with such little memory, and there might be other things going on. The post above with pipe=0 and if you set the read buffer to 32, it should work for you. |
|
Right -- we both agree on that. And its great work! :) Aside from the "closing" bug we also both agree on. It already makes things more usable even just for basic file I/O, and this is amazing, and honestly I'm very thankful for it. :) However now I'm trying to write using pipe (or even fs.write()) from a browser. So I was curious if you had tried writing (say a large file from a browser) to the ESP32. Anyways its not so important I'll probably find a way around it sometime. I'll continue testing and see if I can find anything more concrete for you to work with. |
|
This should work for you on the ESP8266: if ( pipe === 0 ) { Please try that. |
|
Yes I'm sorry I wasn't clear -- I agree some variation of that will work on the ESP8266. I tried that extensively on the weekend -- the do / while construct will fail with out of memory on very large files like in my original example. Your modified example of mine is not large enough to trigger this. Its only a few K not a few 100K. However, this WILL work generally: readInterval=setInterval(function() {
data = f.read(32);
if (data) {res.write(data);}
if (!data) {
clearInterval(readInterval);
f.close();
res.end(page);
console.log("\/\/\/ ::: READ ENDING ::: \\\\\\");
return;
}
},100);
though occasionally will not detect the end of data and will start outputting again. Since Gordon's pipe() works more or less fine, I didn't pursue it too much further. It would seem that buffers are not being cleared fast enough for the do / while construct. If I reduce the setTimeout below 100ms, or increase the f.read() to 64 or higher and keep the setTimeout at 100ms, it eventually does run out of memory about halfway through my example (the 10000 iteration one). But we agree that pipe works for sending data to the browser :) What I am trying to do is WRITE to the file system, using pipe(). My understanding is that if I want to upload stuff to the ESP8266, this would be the best way if the file is large. Again I wasn't clear -- I believe we are confusing "res.write()" with a "write to the file system". Its the second I would like to do. So again my question would be have you tried writing to the file system on the ESP32 this way, using pipe()? If you can, then its clearly a problem on the ESP8266 side, is more or less what I'm trying to figure out. :) Thanks! |
|
ok. I Found the issue for reading a file wth File.pipe. It is the read: This goes into a loop: If this part is changed So it seems the code that optimisation for a flat string returns something different so the receiving pipe does does recognise this as the end of file. |
|
OK great! I think there has been progress then on both sides. I don't know quite enough about the Espruino code to fully get whats going on there, but I get the gist of it. :) On my side I rethought about the problem and realized that the setTimeout was really awkward, and so after going through the docs thought to redo our snippet ("pipe==0"), with res.on("drain", function () {....});
which really seems like the correct way to do it. And sure enough it works, even detecting the end of data and does not leave the browser hanging. Here is the snippet I ended up with: f = E.openFile("data.csv","r");
res.on("drain", function () {
data = f.read(32);
if (data) {res.write(data);}
if (!data) {
f.close();
res.end(page);
console.log("\/\/\/ ::: READ ENDING (DRAIN) ::: \\\\\\");
return;
}
});
This of course is not the actual pipe() function, but is functionally identical (as far as I can tell), though a bit slower by maybe 20 seconds for my "large file" example. Very minimal overhead in terms of JsVars too, which is great (maybe 100). I would imagine this would easily work on the ESP32 as well. :) Cheers, |
|
Yes so the above snippet seems to work after over 20 requests, without any problems, and drops the JsVars by only about only 90 vars while transferring, and then it returns to baseline. Which on the ESP8266 is pretty good! My results from using pipe() dropped available memory by almost 500. Not sure why pipe() would do that, but if you're working on an extremely limited board like mine, the code above seems more efficient. |
|
Oh and finally I'm happy to announce that the reset after save (see my original post), seems to have totally disappeared since I changed boards. So I'd say its not an issue at this point. This I feel is why its important to always test on several of the same type of board, to see if you can replicate the problem. |
|
@gfwilliams https://github.com/espruino/Espruino/blob/master/src/jswrap_pipe.c#L17-L18
https://github.com/espruino/Espruino/blob/master/libs/filesystem/jswrap_file.c#L462-L472 So when the len is 0, a buffer of "" is returned, so it waits for more data and then re-reads, resulting in a loop. Here is the fix, to make it behave like the non flat string code: |
|
Great, thanks for finding that! Looks like it might have been a regression due to my addition of flat strings? Do you want to issue a PR for it? Sorry for the lack of replies but I'm supposed to be on holiday this week :) |
|
I can add to the existing PR I have in the ESP32 branch - Are you happy for me to apply that? I noticed your absence - though you might have been away! |
|
Yes, that's fine. On the ESP32 PR the |
|
Ah - you mean here: Lines 555 to 569 in d87fe18 It's executing this line which is not necessary: |
|
Ok - tidied that up: But now it is showing as a conflict? I think you might have changed permissions? I know see: |
|
Started to work with this on the ESP8266_4MB board What am I missing, after a reboot the file is gone? |
|
Found the missing part, works as designed, great way to use 1MB. |
|
Adding 'FILESYSTEM' and 'FLASHFS' to ESP8266 is using to much heap space: I think a combination of Flash storage Module, any js type , web pages, css, js and images and module FlashEEPROM would be nice to have for systems with poor resources like the ESP8266. I am pretty sure this works nicely for ESP32. |
|
I had a looks this again recently and the module nesting a sub classing no longer works and I'm not sure why. I fixed the compiler warnings in the fatfs code, and made sure on the esp8266 that the 4 pages of flash are not used. |
|
I started to work on module I name "simpleFlashStore", hope to share soon. Concept: Use some pages as directory and the other to store data. |
|
A JS module? Obviously there's FlashEEPROM already to do very similar stuff? |
|
Yes as JS module and using parts of FlashEEPROM as well plus a 4 byte fix for ESPs So I am not reventing the wheel, but going for cherry picking. This is what I am working on: |
|
So with the 4 pages of flash disabled does that mean the FatFS stuff is effectively disabled on the ESP8266? |
The last 4 pages of flash 4x4096 are used by the esp no-os to store wifi data. If This was used for the flashfs, it would wipe over this area and there would be issues for both wifi and the data! So This was protection to ensure if the dfualts are used the areas did not overlap. FlashFs is not enabled on any of the default builds, as it look about 4.5K of heap, of a device that already had low heap so @MaBecker though it was better to leave this off the new 4Mb build. You can still compile it in yourself as you have been and it will work fine. |
|
@wilberforce OK thanks for clarifying about the flash pages! :) Yes I understand about the heap space, too bad -- but I've been using it successfully in my builds -- sometimes with as little as 2.5K of heap space. I'll continue to use it as I have been then. :) |
|
Each time a network buffer is required on the esp8266 I think it allocates 620 bytes on the heap - so you might need more headroom |
|
@wilberforce Thats funny -- I started noticing on my builds that wifi.save() was not saving. So that makes sense now. I'll update my local copy from the latest on github. I see your point about the network buffer, 620 bytes is alot (I assume per connection). Thanks for the heads up, I'll play maybe with the JsVars to see what I can do. |
|
I think we close now. |
This would make a lot of sense for the 4M parts (#1110)
I think this got shot down a while ago with concerns about repeated writes to flash, but I think realistically the use it would be to everyone would far outweigh any problems.
So either the fatfs library could be tweaked such that the block read/write functionality wrote flash blocks (which I imagine would be easy), or SPIFFS could be ported - but that'd mean re-writing every file IO function to use SPIFFS :(
The text was updated successfully, but these errors were encountered: