Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage: Repeated writes cause errors #1707

Closed
alexsocha opened this issue Nov 27, 2019 · 5 comments
Closed

Storage: Repeated writes cause errors #1707

alexsocha opened this issue Nov 27, 2019 · 5 comments
Labels
high priority

Comments

@alexsocha
Copy link

Version: 2.04
Tested Devices: Pixl.js, Puck.js, RuuviTag

Hello! I've been enjoying using Espruino for data collection, but unfortunately have had some issues with the storage module. These may be related to #1559. Consider the code below.

const Storage = require('Storage');
Storage.eraseAll();

let count = 0;
setInterval(() => {
    console.log('Count: ' + count);
    for (let i = 0; i < 50; i++) {
        Storage.write('file' + i, 'a', 0, 100);
    }
    count += 1;
}, 500);

save();

After less than 10 intervals, the write command will produce Uncaught Error: Unable to find or create file.

The exact number of calls seems to depend on the device. Additionally, changing the file size and/or number of files may cause the error to happen sooner, later, or go away completely.

Once this error occurs, the entire file system becomes unusable. Calling Storage.eraseAll() will bring things back normal, until the error happens again.
@gfwilliams
Copy link
Member

Thanks - worth linking #1598 as well.

I'll try and have a look at this over the coming weeks. The issue is the flash memory is paged, so after you've used up all the pages it needs to be able to find a page with nothing in to be able to erase it, and in your case it's not managing to do that.

You might be able to convince it to work fine by allocating files that are almost as big as the page (4000 bytes?) and then using partial writes to fill them up as needed. The 'cutting edge' builds actually provide an API for logging with Storage.open(...) which may help you out as well.

@alexsocha
Copy link
Author

Thank you! Glad to hear you'll take a look over it. What I'm actually trying to do is alternate between a certain number of files of certain sizes to log data from several sensors, similarly to the example shown here.

For now I may have found a workaround - if I keep all of my files roughly below 100 bytes, and of size 4n+1 (e.g. 65), everything seems to work well.

@gfwilliams
Copy link
Member

This seems to also be an issue on Bangle.js where there are lots of flash pages - it's very easy to have more data that will fit in RAM.

Storage needs to be able to erase individual pages but then to mark them as used so that it doesn't have to search through all of Storage each time it needs to list a file.

@gfwilliams
Copy link
Member

Some code to reproduce here: http://forum.espruino.com/conversations/344602/#15145818

@gfwilliams gfwilliams changed the title Repeated storage writes cause errors Storage: Repeated writes cause errors Apr 16, 2020
@gfwilliams
Copy link
Member

Just a note on this: marking pages as used seems like a bad idea. If instead we just keep a variable marking the end of available flash (only needed on devices with external flash) and do a full scan at boot time to work out how much flash is used, we should be good.

For the 4k page case (all nRF52 devices) we can then just work along a page at a time, reading data, deleting a page, and writing the data back to the first available spot. Data that straddles pages should probably just be left.

gfwilliams added a commit that referenced this issue Jul 14, 2020
@gfwilliams
Implement Streaming Storage compaction to allow compact with less RAM (…
b32078a 
…fix #1598,#1707,#1828)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gfwilliams @alexsocha