Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

E.openFile() erroneously returns undefined #1768

Closed
DVR-Ltd opened this issue Mar 4, 2020 · 11 comments
Closed

E.openFile() erroneously returns undefined #1768

DVR-Ltd opened this issue Mar 4, 2020 · 11 comments
Labels
ESP32 This is only a problem on ESP32-based devices

Comments

@DVR-Ltd
Copy link
Contributor

DVR-Ltd commented Mar 4, 2020 •
edited

Hi,

I'm experiencing a problem with E.openFile() returning undefined instead of the expected File object.

I am using Espruino v2.04 on an ESP-WROOM-32.

The problem is more common when I am expecting Espruino to do more. For example, when in addition to my application I invoke a file transfer module for getting files onto the ESP32s file system via Serial3, the likelihood of E.openFile() returning undefined significantly increases. The module does not do anything unless data is sent to the ESP32. Merely having the self contained module loaded affects the problem.

Calling process.memory() reduces the number of failures but the problem is not related to heap memory usage. Data to back up this statement available.

The code below will replicate the problem. There is usually a run of success before failures so it may take a minute.

Although having a large number of timeout loops is quite different to my application, it reliably replicates the problem I am experiencing without attaching hardware to serial ports. My application serves a web interface (the files are being opened to respond to HTTP requests) and communicates with a device on Serial2. It has a main loop that must be continually run without blocking the main thread. This is achieved using setTimeout(main, 1); at every exit point of main().


var fileID    = 0,
	success   = 0,
	fail      = 0,


	fs                 = require("fs"),
	NUM_FILES          = 5,
	NUM_EVENT_HANDLERS = 30,

	writeFiles    = () => {
		var files = fs.readdir();

		if (files.indexOf("test" + (NUM_FILES - 1)) === -1) {
			for (fileID = 0; fileID < NUM_FILES; fileID += 1) {
				fs.writeFile("test" + fileID, "content_doesn't_matter");
				console.log("Wrote file 'test" + fileID + "'");
			}
		}
	},

	backgroundTask = () => {
		setTimeout(backgroundTask, 0);
	},
	
	readFile = () => {
		var f = E.openFile("test" + fileID, "r");

		if (f) {
			success += 1;
			f.close();
		}
		else {
			fail += 1;
		}

		console.log("Success: ", success, "  Failure: ", fail);

		fileID += 1;
		if (fileID === NUM_FILES) {
			fileID = 0;
		}
		setTimeout(readFile, 0);
	},

	test = () => {
		//Create files to read back later.
		writeFiles();
		
		for (let a = 0; a < NUM_EVENT_HANDLERS; a += 1) {
			backgroundTask();  //Causes a timeout to call itself endlessly.

			//I suspect that event handlers listening for HTTP requests, serial
			//ports and timeouts all contribute to the same issue.
		}

		//Read files back.
		fileID = 0;
		readFile();
	};

test();

Any ideas how to locate the cause of the problem appreciated.

Regards,

Gary
@gfwilliams
Copy link
Member

I think this is to do with memory issues on ESP32. Basically FAT support on ESP32 uses massive (4k?) sectors to match the flash's page size, which means it has to allocate bags of memory each time you use a file. Sometimes memory gets fragmented and it just can't allocate those blocks.

You could try E.defragment() on the newest builds and see if that helps, but otherwise I think it's probably preferable to use the built-in Storage module for storage, rather than FAT

@gfwilliams gfwilliams added the ESP32 This is only a problem on ESP32-based devices label Mar 5, 2020
@DVR-Ltd
Copy link
Contributor Author

DVR-Ltd commented Mar 5, 2020

That could be the cause if there are separate heaps for Espruino and the code Espruino is running. I compared the results of ESP32.getState().free against the result of E.openFile() and found no correlation between heap usage and occurrences of undefined.

I used Storage until I found this example. The changes resulted in significant improvements to my project. Right now I have an ugly work around to the E.openFile() issue (try again if undefined, eww) and the result is a more reliable web server than I have achieved in the last two years.

My previous web server based on Storage overcame problems that may have been consigned to history. Espruino has become a lot more stable and feature complete so I can probably do a much better job now. What would really be persuasive is if Storage supported streams.

@gfwilliams
Copy link
Member

I compared the results

It'll be to do with memory fragmentation, not actual usage. If you use one of the cutting edge builds your problem might well go away - or at the very least you can use E.dumpFragmentation() to see what's going on.

if Storage supported streams.

There's now Storage.open - it'll be in 2v05 when that's released.

@opichals
Copy link
Contributor

opichals commented Mar 6, 2020 •
edited

@DVR-Ltd read-only Storage stream support is easy to be done. See #1432

@gfwilliams
Copy link
Member

Looking at it, this seems like a duplicate of this issue: #1559

Again it would be super helpful if you could try on a cutting edge firmware.

@DVR-Ltd
Copy link
Contributor Author

DVR-Ltd commented Mar 6, 2020

@gfwilliams

It'll be to do with memory fragmentation, not actual usage.

Good point.

There's now Storage.open - it'll be in 2v05 when that's released.

Well, that'll be an addition to my jobs list then 👍

Again it would be super helpful if you could try on a cutting edge firmware.

I will get back to you.

Thank you for your time on this.

@DVR-Ltd
Copy link
Contributor Author

DVR-Ltd commented Mar 6, 2020

@opichals Thanks. Useful to know.

@DVR-Ltd
Copy link
Contributor Author

DVR-Ltd commented Mar 6, 2020

@gfwilliams

I downloaded http://www.espruino.com/binaries/travis/master/espruino_2v04.384_esp32.tgz and flashed my board. The flash succeeded but I get the below error when the board boots.

After the first failed attempt, I read README_flash.txt which showed me that I should have included some additional parameters to what I have been using up until now (baud, flash_mode and flash_freq). I made the changes to my bash script that I use to flash the Espruino but I got the same result.

configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:2668
load:0x40078000,len:7304
load:0x40080000,len:5312
entry 0x40080274
E (523) esp_image: Image length 1377632 doesn't fit in partition length 1376256
E (523) boot: Factory app partition is not bootable
E (524) esp_image: image at 0x160000 has invalid magic byte
E (529) boot: OTA app partition slot 0 is not bootable
E (534) boot: No bootable app partitions in the partition table
Fatal exception (0): IllegalInstruction
epc1=0x400802ba, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

@jumjum123
Copy link
Contributor

jumjum123 commented Mar 6, 2020 via email

@jumjum123
Copy link
Contributor

jumjum123 commented Mar 6, 2020 via email

@gfwilliams
Copy link
Member

Pretty sure the build is working again now - and up to date builds have a lot of changes to cope with fragmentation so I think this is fixed

@GaryOtt GaryOtt mentioned this issue Mar 12, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ESP32 This is only a problem on ESP32-based devices
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@opichals @gfwilliams @jumjum123 @DVR-Ltd