I've noticed and thought about this problem as well. I have a couple of other things to add:

• Many ESP modules and boards are advertised with their size in megabits, not megabytes (it sounds larger and I think this appeals to some sellers). So some users will be starting from a megabit value not a megabyte value, even though megabits is a much less familiar term.
• It should be possible to detect flash size in most cases. One possible method is via the flash_id command and a lookup table. @foogod has mentioned another method but I've @-mentioned him because I'm not sure exactly how that works, Alex can you give us any pointers?

One possible solution would be to change the default --flash-size argument to "autodetect", and have it print a warning message if detection is not possible. The message could suggest a more destructive detection method (ie esptool.py detect_flashsize) - and that detection command could in turn suggest the user submits new flash size mappings back to esptool.py for inclusion.

+1 for detecting flash size. you don't even need a lookup table as such, JEDEC ID contains the power of 2 as the 3rd byte, it can be used directly - perhaps with a sanity check.
the following code has been working pretty well for us in F&C:

      qInfo() << "Detecting flash size...";
      auto flashChipIDRes = flasher_client.getFlashChipID();
      if (flashChipIDRes.ok()) {
        quint32 mfg = (flashChipIDRes.ValueOrDie() & 0xff000000) >> 24;
        quint32 type = (flashChipIDRes.ValueOrDie() & 0x00ff0000) >> 16;
        quint32 capacity = (flashChipIDRes.ValueOrDie() & 0x0000ff00) >> 8;
        qInfo() << "Flash chip ID:" << hex << showbase << mfg << type
                << capacity;
        if (mfg != 0 && capacity >= 0x13 && capacity < 0x20) {
          // Capacity is the power of two.
          flashSize_ = 1 << capacity;
        }
      }

To be clear, interpreting the JEDEC ID that way is actually an ugly hack and subject to break without warning at any time.

The fact that there is some size information extractable from the third byte of the JEDEC ID is entirely a quirk of how the particular manufacturer has decided to identify particular flash devices and is not a standard in any way. It's entirely possible for Espressif (or anybody else) to start producing ESP8266 modules with chips from a different manufacturer (or even a different model/line from the same manufacturer) which use those bits for something completely different.

This is actually why back in 1999 a bunch of the JEDEC members got together and actually created a device-independent standard for this stuff so that people like us wouldn't have to do this sort of guessing in the first place. It's called the Common Flash Interface (http://www.jedec.org/standards-documents/docs/jesd-6801).

JESD-6801 defines the "Query" command, which returns all kinds of information about (among other things) the size and geometry of the flash device being used (including things like erase sector size, too). I still don't understand why Espressif didn't bother to use this command to just ask the chip what size it is, instead of doing this clumsy hack with encoding it into the flashed image header so it can easily be accidentally mismatched, etc.

Of course, since Espressif only implemented a function for the much less useful "Get ID" command and didn't bother to create one for "Query", accessing this information would require coding a custom low-level routine to send the query command via the SPI interface and interpret the results. (My plan has been for a while to write just such a routine so that esp-open-rtos could automatically detect the flash size correctly and just ignore the value in the header entirely, but I haven't had time.) It would potentially be possible to have esptool.py send that custom code over to the ESP8266 and run it to do that operation, though..

do you have counter-examples of chips that do not follow this format?
the datasheets i checked all document the format and go to some lengths to maintain this compatibility - like implementing the command with a particular opcode. i have certainly observed it to work across manufacturers. while not used as widely as esptool, F&C has a user base too and we haven't received any complaints. in my book, it's better than just assuming 512K. my code checks if the value is in the "sane" range and uses it if it is. i think it's a reasonable starting point. for chips/vendors that do not follow this convention, a lookup table/blacklist can be maintained. do you have entries to contribute to it?

Sigh.. nevermind.. when digging around more to figure out how to implement this stuff, I stumbled across a tiny footnote on a manufacturer's application note (since found a couple of other places), that says basicallly "BTW, CFI isn't supported for SPI memory, and there probably won't be any standard for it in the future either".. grumble.

So I guess we're stuck with using the JEDEC ID and hoping everybody "plays nice" by always encoding the capacity in the same way in the ID field..

Getting back to the original issue, it occurs to me that maybe a good solution for esptool would be (prior to flashing/erasing/etc) to read the existing value of the size (and speed, DIO/QIO, etc) fields, and if set to reasonable values, make sure they stay the same when writing the new image, ignoring the values in the image file (if set to 0xff or some other unreasonable value, try to determine a correct value from the chip ID or something). That way, once set for a particular ESP8266 module, it won't change (which since the hardware won't be changing, seems appropriate).

There should, of course, be some separate command to explicitly change the value if desired, but under normal circumstances, it should get set once to reflect the actual hardware, and then not touched again..

"BTW, CFI isn't supported for SPI memory, and there probably won't be any standard for it in the future either"

There's a standard (because world needs more). Some expensive vendors (e.g. Winbond) even implement it. The rest of the industry doesn't care as usual... https://www.jedec.org/news/pressreleases/jedec-publishes-new-standard-serial-nor-flash . Original standard was publicly available, but now behind login wall. Still googlable, or here's random appnote: http://www.macronix.com/Lists/ApplicationNote/Attachments/1105/AN114v1-SFDP%20Introduction.pdf .

Anyway, my conclusion about JEDEC ID is the same as @foogod's - there's no standard. Actually, if you read vendor datasheets without trying to find meaning implied by yourself, you may imagine how it actually was. Quote from Winbond datasheet: "The JEDEC assigned Manufacturer ID byte for Winbond (EFh) and two Device ID bytes, Memory Type (ID15-ID8) and Capacity (ID7-ID0)." So, long-long ago, before internets were invented, JEDEC called up vendors and told: you do mess, time to standard up. 3 bytes. First, we control, you can't abuse it. 2nd, we recommend being 'memory type', 3rd - 'capacity'. Did you hear us telling that you can't abuse these last 2 bytes?"

Random example of a chip not conforming to common conventions: https://www.sos.sk/productdata/10/21/03/102103/AT25FS010.pdf .

That said, at this age, we can do as bad as Google does: https://chromium.googlesource.com/chromiumos/platform/ec/+/master/include/spi_flash.h

to read the existing value of the size (and speed, DIO/QIO, etc) fields, and if set to reasonable values, make sure they stay the same

I'd rather have auto-detect, at least of flash size.

That way, once set for a particular ESP8266 module, it won't change (which since the hardware won't be changing, seems appropriate).

This assumes that it was set right in the first place, which is rather optimistic.

Ah, thanks @pfalcon.. Yes, the SFDP stuff does appear to be the equivalent in the SPI flash world. For auto-detection, we should probably use that if it's supported, and if not maybe fall back to trying to interpret the ID bytes (though I wouldn't be surprised if the vendors who don't implement SFDP are also the most likely to not follow Winbond's convention for ID bytes as well, so that may not be that useful)..

I'd rather have auto-detect, at least of flash size.

Well, I could see both being useful.. The "keep it like it was" approach allows somebody to set a particular module the way they want it, and then future flashing will use the same settings (so, for example, the SDK parameter tables in the upper sectors will still be in the same place after flashing, even if the image has a wrong "flash size" setting). The risk with this, of course, is that it could get set differently than the actual hardware, and then esptool.py wouldn't fix it (unless told to).. Maybe it should be a command-line option when flashing which approach to use..

This assumes that it was set right in the first place, which is rather optimistic.

Well, what I was figuring was that any time you do a full-erase or it finds something obviously not valid, it would reset it based on auto-detected values, so all you'd need to do is do an initial erase with esptool.py and you should have the correct value going forward..

@foogod : Sure, I agree, the question is how to do it in the simplest yet useful way, as there're to many things to work on anyway... I'll look into adding auto-detection support when I have a chance.

Cant we just have a "probe_flash" feature that reads a block (and keeps it in RAM) - tries to erase that block, if it erases ok it then writes the block back. Do this for each boundary and you now know the size of the flash chip fitted ?

ESP31/32 specifies its flash sizes across the range 1MB - 16MB, so the megabit/megabyte thing is going to get especially confusing soon!

For the experimental ESP31 support in #111 I've changed --flash_size to be specified in megabytes as KB/MB. Specifying in megabits still works but prints a deprecation warning, may be removed in a future v2.0.

Auto-detection of flash size code would still be a welcome addition, though.

Maybe you could do both things -- leave the old values on the chip, but also do an autodetection, and issue a warning (suggesting to run a command fixing it) if the current values don't match the hardware?

Resolved since #131 (now in the current stable release.)