Further to above, if the line is changed:
if (e) jsError(e);
to
if (e) jsError("error");

so it complies, then the following error occurs:

root@esp8266-VirtualBox:~/Espruino# ./build.sh
CC src/jslex.o
CC src/jsvar.o
CC src/jsvariterator.o
CC src/jsutils.o
CC src/jsnative.o
CC src/jsparse.o
CC src/jspin.o
CC src/jsinteractive.o
CC src/jsdevices.o
CC src/jstimer.o
CC src/jsspi.o
Generating JS wrappers
WRAPPERSOURCES = src/jswrap_array.c src/jswrap_arraybuffer.c src/jswrap_date.c src/jswrap_error.c src/jswrap_espruino.c src/jswrap_flash.c src/jswrap_functions.c src/jswrap_interactive.c src/jswrap_io.c src/jswrap_json.c src/jswrap_modules.c src/jswrap_pin.c src/jswrap_number.c src/jswrap_object.c src/jswrap_onewire.c src/jswrap_pipe.c src/jswrap_process.c src/jswrap_serial.c src/jswrap_spi_i2c.c src/jswrap_stream.c src/jswrap_string.c src/jswrap_waveform.c libs/math/jswrap_math.c libs/network/jswrap_net.c libs/network/http/jswrap_http.c libs/network/js/jswrap_jsnetwork.c libs/network/esp8266/jswrap_esp8266_network.c targets/esp8266/jswrap_esp8266.c libs/network/telnet/jswrap_telnet.c libs/crypto/jswrap_crypto.c
DEFINES = -DGIT_COMMIT=e42ca309cdd9e55cea592bf28ff08cfbfa5c95fb -DBUILDNUMBER="262" -DUSE_DEBUGGER -DUSE_TAB_COMPLETE -DUSE_HEATSHRINK -DUSE_MATH -DUSE_NET -DUSE_ESP8266 -DUSE_TELNET -DUSE_OPTIMIZE_PRINTF -D__ETS__ -DICACHE_FLASH -DXTENSA -DUSE_US_TIMER -DEMBEDDED -DESP8266
CC libs/crypto/jswrap_crypto.o
CC targets/esp8266/spi.o
CC targets/esp8266/user_main.o
CC targets/esp8266/log.o
CC targets/esp8266/jshardware.o
CC targets/esp8266/i2c_master.o
CC targets/esp8266/esp8266_board_utils.o
CC gen/jswrapper.o
LD espruino_esp8266_partial.o
LD espruino_esp8266_user1.elf
LD espruino_esp8266_user2.elf
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user2.elf] Error 1
make: *** Waiting for unfinished jobs....
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user1.e
```lf] Error 1

My understanding is that the linked code is too large?

So does something need to be cut back to make it fit?

I'm adding crypto so that I have access to the SHA1 function for use in web sockets.

Ok, it looks like in the error case, the issue is that ESP8266 has to store strings in flash unless you access them in a special way - there's a hack for jsError because its strings are almost always constant. Try: if (e) jsError("%s", e);

And yes... the other issue you have is probably also related to that. On ESP8266, unless you add special hacks, all constant data is put in RAM. The best solution for now would be to make sure you add "ifndef":"ESP8266" to the JSON for AES and SHA256/etc functions - so they will not get included and RAM will not get used up for their constant data.

Thanks. The jserror %s hack now compiles.

I tried added the ifndef around the other functions, however still ran out of space.

You could try also looking at libs/crypto/mbedtls/config.h and removing mention of AES/etc from there.

Hi, I encountered the same issue and sent a pr with a fix (#810).

@wilberforce did you find out how to make it fit?

@urish Sorry, no.

I tried what Gordon suggested with "ifndef":"ESP8266" but it was not enough. I think we need to exclude functions as I thing we are running out of rom space.

@tve any inputs on this one?

I'm considering writing a JavaScript module, based on the sha1.js code that @jumjum has published in the forums.

It could be a standalone require("crypto") to use on boards where sha1 is not in firmware.

This way a separate firmware would not be require and it's only about 1k.

In my use case for web sockets, the module is used during negotiation so could be unloaded once the socket is established.

@urish

If you save this in a projects folder in your web ide, the ws module find this module and uses it:

https://raw.githubusercontent.com/wilberforce/EspruinoDocs/master/modules/crypto.min.js

I'll post to the Forum with an example:
http://forum.espruino.com/comments/12817146/

thanks @wilberforce !

In my case, I would like to achieve TLS connections, but I am sure that crypto.js can be useful for other people :)

@urish
TLS will need a lot more functions, so I'm not sure if there will be enough space?

No, I think you're out of luck there. In it's default (standards-compliant) mode, mbedtls needs at least 16kB RAM, which is going to mean very little left for Espruino.

If you want TLS, get a Pico and attach an ESP8266 to it. The Pico has enough RAM to handle it.

@wilberforce @gfwilliams thanks for the feedback! I know that the Espressif SDK does have TLS support (I use it with NodeMCU). Do you know whether there was any work done on using their TLS support with Espruino?

No, there wasn't any work done.

IMO it's worth seeing if mbedtls can be tweaked to fit, because then the same code will be used for all Espruino platforms.

If not we're going to end up with a bunch of code that's just left to rot, and that won't get the bugfixes and improvements that the main mbedtls-based code gets.

Just had a quick look at this, and it seems that mbedtls will compile just fine, but as-is there's not enough memory.

You can modify libs/crypto/mbedtls/config.h to remove features and can then remove bits of libs/crypto/jswrap_crypto.c that used the other features, and I think it'll then compile.

@gfwilliams
You have mentioned the hashlib module will be depreciated in favor of the mdedtls, is there any easy way to remove this from being complied - as that would free up some memory.

I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram?
#697

I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram?
Try make topstrings (read the relevant makefile section first)

@tve thanks.

I tired to build using the top strings, but using a recent git pull with the standard Makefile, I can no longer build...

Sorry can't paste here, the error is here:

https://gitter.im/espruino/Espruino?at=56de471268ddef776469772f

It appears that the user1 elf is bigger than 480k and won't fit in the iram1_0_seg _section ?

Seems like it is possible to fit mbedtls into the ESP8266, as it has been done in esp-open-rtos

@tve
I have, ./build-topstrings.sh:

root@esp8266-VirtualBox:/home/esp8266/Espruino# cat build-topstrings.sh 
#! /bin/bash
export DISABLE_LTO=1 
export ESP8266_BOARD=1
export FLASH_4MB=1
export PATH=/home/esp8266/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
export COMPORT=/dev/ttyUSB0
make $* 

make clean
./build-topstrings.sh
./build-topstrings.sh topstrings

root@esp8266-VirtualBox:/home/esp8266/Espruino# ./build-topstrings.sh topstrings

Top 20 from ./topstrings:
To get details: xtensa-lx106-elf-objdump -j .rodata.str1.4 -s src/FILENAME.o
root@esp8266-VirtualBox:/home/esp8266/Espruino# ls -l topstrings 
-rw-r--r-- 1 root root 0 Mar 13 15:23 topstrings

The file size is 0 - what am I doing wrong?

@wilberforce use xtensa-lx106-elf-objdump -h filename.o to check the used named

in my case it is .rodata.str1.1 and not .rodata.str1.4, so change the Makefile

xtensa-lx106-elf-objdump -h ./src/jswrap_pin.o

./src/jswrap_pin.o:     file format elf32-xtensa-le

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .literal      00000000  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .text         000003ce  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .data         00000000  00000000  00000000  00000402  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  3 .bss          00000000  00000000  00000000  00000402  2**0
                  ALLOC
  4 .rodata.str1.1 00000028  00000000  00000000  00000402  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .irom.literal 0000004d  00000000  00000000  0000042c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  6 .comment      00000022  00000000  00000000  00000479  2**0
                  CONTENTS, READONLY
  7 .xtensa.info  00000038  00000000  00000000  0000049b  2**0
                  CONTENTS, READONLY
  8 .xt.lit       00000048  00000000  00000000  000004d3  2**0
                  CONTENTS, RELOC, READONLY
  9 .xt.prop      000001c8  00000000  00000000  0000051b  2**0
                  CONTENTS, RELOC, READONLY

Great! Thanks. here is the output:

Top 20 from ./topstrings:
7250 ./espruino_esp8266_partial.o
765 ./src/jsinteractive.o
542 ./libs/network/socketserver.o
512 ./gen/jswrapper.o
438 ./libs/network/esp8266/jswrap_esp8266_network.o
387 ./libs/network/esp8266/ota.o
295 ./src/jslex.o
245 ./src/jsvar.o
225 ./src/jsparse.o
216 ./libs/network/telnet/jswrap_telnet.o
214 ./src/jswrap_process.o
207 ./targets/esp8266/esp8266_board_utils.o
207 ./src/jswrap_json.o
196 ./src/jswrap_io.o
196 ./libs/network/socketerrors.o
176 ./libs/network/esp8266/network_esp8266.o
151 ./targets/esp8266/user_main.o
146 ./src/jswrap_object.o
137 ./src/jswrap_espruino.o
132 ./src/jswrap_pipe.o

In jsinteractive - there is the load banner, and quite a few calls to jsiConsolePrint which I don't think is using the FLASH_STR #define, so quite a few of these strings could go into flash memory

@wilberforce @gfwilliams gfwilliams I have spent some time on it and after reducing the number of js variables from 1400 to 1000 (inside boards/ESP8266_BOARD.py), it seems to happily compile.

However, make complains about:

** user1.bin uses 529828 bytes of 491520 available

Two questions:

1. Do you see any reason why this would not be usable with 1000 js vars instead of 1400?
2. Since I have a 4MByte board, does it make sense that I would be able to use this firmware somehow despite its size? Any pointers about this one?

@urish
When did you last do a pull, as a recent update of the mathlib reduces the size of the limage:

http://forum.espruino.com/comments/12848811/

This might be small enough.

However, I think you will run out of available RAM as the TLS stuff uses quite large buffers - bigger than the ESP8266 will have available (16K or so from memory). You might be able to compile with a smaller buffer size, perhaps 1K - and see if your https:// server will still connect to this.

One way of testing this might be to use the linux install, and try the smaller packet size with this first.

Is it https:// requests you want to do? You could use a proxy service, or use a pico and esp8266-01, as you won't have the memory limitation then

Yeah, you may run out of memory fot HTTPS (at least without tweaking the config.h to use non-standard (lower than normal) buffer sizes) - however I got the impression you might have just wanted Crypto for the SHA1 and other features - in which case I'd imagine you're fine.

Strange about the high usage from jsinteractive.c - if jsiConsolePrintf could be tweaked with a define the same way the errors have, you might be able to save quite bit of RAM.

Just to add, are you building with RELEASE=1? That will reduce firmware size a whole load.

As I understand it, you could use more flash, but it would require some linker tweaks and effectively stops you being able to do firmware updates over the air.

@gfwilliams building with RELEASE=1 indeed. Any pointers for for linker tweaks?
@wilberforce building with -lmirom, is this the new math lib?

For those who need crypto - I managed to compile a firmware just with crypto (without SSL), and the image size seems to fit (haven't tested it yet on the real device, though)

In my case - I want SSL indeed (for HTTPS and secure MQTT), and my use case is presenting the ESP8266 as an IoT capable device on tech talks I am giving. Currently, I use NodeMCU, but most of my audience is not familiar with Lua, so being able to use Espruino can be a game changer in my case.

I'm not sure exactly. @tve might have some ideas, or I think it's just a matter of googling. You need to spread the firmware over user1 and user2 areas.

Having said that, you could add the SAVE_ON_FLASH define in the Makefile, and remove the debugger and command completion defines if they were in there. It might save you a bit of space.

Also you can edit libs/crypto/libmbedtls/config.h (iirc) and remove any types of crypto you think you can do without.

good work :)

>[tls.connect({host:'www.google.com',port:443}),process.memory(),esp.getState().freeHeap]
Connecting with TLS...
Performing the SSL/TLS handshake...
=[
  { "type": 4,
    "opt": {
      "host": "www.google.com",
      "port": 443 },
    "sckt": 15 },
  { "free": 133, "usage": 667, "total": 800, "history": 59 },
  2240 ]
ERROR: Failed! mbedtls_ssl_handshake returned -0x2

Now need to guess which one is missing - heap or js vars

I removed the IF statement in

if (jsvIsObject(options)) {
    if (!ssl_load_cacert(sd, options) ||
        !ssl_load_owncert(sd, options) ||
        !ssl_load_key(sd, options)) {
      ssl_freeSocketData(sckt);
      return false;
    }
  }

Looks like at least one is required in the or.. I think the forum post mentions these options...

tls.connect(options, callback)
Description
Create a socket connection using TLS
Options can have ca, key and cert fields, which should be the decoded content of the certificate.

To use https:// you should not need to use the cert options... did you try:

require("http").get("https://www.google.­com", function(res) {
res.on('data', function(data) { console.log(data); });
});

Now need to guess which one is missing - heap or js vars

Looks like heap:

https://github.com/espruino/Espruino/search?utf8=%E2%9C%93&q=SOCKET_ERR_MEM

507 int err = code;
508 switch (code) {
509 case ESPCONN_MEM: err = SOCKET_ERR_MEM; break;
510 case ESPCONN_ABRT: err = SOCKET_ERR_RESET; break;
…
1001 releaseSocket(pSocketData);
1002 return SOCKET_ERR_MEM;
1003 }

Yup, seems like it is heap - less js vars, and now we are getting -0x7200 nearly all the time.

Well, after playing with esp-open-rtos for a while, it seems like MBEDTLS_SSL_MAX_CONTENT_LEN has to be somewhere around 3900 bytes (or larger), otherwise the handshake fails.

@gfwilliams is there a way to allocate a big buffer off the js-vars space?

@urish all the memory needed for TLS comes straight out of the JsVars space already - so you need to increase the amount of variables as much as possible.

If you want to change that, you'll have to modify config.h again to replace calls to Espruino's jsvar malloc implementation with malloc and free.

@gfwilliams thanks for the response! I will look into it further, some more questions that come into my head:

1. How much bytes does a single JsVars contribute to the JsVars space size?
2. I see that two of the mbedTls modules have strings that consume great amounts of RAM (nearly 4k), which can make a huge difference on the ESP:

root@dd700b1d591e:/c/Users/Uri/Documents/esp-ssl# cat topstrings
10989 ./espruino_esp8266_partial.o
2493 ./libs/crypto/mbedtls/library/oid.o
1717 ./libs/crypto/mbedtls/library/x509_crt.o

Where can I find information how to move those strings into the Flash so they won't consume any RAM?

Thanks!

1. It's worth looking at http://www.espruino.com/Internals - 12 or 16 bytes, depending on if you have >1023 vars or not.
2. This has quite a lot of info where it's been attempted with Graphics: More built-in code for ESP8266 #770

But actually without much success, so I'm not 100% sure what's going on there. There should be a bunch of stuff on the net about moving ESP8266 stuff from RAM into flash though

1. Here is the reference to work that was done to move some constants to flash
   Constant strings (and other data) moved to flash memory #697

@tve already did a whole bunch here: 294f3ff

It involves #define to wrap the strings.

Yikes . The oid.c is full of strings! Are you sure you need it for tls v 1.2? Might be worth looking into the switches you need for a minimum build, I reference a minimum config.h above that might be worth a shot..

Actually you should note that some of this stuff works by first copying the string from flash into RAM - but you don't need to do that (and probably shouldn't in the case of the crypto). It'd be better to read directly from flash

following on what @gfwilliams is saying, here are the macro's that pack into iRom, and then read from it:
#770 (comment)

Progress report - oid.c is indeed needed for TLS, and I believe that so is x509_crt.c.

So far, the only module that I was able to disable without impacting the ability to establish TLS connection is MBEDTLS_SSL_SRV_C.

I tried to move the big tables into the ROM by adding __attribute__((section(".irom.literal"))):

static const oid_x520_attr_t oid_x520_attr_type[] __attribute__((section(".irom.literal")) =
{
    {
        { ADD_LEN( MBEDTLS_OID_AT_CN ),          "id-at-commonName",               "Common Name" },
        "CN",
    },
...
}

But this seems to have no affect on the size in topstrings (whereas commenting the entire struct content definitely reduces the size of the module in topstrings). Am I doing it right?

check object file with

xtensa-lx106-elf-objdump -h path/name_of_object_file.o

for a line with .irom.literal to make sure that it will be stored in flash

here is the definition of the macro:

and here is example use in the simple case:

FLASH_STR(expect_cb, "Expecting callback function but got %v");
#define EXPECT_CB_EXCEPTION(jsCB) jsExceptionHere_flash(JSET_ERROR, expect_cb, jsCB)

I think in this case you are declaring arrays of strings, so this would need to be modified....
I don't see why your addition (with the aligned(4) would not work now for proof of concept...

Also here is using an array of strings, but there has got to be a better way!

Looks like this is done a cunning way with arduino build:
http://www.esp8266.com/viewtopic.php?f=9&t=5446

lfaustini wrote:how can I push the WString.cpp.o into the .irom0.text section?
I believe that ESP8266-Arduino IDE uses a special linker script that puts many functions that are marked for the .text section into .irom0.text. The excerpt below comes from esp8266/Arduino on GitHub:
CODE: SELECT ALL
.irom0.text : ALIGN(4)
{
irom0_text_start = ABSOLUTE(.);
*core_esp8266_.o(.literal_, .text_)
*spiffs_.o(.literal_, .text_)
.cpp.o(.literal, .text_)
libm.a:(.literal .text .literal._ .text.)
*libsmartconfig.a:(.literal .text .literal. .text.)
*(.irom0.literal .irom.literal .irom.text.literal .irom0.text .irom.text)
_irom0_text_end = ABSOLUTE(.);
_flash_code_end = ABSOLUTE(.);
} >irom0_0_seg :irom0_0_phdr

So this ld file moves stuff into flash, without having to use macros.

I think it works on functions too.... Looks promising..

https://github.com/esp8266/Arduino/blob/55e5bdfc6cb28fa79f94953f292f66b38804fc67/tools/sdk/ld/eagle.app.v6.common.ld

You could try it. I wonder if that's program code only though - He's talking about WString.o - not WStrings in general :)

The issue is that accesses to flash need to word-only and word-aligned. Just dumping a string into ROM will be fine until you go to access a character from it. Hence the macros.

Until the compiler is smart enough to auto-generate only word accesses for stuff in ROM, you will have no choice but to manually modify code.

@tve @gfwilliams

This technique installs an exception handler that catches the mis-aligned reads, and corrects them.

https://github.com/cesanta/smart.js/blob/master/smartjs/platforms/esp8266/user/esp_flash_bytes.c

It looks a lot simpler that having to copy all the ROM strings from iRom to ram before use...

What is the performance impact of this, though?

I have no idea - convenience vs performance?

well, the performance already isn't all that great. At some point it just sucks. You're gonna have to use some macro to put the strings into flash anyhow, is there that much to be gained by not having a macro that does the copy? How many bytes of strings are left? How many of those could possibly be moved to flash?

Top 20 from ./topstrings:
7250 ./espruino_esp8266_partial.o
765 ./src/jsinteractive.o
542 ./libs/network/socketserver.o
512 ./gen/jswrapper.o
438 ./libs/network/esp8266/jswrap_esp8266_network.o
387 ./libs/network/esp8266/ota.o
295 ./src/jslex.o
245 ./src/jsvar.o
225 ./src/jsparse.o
216 ./libs/network/telnet/jswrap_telnet.o
214 ./src/jswrap_process.o
207 ./targets/esp8266/esp8266_board_utils.o
207 ./src/jswrap_json.o
196 ./src/jswrap_io.o
196 ./libs/network/socketerrors.o
176 ./libs/network/esp8266/network_esp8266.o
151 ./targets/esp8266/user_main.o
146 ./src/jswrap_object.o
137 ./src/jswrap_espruino.o
132 ./src/jswrap_pipe.o

If the constants in jswrapper were aligned on 4 byte boundaries we could keep the reference tables in flash... That would save a heap of space".

The non- string tables might be any easy target, as they are generated - these are not showing in the top strings that you have kindly written.

we will probably also be able to get mbedTLS working on the ESP :-)

Assuming we could catch unaligned accesses, we could just copy all the const data to flash and remove the irom defines completely. It sounds very tempting.

the performance already isn't all that great

I guess it depends how you look at it. It's not great, so would people notice if it got a bit worse? :) The main complaint at the moment actually seems to be lack of RAM, which this might help with.

If someone has some free time/inclination, maybe they could try this just dump everything into ROM, removing all the ESP8266-specific flash copying/etc, and using that exception handler. The only way to know for sure is to do some benchmarks and see how much it hits performance.

If it's loads then I think it'd be a bad idea, but if it's a few percent I think it's a hit worth taking.

From my point of view, it'd be nice to remove the slightly iffy preprocessor stuff around print statements, and I think if you're printing/making error messages then you don't really care about performance either.

The one thing I think could hit performance is the symbol table, but then a few very simple ESP8266-specific tweaks in jswrapper.c could fix that.

Perhaps some of the mbedTLS stuff could end up being quite slow, but then if it's a choice of having it slow or not fitting it in at all, I think it's worth having it in.

@gfwilliams has merged the build with sha crypto for the esp8266 - so now the standard build will support web sockets:

#824

In aes.c there is a save-to-rom macro, however in the esp8266 case the tables still end up in ram.

There is still not enough heap space for the buffers, so unless more code is moved from ram to irom, TLS support for https can't work without reducing jsvars.

My initial aim of supporting web sockets via SHA1 is done, so closing issue.