BLE update: #1208

sdkconfig reference:

Status of Bluetooth for ESP32
http://forum.espruino.com/conversations/311436/#comment14154035

modified: targets/esp32/docs/Bluetooth/sdkconfig

Modified files:
https://github.com/espruino/Espruino/compare/ESP32

toolchain:

http://esp-idf.readthedocs.io/en/latest/get-started/linux-setup.html

https://dl.espressif.com/dl/xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz

Current state of play:

• Build tools has a ESP32-v3.0 branch that has the compliled libs for v3.0:
  https://github.com/espruino/EspruinoBuildTools/tree/ESP32-v3.0/esp32

@jumjum123 I have used your sdkconfig.

I reduced jsvars from 5000 to 3000 and now it boots.

now getting:

ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0008,len:8
load:0x3fff0010,len:1932
ho 0 tail 12 room 4
load:0x40078000,len:10012
load:0x40080000,len:252
entry 0x40080034
E (51) boot: ota data partition invalid, falling back to factory
E (237) spiram: SPI RAM enabled but initialization failed. Bailing out.
E (237) cpu_start: Failed to init external RAM!
abort() was called at PC 0x40081247 on core 0

Backtrace: 0x4008b110:0x3ffe3c30 0x4008b237:0x3ffe3c50 0x40081247:0x3ffe3c70 0x4   00786c6:0x3ffe3cb0 0x40078862:0x3ffe3ce0 0x40078c18:0x3ffe3d50 0x400800f6:0x3ffe   3e70 0x40007c31:0x3ffe3eb0 0x4000073d:0x3ffe3f20

Rebooting...

E (237) spiram: SPI RAM enabled but initialization failed. Bailing out.

@jumjum123

So it looks like you need to have a different sdk base firmware for with and without SPI RAM ?

Hey, great. You've done a lot of things. My time is a bit limited this weekend. I'm not at home,
Anyway, there is an option in makemenuconfig, around SPIRAM to skip this error.
It is the first (and only) extension based on me :-)
Just comes to my mind, I did not check availability in V3.0 SDK .........

Just checked the SPI RAM Option, it is :
Component config > ESP32-specific > SPI RAM config ->Ignore PSRAM when not found

Actually I worked on implementation of BLE UART. During that, a windows problem came up. Looks like Windows expects bonding and security.
One more time 150 kb have been added, and available heap discreased.
For partition table my idea is to start from the end. 4MB is available we could set:

• 2MB storage, data, etc.
• all other partitions, except factory, the way they are today
• we could increase js_code to make sure, it works for 20000 vars (WROVER board)
• everything left, for factory binary, about 1500 kb. To give an idea about size, latest binary on my side had 1180KB. 1500 should be good, at least for some time....
  Would you agree to partition table like this ?

Ok. I'm still struggling to get a build working - I'm getting a compiled .bin, but it still has the psram config. I'm trying a fresh build but not winning at this stage.
I have turned it off in the menuconfig, however it is still in the firmware. I might get a chance to try again tomorrow.

I think we need some thought around the partition table layout. It would be great not to move the js-code address, but I understand the firmware is larger. We also need to consider more than 4mb flash, such as 8mb and 16mb in the future.

It's disappointing that the 3.0 firmware takes up more ram - has this been explained?

It would be good to allow for ota updates at a later date. The 1mb size for the flashfs file system is arbitrary- that could get smaller too.

I visited the guys from Espressif at a fair in Nürnberg.
The answer to memory consumption was something between "this is the way it is" and "we have to look into it in future versions"
Plans for next boards have been announced in a way "there will be other boards soon". No plans to have more Flash. One point for next chips was a 1 core only version. Some big customers asked for that. There would be no benefit in power or memory or whatever in a 1core. But as mentioned above, customers want it. Therefore I would not spend too much time right now for larger Flash.
I have to build up my development from scratch now with V3.0. As soon as I've got it running, it will be uploaded to ESP32 branch. My latest test all have been with simple boards without PSRAM, I had no problems on that.
Next week this should be done.

Finally have a working build with 4000 vars (no BT at this stage):

|  __|___ ___ ___ _ _|_|___ ___
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 1v97.5 (c) 2018 G.Williams

Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate

>process.env
={
  "VERSION": "1v97.5",
  "GIT_COMMIT": "fcd1a18",
  "BOARD": "ESP32",
  "FLASH": 0, "RAM": 524288,
  "SERIAL": "240ac400-2f90",
  "CONSOLE": "Serial1",
  "MODULES": "Flash,Storage,fs," ... "r,crypto,neopixel",
  "EXPTR": 1073484920 }
>ESP32.getState();
E (35662) esp_ota_ops: ota data invalid, no current app. Assuming factory
={
  "sdkVersion": "v3.0",
  "freeHeap": 115812, "addr": 65536,
  "partitionBoot": "factory"
 }

Next step is to get build tools updated so that this can be replicated in Travis

How I love these minor surprises :-(
Some header files for Bluetooth have been moved around to make our life more interesting .....

Oh, this minor surprise happend 1 day after release.
Now got next problem, SDK option to ignore missing PSRAM is not available in release.

@jumjum123

I have got a build working and pushed the code to the V3.0 branch:

https://github.com/espruino/Espruino/compare/ESP32-v3.0

I want to check that the build is stable. There is an issue we need to resolve regarding the fileno > 32 - I'll need to ask Gordon about how to solve this properly.

@jumjum123

Now got next problem, SDK option to ignore missing PSRAM is not available in release.
Some header files for Bluetooth have been moved around to make our life more interesting .....

Can we stick to just using the 3.0 release for now and not stay with the bleeding edge? Can we look a thing adding psram support later?

I have got Travis build using the new compiler and linking against 3.0. If I find the firmware build file and travis and then test that on an actual device, and post it on the forum for others to test.

If ok, next steps are changing the partition table and jscode area.

I also think there should be a BLE flag in the board.py so that we can do a build without Bluetooth - give people the option.

I agree to use 3.0 release from now on. Anything else seems not to be reliable. There are already some major changes in structure of master, only one day after release.
Its a strange kind of support, to remove an option from release, which was in master for more than 3 month. At the end, there have been not more than 10 lines. I asked in the forum, lets see the answer.

A flag for BLE already exists, its USE_BLUETOOTH. Its used for NRF and ESP32.
In my local sources, this flag is supported.
AFAIK, we would need 2 different precompiled versions, to support with/without BLE.
Wouldn't this add a lot of maintenance ?

Based on your success, I'm unsure what to do now. Should I work on ESP32 branch, and wait until you moved BLE parts from ESP32 to ESP32V3.0 ?
There are some last minute changes in V3.0 release which need to be fixed.
Last not least, I've simple support of UART, some fixes and full handling of USE_BLUETOOTH flag

@jumjum123
Good news about the Bluetooth flag.

The build I made only has 4 sockets and the previous 2.1 build had 10, so I'm trying to sort that out.

At the moment the tgz files for esp-Idf.tgz and app.tgz are a bit of a mess. The libs are all separate folders and there seems to be different levels, so for the build tools it would be much simpler if all of the lib files were in one folder. The Makefile would be greatly simplified, as Paths for the different levels would not have to be added.

I don't want to break your build, so I'll try to keep it compatible and we can sort this properly later.

Need still need to agree on the js_code addressin the flash and the size. I'm coming around to your idea of just under the 3mb mark.

If you could get your build working with the current v3.0, and the Bluetooth flag, and then update the esp32 branch. Then when I get the core stable, I can then see if I can get a build using Bluetooth esp32 branch and the build tools.

Then we can make the decision on one or two board files. The esp8266 has a smaller firmware, and a special 4mb build. It does not create a lot More to maintain as the board file contains the differences and sets the flags, and there would be conditional code based on the use_bluetooth flag.

@wilberforce
Would like to add some good news, but ....
Just jumped into next trap, an iram overflow and some other strange messages.
Looks like the guys did some minor changes in splitting headerfiles, changes to lib files.

Feedback from Espruino to missing option for SPIRAM was (translated to my understanding) "may be this will be implemented into next version"
We would have the option to do these changes by ourself, but this would be very ugly. IMHO we should not do this.
For now we could add some info in docs, how support for PSRAM could be implemented. If somebody needs a lot of memory, this would show how to help themself.

Comin back to Bluetooth, right now, the best plan I see is, to take your ESP-V3.0 brach and add bluetooth step by step. I'm pretty sure, we have to forget support for windows 10 and may be uart.
Windows 10 needs bonding, and this costs a lot of RAM.
I'll keep you on track, with what I'm doing

Sound goods. Let's out psram aside.
I had the Iran overflow - had to reduce jsvars to get a compile.

Windows 10 web Bluetooth is still not working- it is the reason I have 2 pucks and have done nothing with them!

Well keep chipping away and we'll get there eventually!

I got WIndows 10 and Bluetooth running only with SMP (security module).
Problem is, SMP eats some more RAM. For now, I would like to inactivate this part.
May be later on, there will be a way.
For testing I use my cellphone with nRF connect. This tool is very good.

Iram overflow based on too much jsvars ? Hmm this should not happen, since jsvars are created with malloc during runtime on ESP32. At least in my version. Had a short talk with Gordon, how to open the 3rd option for jsvars (malloc). He agreed to add the option for malloc.

Iram overflow based on too much jsvars ?

Sorry - I was confusing with the compile time error - I think you are taking about a runtime error?

@wilberforce
Do you know, what this means ?
If you are using SSL on wifi, search for buffer sizes. They are gorgeous big.
in https://esp32.com/viewtopic.php?f=13&t=3925&sid=f7f93f5afcc9e63294315e20cecce5a2
Could we save some memory for jsvars this way ?

progress stopped - my main conputer won't boot WSL ubuntu after upgrade....
microsoft/WSL#3134
problem OS build 17661.1001

@jumjum123

Do you know, what this means ?
If you are using SSL on wifi, search for buffer sizes. They are gorgeous big.
We are not using this - as we are using the espruino https code. I need to check if linking to the libs actually increases the size - even if we are not using the libs

@wilberforce, you can continue to use WSL by rolling back to any build before 17655, as a temporary workaround until they fix it for good.

@jumjum123

We have been using the debug versions of esp_idf libs, which are much bigger!

In the sdkconfig changing to these options:

CONFIG_OPTIMIZATION_LEVEL_DEBUG=
CONFIG_OPTIMIZATION_LEVEL_RELEASE=y
CONFIG_OPTIMIZATION_ASSERTIONS_ENABLED=
``
Before:

  906640 May  5 18:52 espruino_esp32-original.bin
  
  make clean && BOARD=ESP32 make 
  821952 May  6 10:43 espruino_esp32-release-lib-release.bin
  
  make clean && BOARD=ESP32 RELEASE=1 make
  843168 May  6 09:34 espruino_esp32-not-release-lib-release.bin
  
  So it would be very interesting to see what size the ESP32 branch with the bluetooth support. We might be able to fit in the existing parition table...

The build tools have been updated with these settings and there is a succesful travis build.
  
Here is a build with v3.0 and no debug - it's 803K if someone would like to give it a go...

http://www.espruino.com/binaries/travis/cfc72335906f0fb612597c8b19fd41f9e4233737/espruino_esp32.bin

@wilberforce, very good point !!
In my version, with BLE and!! SMP, having 3000 jsVars, heap is still 57KB, it should be possible to increase to 4000 jsVars. in my previous tests even with 31kb heap, I didn't run into problems. But to be open, testing of this has been Q&U (Quick and Ugly)
Comin back to size of binary, with BLE, GATTS, GATTC, UART and SMP
Before(debug) 1205 kb after change(release) 1145 kb.
There is no remarkable change in Heap :-(

@jumjum123

Before(debug) 1205 kb after change(release) 1145 kb.

This is using the flag in the espruino compile?

Was debug off in the sdkconfig when you complied the esp-idf libs - as this has a much bigger impact on the the code size?

@jumjum123

I'm at the point were I'm trying to do a pull of the ESP32 branch to compile. I'm getting the error regarding the bluettooth gatt include files have moved.
Are you able to update the ESP32 branch so I don't have to repeat what you have already done?

@wilberforce
I've set the release flag in sdkconfig,
Compiler options --> Optimization Level (Release (-Os))
Oh, forgot to set release in Espruino itself :-(

I will upload my sources today. During my changes, I always jump into the same trap.
Means, during testingt some lines have been added, some others removed, etc.
Have to clean up, at least a bit.
There are also some problems around UART, which is working in special cases only :-(

Anyway, at least GAP, GATTC and GATTS should work.

@jumjum123
Ok - looks like you got 1 of the 3 ;-)

There are 3 settings:

CONFIG_OPTIMIZATION_LEVEL_DEBUG=
CONFIG_OPTIMIZATION_LEVEL_RELEASE=y
CONFIG_OPTIMIZATION_ASSERTIONS_ENABLED=

The last one assertions - that generates a whole heap of strings.

Just pushed my sources to ESP32 branch.
Only poorly tested ;-(
Checked sdkconfig, 3rd setting from your list is missing.
I've added a folder to targets/esp32/BLE named Firmware. There you will see the binarys based on actual files in git. README.md still shows PSRAM support, this will be moved to a seperate file soon.

Now have to do some homework(ironwork), but will take a look to this from time to time.
If there is a problem please tell me. Hopefully I can help immediately.

quickly checked the size with ASSERTIONS not enabled and now its 1108KB

Is this good to close? I know Look at socket re-use - might compile now is still open.