Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test IDE support for socket submission of JS code #605

Closed
nkolban opened this issue Oct 1, 2015 · 25 comments
Closed

Test IDE support for socket submission of JS code #605

nkolban opened this issue Oct 1, 2015 · 25 comments
Labels
ESP8266 This is only a problem on ESP8266 devices

Comments

@nkolban
Copy link
Contributor

nkolban commented Oct 1, 2015

Since the ESP8266 is inherently a WiFi network device, this means that it takes little for it to listen on TCP/IP sockets. Since it is already running the Espruino engine, we have control over the apps it runs. This issue suggests that the ESP8266 be able to listen on a well known TCP port and receive JavaScript code from the Chrome app IDE over a socket connection as an alternative to a Serial connection.

@gfwilliams has posted some clues here that seem to show that this may already have been attempted for some other project in the past. See:

http://forum.espruino.com/conversations/273420/#12497224

This issue will track investigative work for this possibility.
@nkolban nkolban added the ESP8266 This is only a problem on ESP8266 devices label Oct 1, 2015
@samehhady
Copy link

Actually I've already tried to establish web socket connection with my server and I was able to pass the handshake and receive messages from the server but i was stuck with sending the message from ESP as I need to understand the message frame technique first.

Also it is not 100% reliable, sometimes i get the response from the server and sometimes not.

If we fixed the issue with the response and figured out how to send framed message in that case we can send the JS code in messages and just eval() them on the ESP

@gfwilliams
Copy link
Member

I'm not quite sure what you mean about framed messages? Is it not enough to just have a Serial-like device (like USB, Serial1, etc) that is a TCP/IP server? Then all the existing console code (which now even allows debug) would work.

I think I said in that post, but please use the network.h abstraction for this so it's not an ESP8266-specific bit of code. It's something that could be of great use to everyone.

@samehhady
Copy link

actually what i was doing is using web sockets to connect to espruino without the need to serial or usb etc, the server willl open a socket connection and will upload the sketch instantly. It is very usable if you want to update the code of all your external nodes in one go in their place without the need to connect them to the pc or serial

@samehhady
Copy link

Sure my solution will be global and wont be restricted to one chip

@gfwilliams
Copy link
Member

Do you actually mean websockets, and not just normal sockets?

If you're doing a websocket implementation for Espruino, you'd make a lot of people very happy. I tried to arrange socketserver.c such that it wouldn't be too hard to implement.

@gfwilliams
Copy link
Member

So assuming you wanted to get the Espruino 'console' on a TCP/IP connection, you'd:

  • Add EV_TELNET to IOEventFlags
  • Call jshPushIOCharEvents(EV_TELNET, ...) whenever you got data from the socket
  • Whenever you got a chance, call jshGetCharToTransmit(EV_TELNET) and if >=0, add it to the data to send.
  • Make sure there's a Telnet serial device in jswrap_serial.c and jshGetDeviceString and jshFromDeviceString

And you're sorted. Telnet.setConsole() will then put the whole Espruino console interface up for you where you can access it from the net. That'd reset, upload code, even allow debugging - and the Web IDE would 'just work' with it.

@gfwilliams
Copy link
Member

Just to add - the open issue for websockets is: #258

There's a bit of info on it there. Were you specifically looking at Websocket clients rather than servers?

@nkolban nkolban changed the title Test Web IDE support for socket submission of JS code Test IDE support for socket submission of JS code Oct 1, 2015
@nkolban
Copy link
Contributor Author

nkolban commented Oct 1, 2015

I think this GitHub issue is getting a little confused ...

@samehhady This issue is about telnet into Espruino ... and not about web sockets. I can see how you thought that and apologize for the confusion.

@samehhady
Copy link

My apologies for this confusion, yes I was talking about a web socket client and not telnet.
I've seen this post you mentioned @gfwilliams thx for the info you provided.

I will need to study the code and hopefully I would come up with something soon :)

@gfwilliams
Copy link
Member

Just to confuse matters here, if WebSockets were in socketserver.c then Espruino could serve up a Webpage containing <html><script src="github.io/totally_online_web_ide"></script></html>, and that could then emulate the console via websockets. You'd get a neat no-install Web IDE.

IMO that's not a priority though - even a normal Telnet server would allow connection via the 'normal' Web IDE in the same way,

@nkolban
Copy link
Contributor Author

nkolban commented Oct 2, 2015

It looks like @gfwilliams has made some very pertinent changes ... see this forum post:

http://forum.espruino.com/conversations/275277/

@gfwilliams
Copy link
Member

Yes - the Pico only really needs to serve up this file (with full URLs), and someone needs to do a websocket (or AJAX!) version of this and you're sorted.

@nkolban
Copy link
Contributor Author

nkolban commented Oct 3, 2015

I installed @gfwilliams build of the WebIDE directly in Chrome and can now connect the Web IDE as a Telnet client to a test Telnet server. Worked 1st time. Now I am starting to look at the design of Espruino on the ESP8266 as a Telnet listener.

Starting to think about design ...

We need the ESP8266 to start listening on a socket for incoming connections.

  • Should the port number be hard coded or configurable?
  • Should we start listening when ever we get an IP address or when requested by the user through a startup command?
  • Do we need to care about security? Assuming no security, anyone who knows our IP address and port could program our Epruino?
  • How will we publish our IP address? Maybe we need some form of UDP broadcast?

@gfwilliams
Copy link
Member

I'd say configurable (both port, and whether it starts at connect). It could be done by looking for a named JsVar in execInfo.hiddenRoot (which I guess you'd want to do for WiFi autoconnect anyway). You can do that even before the interpreter starts.

It'd be nice to use the existing socketserver code (so WebSockets can be enabled when they're implemented, maybe even TLS), but I'm not sure how we'd do that while being able to survive a reset() or load() as it uses JsVars for state :(

For security, it might be possible to also store a password in the JsVars mentioned above - it'd help a little (but I don't think a password should be required).

Not sure about IP - it'd be nice if it could broadcast its name such that Windows networking picked it up, but I don't know enough about that. To be honest most WiFi Access Points tell you what IP connected devices are so it's not the end of the world if it doesn't broadcast.

Just so you know - I noticed some malloc calls in the ESP8266 code. I imagine that's fine on ESP8266 where it's probably built in, but on other boards, malloc isn't used at all (everything is statically allocated, or is allocated in JsVars).

@nkolban
Copy link
Contributor Author

nkolban commented Oct 5, 2015

When the ESP8266 starts and begins running Espruino ... we should assume that it will be using the UART0 (Serial) as the console. I suspect that what we need to do is that when a telnet client connection is received, we switch the console from UART0 to telnet .... and then when/if the telnet client connection is lost/closed, we switch back to Serial. Study needs to be done to ensure that we know how to switch consoles and what, if anything, special might have to be done to support that.

@gfwilliams
Copy link
Member

Yes - something similar is done for USB right now - there's DEFAULT_CONSOLE_DEVICE (usually Serial1), and then when USB is connected it automatically swaps to that (and vice versa). I wonder whether a slightly more professional solution to it might be better though (not sure what?)

@gfwilliams
Copy link
Member

Just to add: the implementation should be designed to handle the Espruino device being a server, but it should also allow it to be a client that connects to a port on a server (with auto reconnect).

Having it connect to a server means you don't have to worry about sharing the IP, and in many cases it could be significantly easier for users.

@nkolban
Copy link
Contributor Author

nkolban commented Oct 10, 2015

I'm not sure I understand the notion of how submission of JS code to a board hosting Espruino over a TCP/IP connection relates to Espruino being a client of a remote server? I can trivially see Espruino being a server to which an application (such as Web IDE) connects and pushes new JavaScript to it. That seems clean enough. But what would it mean to be a client to a remote server?

Can you elaborate on your thinking with some use cases and some example user stories?

@gfwilliams
Copy link
Member

You have a server that serves up the Web IDE on port 80, and that forwards information from the IDE to a board that is connected to it (possibly remotely)

Having the board initiate the connection means you don't need to know the ip, and it can also connect to a server on the net through a firewall (obv TLS would be needed for that).

@nkolban
Copy link
Contributor Author

nkolban commented Oct 11, 2015

I'm still not getting it. I'm now imagining a Web Server. And I can point my browser to the Web Server and dynamically load the Web IDE (I think that is what you are saying). Now I am running the Web IDE locally in my browser. Now I have an ESP8266 running Espruino ... sure my browser would then make a TCP connection to the ESP8266 and the ESP8266 would be a "server" to my browser request. I got that solidly.

What I am not seeing is how an ESP8266 would make a client request to the browser hosted IDE? I have a feeling that there is a design/architecture story I am missing?

@gfwilliams
Copy link
Member

  • You're at work, and your Espruino is at home - behind your router. How would you then communicate with it?
  • Or what if you got your Espruino working and then the DHCP lease ended and it got given a new IP address?
  • also web browsers can't do normal socket connections, and websocket connections have (I think) a same-origin policy for security, so if you load the IDE off server A, it can't then communicate with server B.

All the server would have to do is to 'relay' the information between a connection to the Espruno, and a websocket implementation to your web browser.

Anyway, the implementation of the server isn't important - all we need to do is to make sure that you can get a working console interface on both client and server.

@nkolban
Copy link
Contributor Author

nkolban commented Oct 12, 2015

Just thinking out loud .... would mDNS not allow us to have the ESP8266 be detectable on a LAN? As I understand it, mDNS is a protocol where something that wants to connect to a device by name and doesn't know its IP address publishes an mDNS find by UDP broadcast. The device (ESP8266) that has that name responds with its IP address ... thus allowing a browser (for example) to find the named ESP8266. If I am reading the mDNS story correctly, this would allow a browser to find a device with the protocol:

http://<ESP8266LogicalName>.local

As for modifying the Espruino at home while you are at work, I have no great answer to that.

For the browser same origin policy ... I could imagine loading the web page from the ESP8266. So for example:

http://myesp8266.local/webide

The web page would then appear on the browser served up by the ESP8266. Does the WebIDE have to be stored on the ESP8266? Not necessarily ... the ESP8266 could be configured with a URL to retrieve the real WebIDE from the Internet and pass it through to the browser for execution. From the browser's perspective though, there would be nothing but the ESP8266.

Again ... I am most assuredly NOT recommending or suggesting these idea ... they are only mental doodles that came to mind while reading the last posting.

@gfwilliams
Copy link
Member

Yes, so I'm not saying ESP8266 being a server is a bad idea or impossible - there are just some benefits to having it be a client as well.

@nkolban
Copy link
Contributor Author

nkolban commented Nov 14, 2015

Back in the very earliest days of ESP8266 porting, an experiment was performed which provided a telnet interface to Espruino ... this was needed as that time, we were still bootstrapping even basic UART connections. The telnet code was commented out but left within the build tree. A commit has been added to remove those source files and their references as there is nothing harvestable from these telnet source files as knowledge has grown and now shown that what they did was not even close to how we would want network loading of code.

gfwilliams added a commit that referenced this issue Nov 16, 2015
@gfwilliams
Merge pull request #709 from nkolban/master
0d4be92 
#605 Removal of redundant source files.
@MaBecker
Copy link
Contributor

MaBecker commented Mar 1, 2016

@gfwilliams or @tve can this be closed ?

@nkolban nkolban closed this as completed Mar 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ESP8266 This is only a problem on ESP8266 devices
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gfwilliams @samehhady @nkolban @MaBecker