Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Found a way to upload data VERY fast by wifi #1853

Closed
raduprv opened this issue Apr 3, 2016 · 40 comments
Closed

Found a way to upload data VERY fast by wifi #1853

raduprv opened this issue Apr 3, 2016 · 40 comments

Comments

@raduprv
Copy link

raduprv commented Apr 3, 2016

I am posting this here because I never found the solution online, and I've been looking for days. Before this 'hack', the transfer speed on Windows it was about 6.7 KBps, and on Linux, about 36KBps
Now the speed is very high, something like many mbps, at least 187 KBps, limited by my SD reading time.

So, here is to to send data really fast, on any host:
First, you have to disable the Nagle 'feature', like so:
WiFiClient client; //stuff client.setNoDelay(1);

Then, you need to set the packet size HIGHER than the MTU, which is 1460 bytes. I got good results sending 1760 at a time, but as I said, the bottleneck is the SD card (I am uploading files by ftp). If the size is lower or equal than the MTU, you won't get good results. I think the trick is forcing it to send two packets.

Can someone please try to reproduce my findings? And if it works, maybe this should be documented somewhere (more than here).

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

@igrr
Copy link
Member

igrr commented Apr 4, 2016

Yes, doing it this way certainly works. There is a pull request (#1570) which attempts to speed up WiFiClient transfers so that ESP8266WebServer and other libraries can benefit from this. It's currently blocked by lack of testing though.

@raduprv
Copy link
Author

raduprv commented Apr 4, 2016

One question would be, why isn't the setNoDelay on by default?
I don't see any good reason for this to be 0.

@baruch
Copy link
Contributor

baruch commented Apr 19, 2016

The nagle algorithm is enabled by default for all systems and all tcp stacks. This is done to protect the network from inadvertant sending of very small packets, with nagle algorithm only the first packet is small and the following ones are batched up thus improving the overall network performance.

It makes sense to keep this behavior as is and allow application developers to override it if they have a good reason to do so.

@raduprv
Copy link
Author

raduprv commented Apr 19, 2016

Most TCP stacks do not send only one packet and wait for an ACK. They send a few packets at once.
Without this little almost undocumented hack, the transfer speed on windows is really, really bad, at around 6-7KBps

I don't think network congestion is an issue with this MCU, it dosn't have the capacity to saturate a network.

@tuxedo0801
Copy link
Contributor

Vote for keeping current behavior and document the tcpNoDelay option (as it is done also with other languages, like java...).

@raduprv
Copy link
Author

raduprv commented Apr 19, 2016

Documenting that feature would certainly help a lot. Speaking of which, is there any place with ALL the current esp8266 specific Arduino API?
There's got to be an easier way to find useful functions than reading the source code :)

@tuxedo0801
Copy link
Contributor

@galed3e3
Copy link

galed3e3 commented Jul 26, 2016

I've tried to implement the suggestions (client.setNoDelay(1); and 1760 packet), but am still getting slow uploads...approximately 82k bits/sec, or about 1/10 of the OP's rate. IE 11 is a little bit faster than Firefox 47 (22 sec vs 30 sec to serve a 307kB image from SPIFFS). I'd greatly appreciate some help (or is this the best that can be done from SPIFFS?)

      WiFiClient client = server.available();
      client.setNoDelay(1);
      // stuff...
      client.println("HTTP/1.1 200 OK");
      client.println("Content-Type: image/bmp");
      client.println("Content-Length: 307254");
      client.println();
      webFile = SPIFFS.open("/2016b.BMP", "r");

      if (webFile) {
        const int bufSize = 1760; 
        byte clientBuf[bufSize];
        int clientCount = 0;
        prevTime=millis();

        while (webFile.available())
        {
          clientBuf[clientCount] = webFile.read();
          clientCount++;

          if (clientCount > bufSize-1)
          {          
            client.write((const uint8_t *)clientBuf, bufSize);
            clientCount = 0;
          }
        }
        // final < bufSize byte cleanup packet
        if (clientCount > 0) client.write((const uint8_t *)clientBuf, clientCount);

        // close the file:
        webFile.close();
      } else {
        Serial.println("file !fnd");
      }

@igrr
Copy link
Member

igrr commented Jul 26, 2016

Which version are you using? Current git master or one of the releases?

@raduprv
Copy link
Author

raduprv commented Jul 26, 2016 via email

@igrr
Copy link
Member

igrr commented Jul 26, 2016

With the latest git version you should get better performance without using setNoDelay. Just do client.write(webFile);. This will handle buffering internally and send data as fast as it is acked, limited only be network roundtrip time and SPIFFS read time.

@galed3e3
Copy link

Thanks for the quick replies. I was using v. 2.2.0... will update to 2.3.0, try client.write(webFile); , and follow-up.

@igrr
Copy link
Member

igrr commented Jul 26, 2016

What i have described above is not available in 2.3.0. You need to get latest git version for that (check repository Readme for instructions).

@galed3e3
Copy link

Ah, so. Will do...

@galed3e3
Copy link

Yipee!!!! It works! Super fast! Less than a second to serve the 307kB file to IE and Firefox. Thank you very much!!!!

@Pe3ucTop
Copy link

Pe3ucTop commented Aug 17, 2016

Sorry,
But could anybody tell final bandwidth/throughput reached ?? 307kB/s it's ~2.5Mbps , is it true ?

And second question is (more dedicated to igrr):
Could anybody explain, how that speed (~2.5Mpbs) is reached?
I'm using direct programming of ESP8266 with NONOS_SDK , but got ~300Kbps (240-300Kbps show iptraf in linux (depend on PC)).
I need advice what to do to improve throughput.

@me-no-dev
Copy link
Collaborator

Yes it's true.
But then Im not igrr to answer the dedicated question

@raduprv
Copy link
Author

raduprv commented Aug 17, 2016

I forgot the exact speed, but it only took a few seconds to transfer a
large image from an external SD card. The speed was limited by the SD
read time, not by the wifi speed.

What exactly are you trying to do? Transfer files? If so, where are they
stored?

On 8/17/2016 6:33 PM, Pe3ucTop wrote:

Sorry,
But could anybody tell final bandwidth/throughput reached ?? 307kB/s
it's ~2.5Mbps , is it true ?

And second question is (more dedicated to igrr):

  • Could anybody explain, how that speed (~2.5Mpbs) is reached? I'm
    using direct programming of ESP8266 with NONOS_SDK , but got
    ~300Kbps (240-300Kbps show iptraf in linux (depend on PC)). I need
    advice what to do to improve throughput.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#1853 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABC3X7UemmQFV10X1G7dXhqhlepB95dwks5qgymxgaJpZM4H-psm.

@Pe3ucTop
Copy link

Thanks for yours answers,
I need to transfer special data I receive from other device over UART (at ~5Mbps) and I2S(~5Mbps) .. I2S part not finished yet, but I have speed problems in WiFi part already.
Currently I transmit over TCP by packets 1460 , tried with/wo NoDelay connection option - same result.
Any advice what to try would be grate.

@me-no-dev
Copy link
Collaborator

NoDelay can only work with packets below 1460 bytes. You send as much as you can (2 x 1460 the first time, then 1460 each) and wait for ack. When ack is received, you send another 1460 bytes.
Speed depends on many things though ;) like time taken to read the bytes and send them and overall network traffic.

@adotagarwal
Copy link

This information has been very useful to me, so thank you all for posting it all.

I wanted to share my experience and see if anyone can comment on a few questions which I had.

I tried compiling the SD Web Server example using the 2.3.0 tools available in the board manager and was able to test the download speeds - they were pretty abysmal at ~ 300kbps (call it 30kb/s). This was also true for uploads.

After finding the information in this thread, I tried pulling the latest ESP package from git and dropping it in - without modifying any of the code in the SD Web Server example, again. This time, I noticed that the files would start transferring (the first 32kb) and then stop, the debug would print "sent less than expected!"

I reverted to the 2.3.0 version, and applied the changes mentioned above to the library (HTTPWebServer) -- that is setNoDelay(1) and changed the HTTP packet size to 1760 from 1460). This resulted in throughput speeds of approx. 1.8mbps (225kb/s).

My questions are:

  1. Am I doing something wrong, or does the 32kb file download stirke a chord with anyone?
  2. Is 1.8mbps approx the max throughput we can expect? I know that the SD card can read at a much higher rate.

Thank you!

@Kakarotas
Copy link

Kakarotas commented Sep 23, 2016

Hello everyone,
After lots of trial I finally was able to get 465 KB/s upload speed using SPIFFS and HTTP post. I am using latest git version SDK. Here is the code I was using:

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
//#include <SD.h>
#include <SPI.h>
#include "FS.h"

const char* ssid     = "linksys"; //change it
const char* password = "******";
File webFile;
char servername[] = "192.168.1.100"; //change it
WiFiClient client;

unsigned long laikas;

void setup() {
  Serial.begin(230400);
  // We start by connecting to a WiFi network
 SPIFFS.begin();
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("WiFi connected,");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.println(rssi); 

  laikas = millis();
  webFile = SPIFFS.open("/u1.png", "r"); //change it

  //webFile.close();
  laikas = millis() - laikas;
  //prints time since program started
  Serial.println(laikas);
  laikas = millis();
  //SIUNTIMAS HTTTP
    if (client.connect(servername, 80)) {
      client.setNoDelay(true);
      Serial.println("Connected to Server");

      laikas = millis();
      // Make a HTTP request:
      client.println("POST /~Carre/errtest.php?azyx=1&d=1 HTTP/1.1");
      client.println("Host: 192.168.1.1"); //change it
      client.println("Connection: close");
      //client.println("Cache-Control: max-age=0");
      client.println("Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryjg2qVIUS8teOAbN3");
      client.println("Content-Length: 1048772"); //file size  //change it
      client.println();
      client.println("------WebKitFormBoundaryjg2qVIUS8teOAbN3");
      client.println("Content-Disposition: form-data; name=\"FileGDF\"; filename=\"pic.png\""); //change it
      client.println("Content-Type: application/octet-stream");
      client.println();
      client.write(webFile);
      client.println(); //file end

      client.println("------WebKitFormBoundaryjg2qVIUS8teOAbN3--");
      client.println();
      webFile.close();
      Serial.println("done");}
  laikas = millis() - laikas;
  //prints time since sending started
  Serial.println(laikas);
  if(!eRcv());
  client.stop();
  SPIFFS.end();
}

void loop() {
}

byte eRcv()
{
  byte respCode;
  byte thisByte;
  while(!client.available()) delay(1);
  respCode = client.peek();
  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);
  }
  return 1;
}

This is errtest.php server side file for receiving the data:

<?php
$target_dir = "uploads/";
$filename = basename(@$_FILES["FileGDF"]["name"]);
$target_file = $target_dir . $filename;
$uploadOk = 0;
$FileType = pathinfo($target_file,PATHINFO_EXTENSION);

if(isset($_GET["azyx"])&&(isset($_GET["d"]))) {
  if (file_exists($target_file)) {
    if (file_put_contents($target_file,file_get_contents(@$_FILES["FileGDF"]["tmp_name"]))){
      echo "The file ". $filename. " has been uploaded.<br />";
      $uploadOk = $uploadOk + 1;
    } else {
      echo "Sorry, there was an error uploading your file.<br />";
    }
  }else{
    if (move_uploaded_file(@$_FILES["FileGDF"]["tmp_name"], $target_file)) {
      echo "The file ". basename($filename). " has been uploaded.<br />";
      $uploadOk = $uploadOk + 1;
    } else {
      echo "Sorry, there was an error uploading your file.<br />";
    }
  }
}
?>

Other notes:

Is 1.8mbps approx the max throughput we can expect? I know that the SD card can read at a much higher rate.

Yes you can speed up SPI using SD.begin(SS, CLOCK_SPEED) (for example 50000000 instead of 8Mhz default, this way you are reading at maximum sd card speed ). This will decrease total reading time from 4s to 1.8s reading 1MB of data. Total uploading time also decreases, but it is of course slower than using SPIFFS.

@Interein2
Copy link

Interein2 commented Sep 23, 2016

Server side File Accept code in php is the same as above, but instead of ussing SPIFFS, reading data from SD example:

 #include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <SD.h>
#include <SPI.h>
//#include "FS.h"

//const char* ssid     = "linksys";
//const char* password = "*******";

File webFile;
char servername[] = "192.168.1.1";
WiFiClient client;

unsigned long laikas;


#define MTU_Size    2*1460  // this size seems to work best

// file sending from SD card
byte clientBuf[MTU_Size];
int clientCount = 0;
int progresas = 0;
int zenklas   = 0;
int zenklas2   = 0;

// change fileName to your file (8.3 format!)
char fileName[13] = "pic.png";

void setup() {
  Serial.begin(230400);
  // We start by connecting to a WiFi network

  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.print("WiFi connected,");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.println(rssi); 
  if(SD.begin(SS,50000000) == 0){    // clk 50MHz
    Serial.println(F("SD init fail"));          
  }else{
    Serial.println(F("SD init success"));   
  }
/** part of timing only to check how long to read file takes  */
  laikas = millis();

  webFile = SD.open(fileName,FILE_READ);
  while(webFile.available())
  {
    if (webFile.available()>= MTU_Size){
      clientCount = MTU_Size;
      webFile.read(&clientBuf[0],clientCount);
    }else{
      clientCount = webFile.available();
      webFile.read(&clientBuf[0],clientCount);
    }
    if(clientCount > 0)
    {
      //client.write((const uint8_t *)&clientBuf[0],MTU_Size);
      clientCount = 0;
    }
  }
  webFile.close();
  laikas = millis() - laikas;
  //prints time since program started
  Serial.println(laikas);
/** END of read file test time   */

  laikas = millis();
  webFile = SD.open(fileName,FILE_READ);
  //SIUNTIMAS (POST) HTTTP
    if (client.connect(servername, 80)) {
      client.setNoDelay(true);
      Serial.println("Connected to Server");

      laikas = millis();
      // Make a HTTP request:

      client.println("POST /~Carre/errtest.php?azyx=1&d=1 HTTP/1.1");
      client.println("Host: 192.168.1.1");
      client.println("Connection: close");
      client.println("Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryjg2qVIUS8teOAbN3");
      client.print("Content-Length: ");client.println(webFile.size()+196);  //+196 is owerhead form boundary tags
      client.println();
      client.println("------WebKitFormBoundaryjg2qVIUS8teOAbN3");
      client.println("Content-Disposition: form-data; name=\"FileGDF\"; filename=\"pic.png\"");
      client.println("Content-Type: application/octet-stream");
      client.println();
      //Serial.println(F("Writing"));
  progresas = 0;   
  while(webFile.available())
  {
    if (webFile.available()>= MTU_Size){
      clientCount = MTU_Size;
      webFile.read(&clientBuf[0],clientCount);
    }else{
      clientCount = webFile.available();
      webFile.read(&clientBuf[0],clientCount);
    }
    if(clientCount > 0)
    {
      client.write((const uint8_t *)&clientBuf[0],clientCount);
      progresas += clientCount;
      clientCount = 0;
      // some kind of progress output
      zenklas = (progresas * 100)/webFile.size();
      if (zenklas != zenklas2){
        zenklas2 = zenklas;
        Serial.print(".");
      }
      // end of progress output
    }
  }
  client.println();//file end

/** if another field required it can be started here **/
//      client.println("------WebKitFormBoundaryjg2qVIUS8teOAbN3");
//      client.println("Content-Disposition: form-data; name=\"okey\"");
//      client.println();
//      client.println("SEND");
      client.println("------WebKitFormBoundaryjg2qVIUS8teOAbN3--"); //form end
      client.println();
      webFile.close();
      Serial.println("done");
    }
  laikas = millis() - laikas;
  //prints time since program started
  Serial.println(laikas);
  if(!eRcv());
  client.stop();
}

void loop() {
}

byte eRcv()
{
  byte respCode;
  byte thisByte;

  while(!client.available()) delay(1);
  respCode = client.peek();
  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);
  }
  return 1;
}

@Pe3ucTop
Copy link

Pe3ucTop commented Sep 23, 2016

Hello to all,

Sorry, I can't comment on Arduino and it's library part. I'm programming ESP8266 with C and NONOS_SDK .
I use direct sending of 2 packets (sending second when first is written, not confirmed "sent") , the same advice from @me-no-dev .
And for throughput test I used same 1460 byte packet (only first 3 bytes differ/changed), for minimal CPU usage.
Got ~17Mbps throughput ( ~ 2 Mb/s ) , but you must have PC side application/WiFi which can handle/accept it.
In my condition I had problem at PC side.

@Interein2
Copy link

in our case server is apache + php

@mph070770
Copy link

mph070770 commented Nov 4, 2016

Hi Kakarotas,

I've just tried your SPIFFS example above in IDE 1.6.12 and SDK 2.3.0 on a NodeMCU 12E and it fails with the attached error. If I cast the write:

client.write((unsigned char)webFile);

Then it compiles ok but I don't have the server side to check if it's functional. I'm a bit new to all of this - is the cast a correct approach?

I'm actually trying to debug a problem in my own code where the client.write won't write the full amount of code (I needed to break it into around 3K chunks... which is very slow). I was originally using an array stored in ROM (with PROGMEM) but realised that using SPIFFS seems like the right approach to store large files in the flash.

Original:

[const char largeFile[] PROGMEM ={ 0xff,0x7f,0xff,0x7f,0xff,0x7f,0xff,0x7f,0xff,0x7f,0x01
,0x50,0x75,0x41,0xae,0x4e,0xa9,0x42,0x97,0x4d ..... (around 150KB in size....)

#define MTU_SIZE 2*1460
      for (int i=0; i<150000; i=i+MTU_SIZE) client.write(&largeFile[i],MTU_SIZE);   
      client.println();]

Now:

SPIFFS.begin();
File spiFile = SPIFFS.open("/big.bin", "r");
if (!spiFile) USE_SERIAL.println("file open failed");
else             USE_SERIAL.println("wav file opened");
client.write(spiFile);  //but I need to cast this to get it to work client.write((char)spiFile) and I'm not sure it's working?
spiFile.close();

This new approach seems to just send 1 byte (checked with wireshark). How can I send the whole file without breaking it up into chunks and having the delays associated with .write?

Any advise would be appreciated!

Error running your example:

In file included from C:\Users\Administrator\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\libraries\ESP8266WiFi\src/ESP8266WiFi.h:39:0,

                 from C:\Users\Administrator\Documents\arduino\POST-example\POST-example.ino:1:

C:\Users\Administrator\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\libraries\ESP8266WiFi\src/WiFiClient.h: In instantiation of 'size_t WiFiClient::write(T&, size_t) [with T = unsigned char [1460]; size_t = unsigned int]':

C:\Users\Administrator\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\libraries\ESP8266WiFi\src/WiFiClient.h:84:55:   required from 'size_t WiFiClient::write(T&) [with T = fs::File; size_t = unsigned int]'

C:\Users\Administrator\Documents\arduino\POST-example\POST-example.ino:63:27:   required from here

C:\Users\Administrator\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\libraries\ESP8266WiFi\src/WiFiClient.h:123:36: error: request for member 'available' in 'source', which is of non-class type 'unsigned char [1460]'

     size_t left = source.available();

                                    ^

C:\Users\Administrator\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\libraries\ESP8266WiFi\src/WiFiClient.h:127:5: error: request for member 'read' in 'source', which is of non-class type 'unsigned char [1460]'

     source.read(buffer.get(), will_send);

     ^

exit status 1
Error compiling for board NodeMCU 1.0 (ESP-12E Module).

@martinayotte
Copy link
Contributor

With the above code, you are sending a single byte with the value of "spiFile" file handle, not the content of the file.

@mph070770
Copy link

Thanks Martin. To be honest, I've just copied the solution from kakarotas:

client.write(webFile);

If that's not correct, can you advise how to fix it?

Thanks

@martinayotte
Copy link
Contributor

I've posted a fix, but in the forum : http://www.esp8266.com/viewtopic.php?p=57668#p57668

@mph070770
Copy link

Thanks Martin. I've replied there too. I thought each additional client.write causes delays, and it didn't seem to be needed by kakarotas. I'll give it a go but i think it'll be slow, and certainly not giving me high KB of tranfer per second?

@patryk161
Copy link

patryk161 commented Feb 18, 2017

For all those who may have problems like mph070770. You have to instal git version and delete, move or rename (I preffer rename) the following folder (which contains boards info and 2.3. version files):
C:\Users\YourUserName\AppData\Local\Arduino15
Also you must restart arduino.
Here is also very simple example:

#include "FS.h"
#include "WiFiClient.h"
#include <ESP8266WiFi.h>

const char *ssidAP = "ESP";
const char *passwordAP = "12345678";
String http_header = ("HTTP/1.1 200 OK\r\n"
                  "Content-Type: text/html\r\n"
                  "\r\n"
                  "<!DOCTYPE HTML>\r\n"
                  "<html>\r\n"
                  "<!DOCTYPE html>\n"
                  "<html>\n");
WiFiServer server(80);
WiFiClient client;

void setup() {
  Serial.begin(115200);
  WiFi.mode(WIFI_AP);
  WiFi.softAP(ssidAP, passwordAP);
  IPAddress myIP = WiFi.softAPIP();
  Serial.print("AP IP address: ");
  Serial.println(myIP);
  server.begin();
  SPIFFS.begin();
}
void loop() {
  client = server.available();
  if (!client) {
    Serial.println("I'm in loop");
    delay(1000);
    return;
     }
 
  File  webFile = SPIFFS.open("/YourFile.html", "r"); // I use html file because .html and .txt has the simplest header
  client.println(http_header);
  client.write(webFile);
  webFile.close();
}

@buzzy
Copy link

buzzy commented Apr 29, 2017

There might be a bug somewhere in the core library. I have a gif-file that is about 200kb. If I try to fetch it, it usually works just fine, but if I keep refreshing the page, it gives me errors once in a while. It looks like the entire file is not sent and gets stuck at about 199kb. You can replicate it with this simple code:

Just make sure you have a file called "test.gif" in your SPIFFS.

#include "Arduino.h"
#include <ESP8266WiFi.h>
#include <FS.h>

WiFiServer server(80);
WiFiClient client;

void setup()
{
  WiFi.mode(WIFI_AP);
  WiFi.softAP("Hello-World");

  //start UART and the server
  Serial.begin(74880);

  server.begin();
  server.setNoDelay(true);

  //Mount file system
  SPIFFS.begin();
}

void loop()
{
  client = server.available();
  if (!client)
  {
    Serial.println("I'm in loop");
    delay(1000);
    return;
  }

  File webFile = SPIFFS.open("/test.gif", "r");

  client.println("HTTP/1.1 200 OK");
  client.println("Connection: close");
  client.println("Content-Type: image/gif");
  client.println("Content-Length: " + String((webFile.size())));
  client.println();

  client.write(webFile);
  webFile.close();
}

@ESP4eva
Copy link

ESP4eva commented May 6, 2017

I am struggling to read a plain text SPIFFS file. The file is 100 lines long with each line being 1kb. It reads the first 30 lines within a couple of seconds but then each subsequent line takes 5 seconds or more. Sample sketch below, anyone know what the problem is?

/* SearchSPIFFS.ino 6 May 2017
   SPIFFS plain text file has 100 lines, each line approx 1kb
   After reading 30 lines (30kb) in less than 2 seconds it takes several seconds PER line
 */
#include "FS.h"

String line;
unsigned int lineNumber;
unsigned int line2Find = 32;
File MyFile = SPIFFS.open("SomeData.txt", "r");

void test(){
      while(MyFile.available()) {   // we could open the file, so loop through it to find the record we require
        lineNumber++;        
        Serial.println(lineNumber);          // show line number of SPIFFS file
        line = MyFile.readStringUntil('\n'); // Read line by line from the file
        if(lineNumber == line2Find){        
          Serial.println(F("Found record"));                             
          break;                             // exit while loop once record is found
        }
      }   
}

void setup() {
  Serial.begin(115200);
  Serial.println(F("\nStarted"));
  test();                                    // find line 32 of plain text SPIFFS file
}

void loop() {
  yield();
}

@devyte
Copy link
Collaborator

devyte commented Oct 17, 2017

The original issue seems to be fixed in latest git per relevant comments above, and per the referenced PR. There is a known issue with concurrent requests (e.g.: running a webserver and refreshing the browser repeatedly), which is going to be addressed with lwip2.
I am therefore closing this. If there is still a performance problem, or performance enhancement that could be implemented, please open a new issue and add the relevant details.

@devyte devyte closed this as completed Oct 17, 2017
@haxord7hasib
Copy link

so its not possible 5 Mbps download spped from esp8266 sd server?

@d-a-v
Copy link
Collaborator

d-a-v commented Feb 7, 2018

@haxord7 raw esp-upload bandwidth exceeds 5Mbits/s.

@Autobot86
Copy link

Hi friends,

I try to Upload SPIFFS large size .txt file [500KB] in HTTP web server POST request method. But my data didn't upload my server after use the given code.

this is my code
String Link = "POST/my_server_URL/HTTP/1.1";
const char*host="MY_server_IP";
File file;
if (client.connect(host, port))
{
client.setNoDelay(true);
Dir dir = SPIFFS.openDir("/");
if (dir.next()) {

  String File_name = dir.fileName();
  Serial.print("File_name : ");
  Serial.println(File_name);
  file = SPIFFS.open(File_name, "r");
  client.println(Link);
  client.print("Host: ");
  client.println(host);
  client.println("Connection: close");
  client.println("Content-Type : application/x-www-form-urlencoded");

  client.println("Content-Length: 124354");
  Serial.println(file.size());
  client.println();
  client.write("Large_data=");   // Variable Name
  client.write(file);     // Send file 
}

}

Serial monitor continuously show only file name and file size otherwise server posting or connection error details not show I don't know will be next please help me.

@devyte
Copy link
Collaborator

devyte commented Jul 3, 2018

There are timing requirements to respect on the ESP, and your code doesn't.
Why are you trying to implement your own http client? There is one already implemented: httpClient. You need to do more research.

@avesus

This comment has been minimized.

@mbenitez01
Copy link

Hi...I'm saving to SPIFFS content edited in a text area.. through web sockets..but I find the limit being 2K bytes aprox...
I'm sure I'm missing something but I can not tell what it is....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests