Weird issues reading from the wifi module

[digi_guy]

I can't seem to get all of the data being sent to wifi module from a webpage, I assume it's a buffering issue but I can't figure out which buffer. 

At first I was trying to read and then print it back to the webpage, but that wasn't quite working.

Lately I've been using the DigiKeyboard to print it to my desktop, which has had a lot more success. 

I'm working off the server example and some of the features work the way they should, for example when starting out
Serial.readStringUntil(' '); will find the GET message, and running it again will get my the requested address, but then things are pretty garbled after that.

I've tried a few loops using:
    do {
       DigiKeyboard.println((char)Serial.read());
      } while(Serial.peek() != -1);
 
And what I've noticed is that it's returning a -1 (empty) pretty quick even if there is still a ton of data left to read.

Now here is where it gets weird: my goal is to be able to send POST requests from the webpage, which means sending a chunk of data buried in the html message, instead of relying on the address part which GET uses.  In my test example I was sending a first and last name so the data looked like fname=Bob&lname=Smith.

The problem is that part of the message is pretty deep, and until now I couldn't seem to read it or parse it, until I used find

Serial.find('fname'); will return true

DigiKeyboard.println(Serial.readString()); will then print the rest of the message "=Bob&lname=Smith"

Does anyone have any suggestions on how to read and then print the entire html request message? 

[defragster]

@digi_guy - do you have a second Pro?   And an OLED?  I'm wondering if I set up a sample with the new OLED code to do serial debugging output if you could use it.  I have the WiFi kit and looked at the Wiki - I'm not sure all the pins used - but DMM says serial 6&7 are free.  It might help you spew debug info without interfering with the WiFi operation.  Since the USB is not hardware supported it can't help to be using that even though hardware pins 3&5 are not used either.

[digi_guy]

I have a second pro, but not the OLED. You are probably right about using USB at the same time, but I can't find a better alternative, maybe once I get softserial working I could try using ttl. 

Even then, I still won't know where I'm running into problems since I've got buffer issues on the way in and on the way out.

[defragster]

Without an OLED you could use real serial and talk PRO to PRO and use the second Pro to push to USB - but yes, you'd need to be in a position of control in the code to push anything out that would be diagnostically useful. 

When I get back to it I am going to add that as a feature on my OLED Serial debugger - so a longer listing on the terminal could track stuff over time that would overwrite on the OLED if you blinked - or left for any time.

SoftSerial is also software based - and might end up in conflict with timing - so I made my test on the real serial and found it to work well.

[digi_guy]

I found a slightly better work around by using my old ttl->serial cable from my RPi. The message I'm trying to read looks like this:

Code: [Select]

POST / HTTP/1.1
Host: 10.10.100.254:8080
Connection: keep-alive
Content-Length: 90
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://10.10.100.254:8080
User-Agent: Mozilla/5.0 (Linux; Android 4.1.2; LG-LS970 Build/JZO54K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.93 Mobile Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://10.10.100.254:8080/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8

name1=your+name+1&name2=your+name+2&name3=your+name+3&mail=your+email&comment=your+comment
The stuff at the bottom is the data sent by the POST, whereas a GET message would have that in the address line.

If I try to use Serial.read() I only get the first 15 characters, and can't seem to get the rest. But if I use Serial.readString() I can get the first 130!

If instead I use Serial.find("en-US,en;q=0.8") it will print out the blank lines and the the data!

I'd really like to be able to read all of what gets sent so that I can properly parse it.

[digistump]

The methods you've come up with are some of the better ways to do this - you can't just load it all into a variable to look at later - there isn't enough RAM to do this (true for Arduinos too, thought the limit is a bit higher on them). If you look at many examples for the Arduino ethernet or wifi shield you'll see that under some of those libraries they are using find() and readUntil(), etc to find the area they need to parse and then read only that into a variable.

These issues are a lot less on a Yun/Due/other heavy duty Arduino because they have tons of ram - of course at a much higher price.

To parse a POST string like this I'd probably start with:
Serial.find("en-US,en;q=0.8");
String var = Serial.readString();

and then split the var on & and then loop through and do what I need to with the different parameters sent

The 15 characters you run into with read is because the Serial buffer holds 16 bytes - so if you don't read it fast enough that is all it can retain. You can up this by selecting the 32 or 64 byte buffer variants from the tools->board menu - but then you have less ram to work with in your sketch because the buffer uses it.

YOu can also slow down the baudrate (try something extreme like 300 to start) on the module and in your sketch and that will give you more time to read it before the buffer fills up.