HTTPClient non-blocking http.get(...) problem

18 May 2011

Hi guys,

I have to face the following problem while playing around with the HTTPClient class. I want to send a non-blocking HTTP get(...) to a device(btw. blocking works fine) as shown in main.cpp /media/uploads/mr_q/main.txt

On wireshark I even cannot see any broadcast/ARP at all. Any idea what is going wrong here?

Best wishes

18 May 2011

Hi Richard

If you want to use the non-blocking method, you need to call Net::poll() in your main loop.

Regards
Daniel

19 May 2011

Hi Daniel

Thanks a lot! Now the get is sent. However my callback routine is not entered at all!

I have doen some traces now and mentioned that the mbed seems to not get the ack from the server! The server does some retransmission. mbed retransmitts the http get once, immediately acknowledged by the server... followed by retransmissions and then mbed runs into timeout and closes the connection???

Seems as if there was a bug in the lib!?

Btw. is it possible to set up a client in parallel to a server on the mbed!

Thanks a lot for your help!

Best wishes

20 May 2011

Hi Richard

Have you looked at Donatien's non-blocking streaming example here http://mbed.org/users/donatien/programs/HTTPClientStreamingExample/5yz1c

Regards
Daniel

23 May 2011

Hi Daniel,

yes, but it's the same problem with that example, too!

The client seems to not get the HTTP message "200 OK" from the server, so the server continues retransmitting [TCP Retransmission]!

As mentioned I use static IP address!? Could this be the reason!????

Best wishes

23 May 2011

Hi Richard

I doubt that a static IP address could be the issue.

Are you starting with a template for your project like this one? http://mbed.org/users/segundo/programs/NetServices_HelloWorld/ljh122 (once you've imported the program then update the libraries within it).

There are plenty of people using that code quite happily for http clients, so if you still can't get it to work after that, it points to either a problem with the server you are connecting to (in which case try another) or a problem in the code for the particular case you are attempting.

If the latter, then you can debug the code by turning on debug options in the lwipopts.h file (you'll have to turn the library back into a folder to access the individual source files). If you need more debug code, then add your own using the DBG() macro.

Hope that helps
Daniel

30 May 2011

Hi Daniel,

I didn't get my HTTPClient work so far!

Please find attached the source code (/media/uploads/mr_q/cig_httpclient.zip) as well as two wireshark traces (one trace shows the HTTPClient request done by Internet Explorer (/media/uploads/mr_q/browser_trace.pcap.txt) and the other on shows the mbed HTTPClient request (/media/uploads/mr_q/mbed_trace.pcap.txt) - just rename the trace files by deleting the ".txt" ending). As you can see. the server works great with the browser. When sending a get via mbed, mbed seems to not get the response from the server!

I tried to debug the TCPSocket of the NetServices library without success...

Hope you could give me some hints where to include some debug prints in the NetServices to trace if mbed receives the response from the server correctly...

Best wishes

30 May 2011

Hi Richard

Can you publish your program? Right-click on the program name in the compiler and select "Publish Program..." then post the generated link here. It will make it easier for me to import into my workspace.

Regards
Daniel

30 May 2011

Hi Daniel,

the complete program is included in the .zip file! Does that help?

When I try to publish my program, the compiler tells me that the program includes unpublished libraries due to my debug printfs in the NetServices lib and that I should publish the libraries first! I prefer to not publish the NetServices lib as it is better to use the original ones without my debug printfs!

Best wishes

30 May 2011

Hi Richard

You can publish a library and not make it public for the purposes of this kind of exercise. I should be able to import a zip file instead but it's more work for me ;-)

Is the server in your code meant to be accessible? If not (because it's your home server for example), can you try and reproduce this problem with a server that I can access as well?

Thanks
Daniel

30 May 2011

Hi Richard

An alternative to publishing a library that you've started editing is to right-click its name then select "Convert to Folder". Then you should be able to publish the program without complaint.

Regards
Daniel

30 May 2011

Hi Daniel,

you can find the published program under the following link:

http://mbed.org/users/mr_q/programs/CIG_HTTPClient/lrz5zi

I'm going to investigate a way for you to get access to the server I use!

Thanks a lot for your help!

Best wishes

30 May 2011

Hi Richard

I modified your code (and published it here http://mbed.org/users/mbed714/programs/cig_httpclient2/lrzskc):

  • made it DHCP (for convenience on my network and also assigns gateway and DNS for connection to mbed server)
  • took out some printf lines in TCPSocket (for recv and checkInst methods)
  • modified it to do "callback" get to mbed hello.txt and print the result

The result is this:

Debug

Setup OK
Send message
Enter TCPSocket::connect()
!m_pNetTcpSocket
Enter TCPSocket::onNetTcpSpcketEvent()
Enter TCPSocket::send()
Enter TCPSocket::send()
Enter TCPSocket::onNetTcpSpcketEvent()
Enter TCPSocket::onNetTcpSpcketEvent()
HTTPGetCallbackEvent->Result: 200
Result ok : Hello world!
Enter TCPSocket::resetOnEvent()
Enter TCPSocket::close()
Enter TCPSocket::close()

This seems to work with the get from the mbed server. It's quite possible your server responds differently from the mbed server, and it breaks the httpclient.

The httpclient was developed and tested using a limited number of servers (not like Internet Explorer one would imagine), so there is no reason it would work in every possible case.

In terms of debugging, I suggest you get an example working with your mbed (like I've just done), and then work out what the difference is between your working example and trying to connect to your particular server. The server may be doing something the httpclient is not expecting, and since the client is based on a state machine it will never complete.

You've got the idea with throwing in the printf for debugging purposes. You could increase your USB serial speed to make sure you don't slow down if you are printing a lot.

Regards
Daniel

PS To increase USB serial speed add this declaration:

Serial pc(USBTX, USBRX); 

Then add this at the top of your main() method:

pc.baud(115200);

Don't forget to change the baud rate on your terminal program.

30 May 2011

Hi again,

Thanks again. In my terminal program I see the following trace:

/media/uploads/mr_q/terminal_trace.txt

Can you give me a hint where in the library I can set a debug output to trace any incoming and outgoing IP/TCP telegram!?

Best wishes

30 May 2011

Hi Richard

I've modified the program to print out the Ethernet frames received by the mbed which are carrying non-broadcast IP packets (see http://mbed.org/users/mbed714/programs/cig_httpclient2/lrz749). This will allow you to see at the lowest level what is being received by the mbed stack - for your info, these changes are made in eth_drv.cpp.

Again, test this on an http get to a working server - you should see the Ethernet frames arriving.

Then try with your server and see whether the frames come in the same place, or whether the time out is due to nothing coming at all.

The NetServices code is built on top of the lwip stack, so if you want to start debugging you can look at what goes into the stack via the driver (eth_drv.cpp) and also what happens to convert the raw lwip information into TCPSockets - have a look at the lwipNetTcpSocket.cpp code, and if you want debugging of what is happening here, uncomment line 27 in the source file (to define the __DEBUG symbol).

With debug traces, it would really help to have an example of something working (eg. get of hello world from mbed server) as well as something that doesn't work (eg. your server), just to show that your mbed has actually got a fully functioning connection to the network.

Regards
Daniel

31 May 2011

Hi Daniel,

Thanks a lot! I enabled all the DBG outputs in eth_drv.cpp and lwipNetTcpSocket.cpp as well (added some additional, too).

Mbed receives the retransmission telegrams (as you can see in the terminal trace file attached). However it seems as the data aren't recevied in lwipNetTcpSocket.cpp...

/media/uploads/mr_q/terminal_trace_v002.txt

So from my understanding there must go someting wrong in between! What is the next "layer" above eth_drv.cpp that I can debug?

Best wishes

31 May 2011

Hi Richard

The layer between the driver (eth_drv) and lwipNetTcpSocket is the standard lwip stack. Have a look here for further information:
http://lwip.wikia.com/wiki/LwIP_Wiki

Regards
Daniel

02 Jul 2011

Hi Richard

Did you get anywhere with this?

One thing might be to try Andrew Bonney's fix in http client.

Import libraryNetServicesMod

Modified version of NetServices. Fixes an issue where connections failed should the HTTP response status line be received in a packet on its own prior to any further headers. Changes are made to the HTTPClient.cpp file's readHeaders method.

Regards
Daniel