Stumped on HTTP client problem

22 May 2012

Stumped on HTTP client problem... Using an HTTPS: URL, connecting to google's calendar access via private password. Works fine if google calendar has just one event to return from the URL time-span I pass. This is about 2700 bytes. I'm using a blocking http.get(). If there are 2 calendar events in the time window, the HTTP client get function hangs. never returns, never times out. My declaration of the data buffer is increased to huge... HTTPText responseText("text/html", 8192);

Plug same URL into web browser... result is as expected.. browser displays XML text.

any ideas?

ALSO... How do I know which version of the HTTP client class is in the netservices library (there are 2+ versions around)?

22 May 2012

The variety of "stuff" that you can get back from an HTTP request is crazy. I would suggest running your code from your PC first, so that you can capture and analyze the response you are getting.

I would bet that the MBED HTTP classes don't handle many of the advanced HTTP response features, such as multi-part documents.

Get a traffic capture tool like TCPDump or TCPFlow, or WireShark, or curl - there are lots of tools that will monitior your network traffic. I'll bet even Firefox or Chrome have debug modes where they will show you the HTTP headers and document source.

Also, be aware that the MBED's HTTP classes probably aren't going to deal with all the Web 2.0 goodies like Javascript and AJAX and so forth. Anything even remotely interesting on the web nowadays is going to be built from all those advanced goodies - not just a simple HTML document.

23 May 2012

Thanks... I did run this on the PC; browser is happy. And I did run this on the mbed and print out what it got- for case (1), a single calendar event from Google via HTTPS. I can only guess that the mbed HTTP Client hangs (does not timeout) is related to the slight increase in size for case 2, the two-calendar event GET. I have wireShark and use it. And I've used FireFox's "show web page" as text... there is nothing I can see that should affect the client. Again, the browser works OK.

I don't see any AJAX or scripts... just XML in an HTLM tag wrapper. I wouldn't expect the client to do any client-side processing.. just a GET. And if it works for case (1) it should be the same for (2). I wouldn't think the client parses any of this.. just looks for end of transfer due to a TCP FIN or whatever.

Still stumped.

23 May 2012

OK. More confused. There are two .cpp and .h files for HTTPclient. one that seems to be in NetServices, " 2010 Donatien Garnier" and one floating about "Copyright (C) 2012 ARM Limited." not the same.

This makes me have indigestion. The get() methods differ. One seems to use HTTPtext and the other HTTPdata as the data container. And different function signatures.

My confusion must be due to me being a relative newbie here?

23 May 2012

First, I would test your hypothesis about the size of the response. Find a website (or create one on a local server if you can) that has a simple HTML page that is the same size or larger than the 2-record response that is failing. See if the MBED can handle it - this will tell you if size is the problem, or if it's something else.

Second - try both HTTP projects, see which one you like better. There used to be at least six different IP protocol projects to choose from.

23 May 2012

thanks...

Isn't NetServices supposed to be the mBed standard/official library? It's in the cookbook/code.

This duplication of solutions, with no explanations of history, which is least buggy, etc. is discouraging me from continuing with mBed.

23 May 2012

Hi Steve,

Anything in the Handbook is mbed/us, and something we officially support. Up until now, we have not published a networking stack, although a number have been written and published by various excellent mbed users in the cookbook.

One of them was Donatien who did it as a side project when he worked for me one summer, and it turns out that one was incredibly popular! But I know it was definitely an experiment in what was possible rather than a core stable stack, hence the focus on lots of interfaces/clients/etc. I know some people pushed some versions of NetServices with fixes/improvements, but i'd agree it is not a stack focused on correctness.

We were very happy to see these contributions and the amount they were enabling people as starting points. But as mbed has become more and more popular, it became obvious that we might want to think about taking on the task of putting an official stack in place that we can support. Our decision was that if we did want to do this, we'd want to be able to support bsd-like sockets, which needs an RTOS. With CMSIS-RTOS in place, this becomes possible!

So the plan is now to write and release an official networking library based on LwIP and a DMA driven ethernet stack, inspired by some of Donatien's work (Donatien will be contributing too :), that supports a very simple sockets programming model to allow all sorts of IP (TCP/UDP) applications to be built on top. The goal is something with good performance, excellent stability, and simple usage. Emilio has been working hard on this and a lot of the core and tests are in place and the results are looking very promising! We will soon be working on it "live" within the collaboration infrastructure, which is due to roll out to early testers this week, so some people can start playing with the low level code from then as we build up :)

In the mean time, maybe look for some of the versions of NetServices that have been published since Donatien's version, and you may find some fixes.

Simon

23 May 2012

Steve, just try to do development on these chips with one of the standard development tools (especially the free ones - *shudder*). Those are just brutal in comparison - you'll feel like those companies are actively trying make your life as hard as possible.

MBED is sort of between that world and the clicky-draggy-plug-the-legos together world. True the libraries are still evolving, but the ARM/MBED guys are actually pretty involved (as you can see above). It's a bit better than the open source world - where there's a wealth of free code available and none of it does what you need to do or works on the platform you need to use.

So hang in there, feel free to grouse a bit, but keep at it and you will win the fight!

23 May 2012

The timeout for a request looks like it is 30 seconds. Do you hang for longer than that? If so, you crashed somehow. If not, you can try to extend the timeout.

You should be careful of memory usage - there is only 64K RAM, and you are trying to work with an 8K response buffer. You may be overflowing the stack.

You should try turning on the debugging output in the HTTP classes. Get the library as source code (you can do this from both the project page and the compiler/import I believe), and un-comment the debug #define, and build that into your project.

I'm also surprised that you are getting anything at all making requests against an HTTPS url. There must be a request header that specifies plain text so that the response is not encrypted, because I don't see any code for handling SSL encryption.

24 May 2012

yes, it never times out. I declared the response buffer as 8KB and as a static global rather than a local variable off the stack. I know the response size is well under 8KB. I did go searching for the library source - came up with classes of the same name. But different author. No clue where to find the sources for what is in the offical NetServices. Probably obvious to veterans, but not a newbie like me.

SSL- not? Really? hmm. How can this work! I'd better check that. I jumped to the conclusion, (absence of documentation) that there was a self-signed certificate for SSL.

I've done lots of development with IAR's EWARM; a competitor of the beloved Keil/ARM. Both are excellent. Atmel's new Studio 6 supporting their ARM chips with MS's Studio and GCC underneath is a brilliant move. NXP should do the same. Eclipse and the lot are a career tangent to be avoided (focus on the solution, not the tools!).

I have to recalibrate my expectations for mBed. I had hoped that things like NetServices were vetted and documented before becoming condoned.

24 May 2012

I like Rowley Crossworks myself.

Eclipse is... not for me.

I have coincidentally just been putting TCP/IP support together for a Cortex M3 and the story is remarkably similar. Get the vendor's port of LwIP and figure out how to set it up in your project. Hit the forums for advice.