HTTP and HTTPS library for Mbed OS 5

Dependents:   MQTTGateway2 MQTTGatewayK64 http-example-wnc GuardRoom ... more

For the example program, see: sandbox/http-example.

This library is used to make HTTP and HTTPS calls from Mbed OS 5 applications.

HTTP Request API

NetworkInterface* network = /* obtain a NetworkInterface object */

const char body[] = "{\"hello\":\"world\"}";

HttpRequest* request = new HttpRequest(network, HTTP_POST, "http://httpbin.org/post");
request->set_header("Content-Type", "application/json");
HttpResponse* response = request->send(body, strlen(body));
// if response is NULL, check response->get_error()

printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
printf("body is:\n%s\n", response->get_body_as_string().c_str());

delete request; // also clears out the response

HTTPS Request API

// pass in the root certificates that you trust, there is no central CA registry in Mbed OS
const char SSL_CA_PEM[] = "-----BEGIN CERTIFICATE-----\n"
    /* rest of the CA root certificates */;

NetworkInterface* network = /* obtain a NetworkInterface object */

const char body[] = "{\"hello\":\"world\"}";

HttpsRequest* request = new HttpsRequest(network, SSL_CA_PEM, HTTP_GET "https://httpbin.org/status/418");
HttpResponse* response = request->send();
// if response is NULL, check response->get_error()

printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
printf("body is:\n%s\n", response->get_body().c_str());

delete request;

Note: You can get the root CA for a domain easily from Firefox. Click on the green padlock, click More information > Security > View certificate > Details. Select the top entry in the 'Certificate Hierarchy' and click Export.... This gives you a PEM file. Add the content of the PEM file to your root CA list (here's an image).

Mbed TLS Entropy configuration

If your target does not have a built-in TRNG, or other entropy sources, add the following macros to your mbed_app.json file to disable entropy:

{
    "macros": [
        "MBEDTLS_TEST_NULL_ENTROPY",
        "MBEDTLS_NO_DEFAULT_ENTROPY_SOURCES"
    ]
}

Note that this is not secure, and you should not deploy this device into production with this configuration.

Memory usage

Small requests where the body of the response is cached by the library (like the one found in main-http.cpp), require 4K of RAM. When the request is finished they require 1.5K of RAM, depending on the size of the response. This applies both to HTTP and HTTPS. If you need to handle requests that return a large response body, see 'Dealing with large body'.

HTTPS requires additional memory: on FRDM-K64F about 50K of heap space (at its peak). This means that you cannot use HTTPS on devices with less than 128K of memory, asyou also need to reserve memory for the stack and network interface.

Dealing with large response body

By default the library will store the full request body on the heap. This works well for small responses, but you'll run out of memory when receiving a large response body. To mitigate this you can pass in a callback as the last argument to the request constructor. This callback will be called whenever a chunk of the body is received. You can set the request chunk size in the HTTP_RECEIVE_BUFFER_SIZE macro (see mbed_lib.json for the definition) although it also depends on the buffer size ofthe underlying network connection.

void body_callback(const char* data, uint32_t data_len) {
    // do something with the data
}

HttpRequest* req = new HttpRequest(network, HTTP_GET, "http://pathtolargefile.com", &body_callback);
req->send(NULL, 0);

Dealing with a large request body

If you cannot load the full request into memory, you can pass a callback into the send function. Through this callback you can feed in chunks of the request body. This is very useful if you want to send files from a file system.

const void * get_chunk(uint32_t* out_size) {
    // set the value of out_size (via *out_size = 10) to the size of the buffer
    // return the buffer

    // if you don't have any more data, set *out_size to 0
}

HttpRequest* req = new HttpRequest(network, HTTP_POST, "http://my_api.com/upload");
req->send(callback(&get_chunk));

Socket re-use

By default the library opens a new socket per request. This is wasteful, especially when dealing with TLS requests. You can re-use sockets like this:

HTTP

TCPSocket* socket = new TCPSocket();

nsapi_error_t open_result = socket->open(network);
// check open_result

nsapi_error_t connect_result = socket->connect("httpbin.org", 80);
// check connect_result

// Pass in `socket`, instead of `network` as first argument
HttpRequest* req = new HttpRequest(socket, HTTP_GET, "http://httpbin.org/status/418");

HTTPS

TLSSocket* socket = new TLSSocket();

nsapi_error_t r;
// make sure to check the return values for the calls below (should return NSAPI_ERROR_OK)
r = socket->open(network);
r = socket->set_root_ca_cert(SSL_CA_PEM);
r = socket->connect("httpbin.org", 443);

// Pass in `socket`, instead of `network` as first argument, and omit the `SSL_CA_PEM` argument
HttpsRequest* get_req = new HttpsRequest(socket, HTTP_GET, "https://httpbin.org/status/418");

Request logging

To make debugging easier you can log the raw request body that goes over the line. This also works with chunked encoding.

uint8_t *request_buffer = (uint8_t*)calloc(2048, 1);
req->set_request_log_buffer(request_buffer, 2048);

// after the request is done:
printf("\n----- Request buffer -----\n");
for (size_t ix = 0; ix < req->get_request_log_buffer_length(); ix++) {
    printf("%02x ", request_buffer[ix]);
}
printf("\n");

Integration tests

Integration tests are located in the TESTS folder and are ran through Greentea. Instructions on how to run the tests are in http-example.

Mbed OS 5.10 or lower

If you want to use this library on Mbed OS 5.10 or lower, you need to add the TLSSocket library to your project. This library is included in Mbed OS 5.11 and up.

Tested on

  • K64F with Ethernet.
  • NUCLEO_F411RE with ESP8266.
  • ODIN-W2 with WiFi.
  • K64F with Atmel 6LoWPAN shield.
  • DISCO-L475VG-IOT01A with WiFi.
  • Mbed Simulator.

Changes

RevisionDateWhoCommit message
39:a8d157986ad8 2019-08-12 Jan Jongboom Fix parsed url leaking memory if path is empty default tip
38:bfd37a21f367 2019-08-09 Jan Jongboom wss schema should default to 443
37:98d83ca14b7b 2019-08-09 Jan Jongboom Revert adding parsed URL to constructor
36:d46da03715db 2019-08-09 Jan Jongboom Allow sending in ParsedURL instead of just string
35:b3ee394d1d2e 2019-08-09 Jan Jongboom Add ws/wss schema in the request builder
34:6daf67a96a91 2019-01-04 Jan Jongboom Add integration tests
33:5b2869cc8934 2019-01-04 Jan Jongboom Remove TLSSocket (now in Mbed OS 5.11)
32:fa4d71265625 2018-10-30 Jan Jongboom Update to use TLSSocket from upstream Mbed OS
31:b3730a2c4f39 2018-10-30 Jan Jongboom Switch to TLSSocket library tlssocket
30:3ad153a3fdfd 2018-09-06 Jennifer Plunkett Updated README with Mbed TLS config info
29:383e9bfbfbed 2018-08-10 Jan Jongboom Force usage of uint32_t instead of size_t - required for compilation on 64-bit systems
28:9a04ed79d67e 2018-03-27 Jan Jongboom Merge default
27:42b319540a74 2018-03-27 Jan Jongboom Reference TCPSocket in tls_socket
26:fe4e03a404fb 2018-01-25 Jan Jongboom Do not add \r\n after request body. Landing patch from https://os.mbed.com/users/mpeylo/code/mbed-http/rev/6c9da880cccd/
25:47d5c90c9ceb 2018-01-11 Jan Jongboom Add a note about the root-ca-selection
24:6c1651fd26b9 2018-01-11 Jan Jongboom Add chunked requests to readme
23:15fa2726f793 2018-01-11 Jan Jongboom Implement Transfer-Encoding: chunked on HTTP and HTTPS requests, allows streaming large blocks of data over HTTP
22:71fc1b1894f8 2018-01-03 Jan Jongboom Fix for https://github.com/ARMmbed/mbed-os/issues/5774
21:fcd2bfd31a39 2017-12-14 Jan Jongboom Add port Host header if not 80/443
20:0e63d6a93c02 2017-11-18 Jan Jongboom ifndef around HTTP_RECEIVE_BUFFER_SIZE
19:a5371b71de6f 2017-09-04 Jan Jongboom Add a note on RAM usage in the library
18:f7a85895a941 2017-09-04 Jan Jongboom tls_socket was leaking 1024 bytes of memory
17:6e0025e01b98 2017-07-31 Jan Jongboom http_server example will live at https://developer.mbed.org/teams/sandbox/code/http-webserver-example
16:1c918cc026c5 2017-07-28 Jan Jongboom Add request parsing to prepare for HTTP server example (2)
15:ffc77f212382 2017-07-28 Jan Jongboom Add request parsing to prepare for HTTP server example
14:3004056e4661 2017-03-30 Jan Jongboom Request with a body, but without Content-Length header should pretend it's chunked to allocate enough memory
13:992d953ecfb9 2017-03-30 Jan Jongboom Add tls_socket.h
12:530c2ebee349 2017-03-28 Jan Jongboom Update README for socket reuse
11:96e4dcb9c0c2 2017-03-28 Jan Jongboom Allow socket re-use in HTTPS and HTTP request
10:b017c7d2cf23 2017-03-28 Jan Jongboom Don't rely on strlen for request length, as it prevents sending \0 over the line. Patch via https://developer.mbed.org/users/ihere1/code/mbed-http/rev/ec9b75c349aa
9:1289162d9530 2017-03-02 Jan Jongboom If path is empty, default path to '/'. Otherwise HTTP requests become malformed on URLs without a path
8:6156404278bb 2017-03-02 Jan Jongboom Implement tolower, so ARMCC can compile the library too
7:2e3eedb9ca5c 2017-02-24 Jan Jongboom Add support for chunked encoding. Fix bug handle querystrings. HttpResponse should own header strings.
6:112d72c60e07 2017-02-23 Jan Jongboom Call is_body_complete() in HTTPS request as well
5:2456c90f02e9 2017-02-23 Jan Jongboom Update README
4:539df159e058 2017-02-23 Jan Jongboom Allocate body on the heap, not on the stack
3:8a6b003e3874 2017-02-23 Jan Jongboom Rely on Content-Length header to determine when request is processed
2:959baaa89148 2017-02-16 Jan Jongboom size_t is not available in ARMCC
1:52947b0505de 2017-02-16 Jan Jongboom Change receive buffer size to 8192
0:910f5949759f 2017-02-16 Jan Jongboom Initial commit