HTTP and HTTPS library for Mbed OS 5

Dependents:   MQTTGateway2 MQTTGatewayK64 http-example-wnc GuardRoom ... more

For the example program, see: sandbox/http-example.

This library is used to make HTTP and HTTPS calls from Mbed OS 5 applications.

HTTP Request API

NetworkInterface* network = /* obtain a NetworkInterface object */

const char body[] = "{\"hello\":\"world\"}";

HttpRequest* request = new HttpRequest(network, HTTP_POST, "http://httpbin.org/post");
request->set_header("Content-Type", "application/json");
HttpResponse* response = request->send(body, strlen(body));
// if response is NULL, check response->get_error()

printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
printf("body is:\n%s\n", response->get_body_as_string().c_str());

delete request; // also clears out the response

HTTPS Request API

// pass in the root certificates that you trust, there is no central CA registry in Mbed OS
const char SSL_CA_PEM[] = "-----BEGIN CERTIFICATE-----\n"
    /* rest of the CA root certificates */;

NetworkInterface* network = /* obtain a NetworkInterface object */

const char body[] = "{\"hello\":\"world\"}";

HttpsRequest* request = new HttpsRequest(network, SSL_CA_PEM, HTTP_GET "https://httpbin.org/status/418");
HttpResponse* response = request->send();
// if response is NULL, check response->get_error()

printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
printf("body is:\n%s\n", response->get_body().c_str());

delete request;

Note: You can get the root CA for a domain easily from Firefox. Click on the green padlock, click More information > Security > View certificate > Details. Select the top entry in the 'Certificate Hierarchy' and click Export.... This gives you a PEM file. Add the content of the PEM file to your root CA list (here's an image).

Mbed TLS Entropy configuration

If your target does not have a built-in TRNG, or other entropy sources, add the following macros to your mbed_app.json file to disable entropy:

{
    "macros": [
        "MBEDTLS_TEST_NULL_ENTROPY",
        "MBEDTLS_NO_DEFAULT_ENTROPY_SOURCES"
    ]
}

Note that this is not secure, and you should not deploy this device into production with this configuration.

Memory usage

Small requests where the body of the response is cached by the library (like the one found in main-http.cpp), require 4K of RAM. When the request is finished they require 1.5K of RAM, depending on the size of the response. This applies both to HTTP and HTTPS. If you need to handle requests that return a large response body, see 'Dealing with large body'.

HTTPS requires additional memory: on FRDM-K64F about 50K of heap space (at its peak). This means that you cannot use HTTPS on devices with less than 128K of memory, asyou also need to reserve memory for the stack and network interface.

Dealing with large response body

By default the library will store the full request body on the heap. This works well for small responses, but you'll run out of memory when receiving a large response body. To mitigate this you can pass in a callback as the last argument to the request constructor. This callback will be called whenever a chunk of the body is received. You can set the request chunk size in the HTTP_RECEIVE_BUFFER_SIZE macro (see mbed_lib.json for the definition) although it also depends on the buffer size ofthe underlying network connection.

void body_callback(const char* data, uint32_t data_len) {
    // do something with the data
}

HttpRequest* req = new HttpRequest(network, HTTP_GET, "http://pathtolargefile.com", &body_callback);
req->send(NULL, 0);

Dealing with a large request body

If you cannot load the full request into memory, you can pass a callback into the send function. Through this callback you can feed in chunks of the request body. This is very useful if you want to send files from a file system.

const void * get_chunk(uint32_t* out_size) {
    // set the value of out_size (via *out_size = 10) to the size of the buffer
    // return the buffer

    // if you don't have any more data, set *out_size to 0
}

HttpRequest* req = new HttpRequest(network, HTTP_POST, "http://my_api.com/upload");
req->send(callback(&get_chunk));

Socket re-use

By default the library opens a new socket per request. This is wasteful, especially when dealing with TLS requests. You can re-use sockets like this:

HTTP

TCPSocket* socket = new TCPSocket();

nsapi_error_t open_result = socket->open(network);
// check open_result

nsapi_error_t connect_result = socket->connect("httpbin.org", 80);
// check connect_result

// Pass in `socket`, instead of `network` as first argument
HttpRequest* req = new HttpRequest(socket, HTTP_GET, "http://httpbin.org/status/418");

HTTPS

TLSSocket* socket = new TLSSocket();

nsapi_error_t r;
// make sure to check the return values for the calls below (should return NSAPI_ERROR_OK)
r = socket->open(network);
r = socket->set_root_ca_cert(SSL_CA_PEM);
r = socket->connect("httpbin.org", 443);

// Pass in `socket`, instead of `network` as first argument, and omit the `SSL_CA_PEM` argument
HttpsRequest* get_req = new HttpsRequest(socket, HTTP_GET, "https://httpbin.org/status/418");

Request logging

To make debugging easier you can log the raw request body that goes over the line. This also works with chunked encoding.

uint8_t *request_buffer = (uint8_t*)calloc(2048, 1);
req->set_request_log_buffer(request_buffer, 2048);

// after the request is done:
printf("\n----- Request buffer -----\n");
for (size_t ix = 0; ix < req->get_request_log_buffer_length(); ix++) {
    printf("%02x ", request_buffer[ix]);
}
printf("\n");

Integration tests

Integration tests are located in the TESTS folder and are ran through Greentea. Instructions on how to run the tests are in http-example.

Mbed OS 5.10 or lower

If you want to use this library on Mbed OS 5.10 or lower, you need to add the TLSSocket library to your project. This library is included in Mbed OS 5.11 and up.

Tested on

  • K64F with Ethernet.
  • NUCLEO_F411RE with ESP8266.
  • ODIN-W2 with WiFi.
  • K64F with Atmel 6LoWPAN shield.
  • DISCO-L475VG-IOT01A with WiFi.
  • Mbed Simulator.
Committer:
Jan Jongboom
Date:
Mon Aug 12 11:45:31 2019 +0200
Revision:
39:a8d157986ad8
Parent:
33:5b2869cc8934
Fix parsed url leaking memory if path is empty

Who changed what in which revision?

UserRevisionLine numberNew contents of line
Jan Jongboom 0:910f5949759f 1 # HTTP and HTTPS library for mbed OS 5
Jan Jongboom 0:910f5949759f 2
Jennifer Plunkett 30:3ad153a3fdfd 3 This library is used to make HTTP and HTTPS calls from Mbed OS 5 applications.
Jan Jongboom 0:910f5949759f 4
Jan Jongboom 0:910f5949759f 5 ## HTTP Request API
Jan Jongboom 0:910f5949759f 6
Jan Jongboom 0:910f5949759f 7 ```cpp
Jan Jongboom 0:910f5949759f 8 NetworkInterface* network = /* obtain a NetworkInterface object */
Jan Jongboom 0:910f5949759f 9
Jan Jongboom 0:910f5949759f 10 const char body[] = "{\"hello\":\"world\"}";
Jan Jongboom 0:910f5949759f 11
Jan Jongboom 5:2456c90f02e9 12 HttpRequest* request = new HttpRequest(network, HTTP_POST, "http://httpbin.org/post");
Jan Jongboom 5:2456c90f02e9 13 request->set_header("Content-Type", "application/json");
Jan Jongboom 5:2456c90f02e9 14 HttpResponse* response = request->send(body, strlen(body));
Jan Jongboom 0:910f5949759f 15 // if response is NULL, check response->get_error()
Jan Jongboom 0:910f5949759f 16
Jan Jongboom 0:910f5949759f 17 printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
Jan Jongboom 5:2456c90f02e9 18 printf("body is:\n%s\n", response->get_body_as_string().c_str());
Jan Jongboom 5:2456c90f02e9 19
Jan Jongboom 5:2456c90f02e9 20 delete request; // also clears out the response
Jan Jongboom 0:910f5949759f 21 ```
Jan Jongboom 0:910f5949759f 22
Jan Jongboom 0:910f5949759f 23 ## HTTPS Request API
Jan Jongboom 0:910f5949759f 24
Jan Jongboom 0:910f5949759f 25 ```cpp
Jennifer Plunkett 30:3ad153a3fdfd 26 // pass in the root certificates that you trust, there is no central CA registry in Mbed OS
Jan Jongboom 0:910f5949759f 27 const char SSL_CA_PEM[] = "-----BEGIN CERTIFICATE-----\n"
Jan Jongboom 0:910f5949759f 28 /* rest of the CA root certificates */;
Jan Jongboom 0:910f5949759f 29
Jan Jongboom 0:910f5949759f 30 NetworkInterface* network = /* obtain a NetworkInterface object */
Jan Jongboom 0:910f5949759f 31
Jan Jongboom 0:910f5949759f 32 const char body[] = "{\"hello\":\"world\"}";
Jan Jongboom 0:910f5949759f 33
Jan Jongboom 5:2456c90f02e9 34 HttpsRequest* request = new HttpsRequest(network, SSL_CA_PEM, HTTP_GET "https://httpbin.org/status/418");
Jan Jongboom 5:2456c90f02e9 35 HttpResponse* response = request->send();
Jan Jongboom 0:910f5949759f 36 // if response is NULL, check response->get_error()
Jan Jongboom 0:910f5949759f 37
Jan Jongboom 0:910f5949759f 38 printf("status is %d - %s\n", response->get_status_code(), response->get_status_message());
Jan Jongboom 0:910f5949759f 39 printf("body is:\n%s\n", response->get_body().c_str());
Jan Jongboom 5:2456c90f02e9 40
Jan Jongboom 5:2456c90f02e9 41 delete request;
Jan Jongboom 0:910f5949759f 42 ```
Jan Jongboom 0:910f5949759f 43
Jan Jongboom 25:47d5c90c9ceb 44 **Note:** You can get the root CA for a domain easily from Firefox. Click on the green padlock, click *More information > Security > View certificate > Details*. Select the top entry in the 'Certificate Hierarchy' and click *Export...*. This gives you a PEM file. Add the content of the PEM file to your root CA list ([here's an image](img/root-ca-selection.png)).
Jan Jongboom 25:47d5c90c9ceb 45
Jan Jongboom 32:fa4d71265625 46 ### Mbed TLS Entropy configuration
Jan Jongboom 32:fa4d71265625 47
Jan Jongboom 32:fa4d71265625 48 If your target does not have a built-in TRNG, or other entropy sources, add the following macros to your `mbed_app.json` file to disable entropy:
Jan Jongboom 32:fa4d71265625 49
Jan Jongboom 32:fa4d71265625 50 ```json
Jan Jongboom 32:fa4d71265625 51 {
Jan Jongboom 32:fa4d71265625 52 "macros": [
Jan Jongboom 32:fa4d71265625 53 "MBEDTLS_TEST_NULL_ENTROPY",
Jan Jongboom 32:fa4d71265625 54 "MBEDTLS_NO_DEFAULT_ENTROPY_SOURCES"
Jan Jongboom 32:fa4d71265625 55 ]
Jan Jongboom 32:fa4d71265625 56 }
Jan Jongboom 32:fa4d71265625 57 ```
Jan Jongboom 32:fa4d71265625 58
Jan Jongboom 32:fa4d71265625 59 Note that this is **not** secure, and you should not deploy this device into production with this configuration.
Jan Jongboom 32:fa4d71265625 60
Jan Jongboom 19:a5371b71de6f 61 ## Memory usage
Jan Jongboom 19:a5371b71de6f 62
Jan Jongboom 19:a5371b71de6f 63 Small requests where the body of the response is cached by the library (like the one found in main-http.cpp), require ~4K of RAM. When the request is finished they require ~1.5K of RAM, depending on the size of the response. This applies both to HTTP and HTTPS. If you need to handle requests that return a large response body, see 'Dealing with large body'.
Jan Jongboom 19:a5371b71de6f 64
Jan Jongboom 32:fa4d71265625 65 HTTPS requires additional memory: on FRDM-K64F about 50K of heap space (at its peak). This means that you cannot use HTTPS on devices with less than 128K of memory, as you also need to reserve memory for the stack and network interface.
Jan Jongboom 19:a5371b71de6f 66
Jan Jongboom 24:6c1651fd26b9 67 ### Dealing with large response body
Jan Jongboom 0:910f5949759f 68
Jan Jongboom 5:2456c90f02e9 69 By default the library will store the full request body on the heap. This works well for small responses, but you'll run out of memory when receiving a large response body. To mitigate this you can pass in a callback as the last argument to the request constructor. This callback will be called whenever a chunk of the body is received. You can set the request chunk size in the `HTTP_RECEIVE_BUFFER_SIZE` macro (see `mbed_lib.json` for the definition) although it also depends on the buffer size of the underlying network connection.
Jan Jongboom 0:910f5949759f 70
Jan Jongboom 0:910f5949759f 71 ```cpp
Jan Jongboom 29:383e9bfbfbed 72 void body_callback(const char* data, uint32_t data_len) {
Jan Jongboom 0:910f5949759f 73 // do something with the data
Jan Jongboom 0:910f5949759f 74 }
Jan Jongboom 0:910f5949759f 75
Jan Jongboom 22:71fc1b1894f8 76 HttpRequest* req = new HttpRequest(network, HTTP_GET, "http://pathtolargefile.com", &body_callback);
Jan Jongboom 22:71fc1b1894f8 77 req->send(NULL, 0);
Jan Jongboom 0:910f5949759f 78 ```
Jan Jongboom 0:910f5949759f 79
Jan Jongboom 24:6c1651fd26b9 80 ### Dealing with a large request body
Jan Jongboom 24:6c1651fd26b9 81
Jan Jongboom 24:6c1651fd26b9 82 If you cannot load the full request into memory, you can pass a callback into the `send` function. Through this callback you can feed in chunks of the request body. This is very useful if you want to send files from a file system.
Jan Jongboom 24:6c1651fd26b9 83
Jan Jongboom 24:6c1651fd26b9 84 ```cpp
Jan Jongboom 29:383e9bfbfbed 85 const void * get_chunk(uint32_t* out_size) {
Jan Jongboom 24:6c1651fd26b9 86 // set the value of out_size (via *out_size = 10) to the size of the buffer
Jan Jongboom 24:6c1651fd26b9 87 // return the buffer
Jan Jongboom 24:6c1651fd26b9 88
Jan Jongboom 24:6c1651fd26b9 89 // if you don't have any more data, set *out_size to 0
Jan Jongboom 24:6c1651fd26b9 90 }
Jan Jongboom 24:6c1651fd26b9 91
Jan Jongboom 24:6c1651fd26b9 92 HttpRequest* req = new HttpRequest(network, HTTP_POST, "http://my_api.com/upload");
Jan Jongboom 24:6c1651fd26b9 93 req->send(callback(&get_chunk));
Jan Jongboom 24:6c1651fd26b9 94 ```
Jan Jongboom 24:6c1651fd26b9 95
Jan Jongboom 12:530c2ebee349 96 ## Socket re-use
Jan Jongboom 12:530c2ebee349 97
Jan Jongboom 12:530c2ebee349 98 By default the library opens a new socket per request. This is wasteful, especially when dealing with TLS requests. You can re-use sockets like this:
Jan Jongboom 12:530c2ebee349 99
Jan Jongboom 12:530c2ebee349 100 ### HTTP
Jan Jongboom 12:530c2ebee349 101
Jan Jongboom 12:530c2ebee349 102 ```cpp
Jan Jongboom 12:530c2ebee349 103 TCPSocket* socket = new TCPSocket();
Jan Jongboom 12:530c2ebee349 104
Jan Jongboom 12:530c2ebee349 105 nsapi_error_t open_result = socket->open(network);
Jan Jongboom 12:530c2ebee349 106 // check open_result
Jan Jongboom 12:530c2ebee349 107
Jan Jongboom 12:530c2ebee349 108 nsapi_error_t connect_result = socket->connect("httpbin.org", 80);
Jan Jongboom 12:530c2ebee349 109 // check connect_result
Jan Jongboom 12:530c2ebee349 110
Jan Jongboom 12:530c2ebee349 111 // Pass in `socket`, instead of `network` as first argument
Jan Jongboom 12:530c2ebee349 112 HttpRequest* req = new HttpRequest(socket, HTTP_GET, "http://httpbin.org/status/418");
Jan Jongboom 12:530c2ebee349 113 ```
Jan Jongboom 12:530c2ebee349 114
Jan Jongboom 12:530c2ebee349 115 ### HTTPS
Jan Jongboom 12:530c2ebee349 116
Jan Jongboom 12:530c2ebee349 117 ```cpp
Jan Jongboom 31:b3730a2c4f39 118 TLSSocket* socket = new TLSSocket();
Jan Jongboom 32:fa4d71265625 119
Jan Jongboom 32:fa4d71265625 120 nsapi_error_t r;
Jan Jongboom 31:b3730a2c4f39 121 // make sure to check the return values for the calls below (should return NSAPI_ERROR_OK)
Jan Jongboom 32:fa4d71265625 122 r = socket->open(network);
Jan Jongboom 32:fa4d71265625 123 r = socket->set_root_ca_cert(SSL_CA_PEM);
Jan Jongboom 32:fa4d71265625 124 r = socket->connect("httpbin.org", 443);
Jan Jongboom 12:530c2ebee349 125
Jan Jongboom 12:530c2ebee349 126 // Pass in `socket`, instead of `network` as first argument, and omit the `SSL_CA_PEM` argument
Jan Jongboom 12:530c2ebee349 127 HttpsRequest* get_req = new HttpsRequest(socket, HTTP_GET, "https://httpbin.org/status/418");
Jan Jongboom 12:530c2ebee349 128 ```
Jan Jongboom 12:530c2ebee349 129
Jan Jongboom 31:b3730a2c4f39 130 ## Request logging
Jan Jongboom 31:b3730a2c4f39 131
Jan Jongboom 31:b3730a2c4f39 132 To make debugging easier you can log the raw request body that goes over the line. This also works with chunked encoding.
Jan Jongboom 31:b3730a2c4f39 133
Jan Jongboom 31:b3730a2c4f39 134 ```cpp
Jan Jongboom 31:b3730a2c4f39 135 uint8_t *request_buffer = (uint8_t*)calloc(2048, 1);
Jan Jongboom 31:b3730a2c4f39 136 req->set_request_log_buffer(request_buffer, 2048);
Jan Jongboom 31:b3730a2c4f39 137
Jan Jongboom 31:b3730a2c4f39 138 // after the request is done:
Jan Jongboom 31:b3730a2c4f39 139 printf("\n----- Request buffer -----\n");
Jan Jongboom 31:b3730a2c4f39 140 for (size_t ix = 0; ix < req->get_request_log_buffer_length(); ix++) {
Jan Jongboom 31:b3730a2c4f39 141 printf("%02x ", request_buffer[ix]);
Jan Jongboom 31:b3730a2c4f39 142 }
Jan Jongboom 31:b3730a2c4f39 143 printf("\n");
Jan Jongboom 31:b3730a2c4f39 144 ```
Jan Jongboom 31:b3730a2c4f39 145
Jan Jongboom 33:5b2869cc8934 146 ## Integration tests
Jan Jongboom 33:5b2869cc8934 147
Jan Jongboom 33:5b2869cc8934 148 Integration tests are located in the `TESTS` folder and are ran through [Greentea](https://github.com/ARMmbed/greentea). Instructions on how to run the tests are in [http-example](https://os.mbed.com/teams/sandbox/code/http-example/).
Jan Jongboom 33:5b2869cc8934 149
Jan Jongboom 33:5b2869cc8934 150 ## Mbed OS 5.10 or lower
Jan Jongboom 33:5b2869cc8934 151
Jan Jongboom 33:5b2869cc8934 152 If you want to use this library on Mbed OS 5.10 or lower, you need to add the [TLSSocket](https://github.com/ARMmbed/TLSSocket) library to your project. This library is included in Mbed OS 5.11 and up.
Jan Jongboom 33:5b2869cc8934 153
Jan Jongboom 0:910f5949759f 154 ## Tested on
Jan Jongboom 0:910f5949759f 155
Jan Jongboom 0:910f5949759f 156 * K64F with Ethernet.
Jan Jongboom 0:910f5949759f 157 * NUCLEO_F411RE with ESP8266.
Jan Jongboom 19:a5371b71de6f 158 * ODIN-W2 with WiFi.
Jan Jongboom 24:6c1651fd26b9 159 * K64F with Atmel 6LoWPAN shield.
Jan Jongboom 33:5b2869cc8934 160 * DISCO-L475VG-IOT01A with WiFi.
Jan Jongboom 32:fa4d71265625 161 * [Mbed Simulator](https://github.com/janjongboom/mbed-simulator).
Jan Jongboom 24:6c1651fd26b9 162
Jan Jongboom 24:6c1651fd26b9 163 But this should work with any Mbed OS 5 device that implements the `NetworkInterface` API.