Keil Studio Cloud
Mistake on this page?
Report an issue in GitHub or email us
Docs › Mbed OS and the Pelion IoT Platform › Integrating with Arm Treasure Data › Send data securely to Arm Treasure Data

Send data securely to Arm Treasure Data

This tutorial introduces securely sending data from Mbed OS devices to Arm Treasure Data using two methods:

  • HTTPS library - Send data directly to the Treasure Data REST API.
  • Fluentd using fluent logger library - Send data to a hosted Fluentd instance that aggregates and forwards the data on to your treasure data account.

Arm Mbed TLS provides equal in-transit security to both libraries. We recommend the HTTPS library for development and the Fluentd library for production. The tradeoff between the two is size of code on chip, size of data in transit and setup complexity:

  • Code size on chip - The HTTPS library is ~50KB of ROM space on chip. This is due to the HTTP stack. Both libraries use Mbed TLS to secure the connections, which is ~7KB per connection on your stack for both libraries.
  • Data size in transit - The HTTPS library sends data as an ASCII JSON string. The Fluend library uses MessagePack (binary encoded JSON) across a TLS connection. This means that on average the Fluentd library uses less bandwidth to send an equivalent message. When you pay per byte transmitted from both your power budget and data plan it matters.
  • Maintenance - Initially, setting up the HTTPS library on a device and having it send data directly to Treasure Data is easier, but what if you want to change what the device is doing or how its data is reported? If you are using the HTTPS library, you must issue a firmware update to every device to change how it formats its data, whereas if you are using a Fluend server, you can modify the Fluentd config file on the server to change how data is formatted and processed.

The following steps show how to send data using first the HTTPS library and then Fluentd.

HTTPS library

To use the HTTPS library, use the mbed-os-example-treasuredata-rest program. This program turns on Mbed OS device statistics by enabling the MBED_ALL_STATS_ENABLED macro and then sends heap/CPU/stack/system information to Treasure Data.

Video tutorial

Import code

You can compile the program using any of the following development tools:

  • Arm Mbed CLI (offline):
mbed import https://github.com/armmbed/mbed-os-example-treasuredata-rest
  • Arm Online Compiler:
main.cppImport into Mbed IDE 
/* WiFi Example
 * Copyright (c) 2016 ARM Limited
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include "mbed.h"
#include "treasure-data-rest.h"

#define BUFF_SIZE   100

// Default network interface object. Don't forget to change the WiFi SSID/password in mbed_app.json if you're using WiFi.
NetworkInterface *net = NetworkInterface::get_default_instance();

int main(void){

    printf("\r\nTreasure Data REST API Demo\n");

    // Connect to the internet (DHCP is expected to be on)
    printf("Connecting to the network using the default network interface...\n");
    net = NetworkInterface::get_default_instance();

    nsapi_error_t net_status = -1;
    for (int tries = 0; tries < 3; tries++) {
        net_status = net->connect();
        if (net_status == NSAPI_ERROR_OK) {
            break;
        } else {
            printf("Unable to connect to network. Retrying...\n");
        }
    }

    if (net_status != NSAPI_ERROR_OK) {
        printf("ERROR: Connecting to the network failed (%d)!\n", net_status);
        return -1;
    }

    printf("Connected to the network successfully. IP address: %s\n", net->get_ip_address());

    printf("Success\n\n");
    printf("MAC: %s\n", net->get_mac_address());
    printf("IP: %s\n", net->get_ip_address());
    printf("Netmask: %s\n", net->get_netmask());
    printf("Gateway: %s\n", net->get_gateway());
    // printf("RSSI: %d\n\n", net.get_rssi());

    // Create Treasure data objects (Network, Database, Table, APIKey)
    TreasureData_RESTAPI* heap  = new TreasureData_RESTAPI(net,"test_database","heap_info", MBED_CONF_APP_API_KEY);
    TreasureData_RESTAPI* cpu   = new TreasureData_RESTAPI(net,"test_database","cpu_info",  MBED_CONF_APP_API_KEY);
    TreasureData_RESTAPI* stack = new TreasureData_RESTAPI(net,"test_database","stack_info",MBED_CONF_APP_API_KEY);
    TreasureData_RESTAPI* sys   = new TreasureData_RESTAPI(net,"test_database","sys_info",  MBED_CONF_APP_API_KEY);

    // Device Information Objects
    mbed_stats_cpu_t    cpuinfo;
    mbed_stats_heap_t   heapinfo;
    mbed_stats_stack_t  stackinfo;
    mbed_stats_sys_t    sysinfo;

    // Buffers to create strings in
    char cpu_buff  [BUFF_SIZE] = {0};
    char heap_buff [BUFF_SIZE] = {0};
    char stack_buff[BUFF_SIZE] = {0};
    char sys_buff  [BUFF_SIZE] = {0};

    int x = 0;

    // Get device health data, send to Treasure Data every 10 seconds
    while(1){
        {
            // Collect local data
            mbed_stats_cpu_get(  &cpuinfo);

            // Construct strings to send
            x = sprintf(cpu_buff,"{\"uptime\":%llu,\"idle_time\":%llu,\"sleep_time\":%llu,\"deep_sleep_time\":%llu}",
                                cpuinfo.uptime,
                                cpuinfo.idle_time,
                                cpuinfo.sleep_time,
                                cpuinfo.deep_sleep_time);
            cpu_buff[x]=0; // null terminate the string

            // Send data to Treasure data
            printf("\r\n Sending CPU Data: '%s'\r\n",cpu_buff);
            cpu->sendData(cpu_buff,strlen(cpu_buff));
        }
        {
            // Collect local data
            mbed_stats_heap_get(  &heapinfo);

            // Construct strings to send
            x=sprintf(heap_buff,"{\"current_size\":%d,\"max_size\":%d,\"total_size\":%d,\"reserved_size\":%d,\"alloc_cnt\":%d,\"alloc_fail_cnt\":%d}",
                                heapinfo.current_size,
                                heapinfo.max_size,
                                heapinfo.total_size,
                                heapinfo.reserved_size,
                                heapinfo.alloc_cnt,
                                heapinfo.alloc_fail_cnt);
            heap_buff[x]=0; // null terminate the string

            // Send data to Treasure data
            printf("\r\n Sending Heap Data: '%s'\r\n",heap_buff);
            heap->sendData(heap_buff,strlen(heap_buff));
        }
        {
            // Collect local data
            mbed_stats_stack_get(  &stackinfo);

            // Construct strings to send
            x=sprintf(stack_buff,"{\"thread_id\":%d,\"max_size\":%d,\"reserved_size\":%d,\"stack_cnt\":%d}",
                                stackinfo.thread_id,
                                stackinfo.max_size,
                                stackinfo.reserved_size,
                                stackinfo.stack_cnt);
            stack_buff[x]=0; // null terminate the string

            // Send data to Treasure data
            printf("\r\n Sending Stack Data: '%s'\r\n",stack_buff);
            stack->sendData(stack_buff,strlen(stack_buff));
        }
        {
            // Collect local data
            mbed_stats_sys_get(  &sysinfo);

            // Construct strings to send
            x=sprintf(sys_buff,"{\"os_version\":%d,\"cpu_id\":%d,\"compiler_id\":%d,\"compiler_version\":%d}",
                                sysinfo.os_version,
                                sysinfo.cpu_id,
                                sysinfo.compiler_id,
                                sysinfo.compiler_version);
            sys_buff[x]=0; // null terminate the string

            // Send data to Treasure data
            printf("\r\n Sending System Data: '%s'\r\n",sys_buff);
            sys->sendData(sys_buff,strlen(sys_buff));
        }
        
        thread_sleep_for(10000);

    }

    // Code should not get here, included for completeness
    net->disconnect();

    printf("\nDone, x=%d\n",x);

}

Set up variables

  1. Configure the Treasure Data API key in mbed_app.json by changing the api-key variable:

    "api-key":{


"help": "REST API Key for Treasure Data",

"value": "\"REPLACE_WITH_YOUR_KEY\""

},

  2. Wi-Fi credentials: If you're using Wi-Fi, add your SSID/password. If you are using ethernet, you do not need to add Wi-Fi credentials.

  3. Create a database called test_database in Treasure Data. Note: The tables are created automatically.

Compile and load

Next, you can compile and load your code onto your board. If you are not familiar with how to compile and load code, please look at the Mbed OS quick start tutorial.

After you have compiled your code and loaded it onto your board, open a serial terminal, and connect it to the board. View the output:

--- Terminal on /dev/tty.usbmodem146103 - 9600,8,N,1 ---
Treasure Data REST API Demo
Connecting to the network using the default network interface...
Connected to the network successfully. IP address: 192.168.43.202
Success

MAC: C4:7F:51:02:D9:5D
IP: 192.168.43.202
Netmask: 255.255.255.0
Gateway: 192.168.43.249

 Sending CPU Data: '{"uptime":6918609,"idle_time":0,"sleep_time":509277,"deep_sleep_time":0}'

 Sending Heap Data: '{"current_size":15260,"max_size":75334,"total_size":747954,"reserved_size":307232,"alloc_cnt":12,"alloc_fail_cnt":0}'

 Sending Stack Data: '{"thread_id":0,"max_size":4820,"reserved_size":12632,"stack_cnt":4}'

 Sending System Data: '{"os_version":51104,"cpu_id":1091551809,"compiler_id":2,"compiler_version":60300}'

Verify data in Treasure Data

Go to the Database list in Treasure Data, and open the test_database you created earlier. You can see the data from the board in the database. There is a 3- to 5-minute delay from when the data is sent to the database until the visualization system lets you see it, so please be patient, and wait for it to arrive. Be sure to refresh the page.

Note: The database tab shows how much data you have in the database and gives a few samples, but it does not show all your data. For that, you need to run queries.

Run queries

Now that you have data in Treasure Data, it's time to analyze and use it.

  1. Go to the Queries tab.
  2. Select the test_database, and run some queries. To learn more about how to run queries, please read the Treasure Data documentation.

Select all fields

Run select * from cpu_info to get a full list of all fields in the table.

Select certain fields, order by time

This query selects only certain columns from the table and orders them by the time field in ascending value. You can also replace asc with desc to get the order reversed.

select time, current_size, total_size, alloc_cnt, max_size, reserved_size, alloc_fail_cnt from heap_info
order by time asc;

Troubleshooting

If you experience issues, ensure you have at least 10KB of space left on your stack. You can also change the TD_DEBUG macro to true to turn on the Treasure Data debug printfs.

Fluentd

For mass deployments, we recommend you use Fluentd or fluentbit to aggregate and forward the data into Treasure Data. Depending on where you host your Fluentd instance, you will need to follow slightly different setup instructions. (localhost on your machine with self signed certificates or at a public IP address in the cloud with Certificate Authority (CA) signed certificates). This example uses MessagePack (a binary encoded JSON) to encode the data.

Set up Fluentd

Install

First, install Fluentd. Please see the Fluentd quick start for details.

Experienced users can use gem install Fluentd fluent-plugin-td.

Download example code

Download the example code. This repository contains both the embedded example code and the Fluentd configuration files.

Set configuration file

Run Fluentd using the provided configuration file fluentd --config ./fluentd-setup/fluentd.conf -vv. This file opens two ports: port 24227 for unencrypted TCP traffic and port 24228 for TLS encrypted traffic. The configuration is provided for reference. We strongly suggest using TLS encryption on port 24228 to secure your data in transit.

You can either run Fluentd on a public IP address with CA signed certificates (suggested for deployments), or locally on your machine using self signed certificates (recommended for prototyping/testing).

Signed by CA, running in cloud

If you have valid certificates from a CA, replace the fluentd.crt and fluentd.key files with the CA certificates. Then uncomment the lines in the fluentd.conf file for CA trusted certificates, comment out the lines for self-signed certificates and change the passphrase to match for your certificate:

	# cert_path ~/mbed-os-example-fluentlogger/fluentd-setup/fluentd.crt
	# private_key_path ~mbed-os-example-fluentlogger/fluentd-setup/fluentd.key
	# private_key_passphrase YOUR_PASSPHRASE
Self-signed certificates on localhost

Video tutorial

If you are running the Fluentd server locally on your machine to develop a proof of concept (PoC), you need to generate a new self-signed certificate (where the Common Name (CN) is the IP address of your machine), and modify the fluentd.conf file with the IP address of your machine. Each time you restart the Fluentd instance, it generates a new certificate that you need to copy and paste into your embedded code.

  1. Change the generate_cert_common_name parameter in fluentd.conf to be the IP address of the computer running the Fluentd server.
  2. Run openssl req -new -x509 -sha256 -days 1095 -newkey rsa:2048 -keyout fluentd.key -out fluentd.crt to generate new certificates. When entering the prompted values, make sure to match the parameters in the fluentd.conf file (US, CA, Mountain View and so on). Make sure the CN field is set to the IP address of the Fluentd server.

For example:

Country Name (2 letter code) []:US
State or Province Name (full name) []:CA
Locality Name (eg, city) []:Mountain View
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:192.168.1.85
Email Address []:

Mbed OS setup

Run the example code on your device. You can either import to the Mbed Online Compiler or use Mbed CLI to clone it locally, compile and load it to the board:

$ mbed import https://github.com/armmbed/mbed-os-example-fluentlogger
$ mbed compile --target auto --toolchain GCC_ARM --flash --sterm

Secure (TLS)

To send data to Fluentd over TLS (securely):

  1. Run openssl s_client -connect localhost:24228 -showcerts.
  2. Copy the certificate to fluentd-sslcert.h. If you are running the Fluentd server on localhost, this certificate will change every time you restart Fluentd. You need to rerun this command and recompile your embedded code every time you restart Fluentd.
  3. Modify the call in main.cpp to the FluentLogger object.
  4. Change the IP address to the IP address of the Fluentd server, or if you are hosting it in the cloud, change it to the web address where it is hosted. It is important that the IP address in the main.cpp file matches the IP address set in the CN field of the Fluentd server. Otherwise, it will not work because Mbed TLS uses strict CN verification.
  5. Compile the code and load it onto your board.

Success

Successful output on the Fluentd terminal:

 -0500 debug.test: ["sint",0,1,-1,-128,-32768,-2147483648]
 -0500 [trace]: #0 fluent/log.rb:281:trace: connected fluent socket addr="192.168.1.95" port=5522
 -0500 [trace]: #0 fluent/log.rb:281:trace: accepted fluent socket addr="192.168.1.95" port=5522
 -0500 debug.test: ["uint",0,1,128,255,65535,4294967295]
 -0500 [trace]: #0 fluent/log.rb:281:trace: connected fluent socket addr="192.168.1.95" port=5523
 -0500 [trace]: #0 fluent/log.rb:281:trace: accepted fluent socket addr="192.168.1.95" port=5523
 -0500 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=70248976563020
 -0500 debug.test: {"string":"Hi!","float":0.3333333432674408,"double":0.3333333333333333}
 -0500 [trace]: #0 fluent/log.rb:281:trace: connected fluent socket addr="192.168.1.95" port=5524
 -0500 [trace]: #0 fluent/log.rb:281:trace: accepted fluent socket addr="192.168.1.95" port=5524
 -0500 debug.test: {"string":"Hi!","float":0.3333333432674408,"double":0.3333333333333333}
 -0500 [trace]: #0 fluent/log.rb:281:trace: connected fluent socket addr="192.168.1.95" port=5525
 -0500 [trace]: #0 fluent/log.rb:281:trace: accepted fluent socket addr="192.168.1.95" port=5525
 -0500 [trace]: #0 fluent/log.rb:281:trace: adding metadata instance=70248976563020 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="td.fluentd_database.test", variables=nil>
 -0500 [trace]: #0 fluent/log.rb:281:trace: writing events into buffer instance=70248976563020 metadata_size=1
 -0500 [debug]: #0 fluent/log.rb:302:debug: Created new chunk chunk_id="585c249fd2ebe20867267de2fde7c4bc" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="td.fluentd_database.test", variables=nil>
 -0500 [trace]: #0 fluent/log.rb:281:trace: connected fluent socket addr="192.168.1.95" port=5526
 -0500 [trace]: #0 fluent/log.rb:281:trace: accepted fluent socket addr="192.168.1.95" port=5526
 -0500 debug.test: {"string":"Hi!","float":0.3333333432674408,"double":0.3333333333333333}

Setting Treasure Data databases and tables

The second field in the tag of your embedded code determines the database. For example, sending data to a tag called td.mydatabase.mytable logs the data to the database called mydatabase in the table mytable. You can modify the example configuration file to see this.

Debugging

For more verbose debug messages, turn on the following flags in mbed_app.json:

{
	"macros": [
		"MBEDTLS_SSL_DEBUG_ALL = 1"
	],
	"target_overrides": {
		"*": {
			"mbed-trace.enable": true
		}
	}
}
Previous Next
Important Information for this Arm website

This site uses cookies to store information on your computer. By continuing to use our site, you consent to our cookies. If you are not happy with the use of these cookies, please review our Cookie Policy to learn how they can be disabled. By disabling cookies, some features of the site will not work.