Sample program that can send the recognition data from HVC-P2 to Fujitsu IoT Platform using REST (HTTP)

Dependencies:   AsciiFont GR-PEACH_video GraphicsFramework LCD_shield_config R_BSP USBHost_custom easy-connect-gr-peach mbed-http picojson

Information

Here are both English and Japanese description. First, English description is shown followed by Japanese one. 本ページには英語版と日本語版の説明があります。まず英語版、続いて日本語版の説明となります。

Overview

This sample program shows how to send the cognitive data gathered by Omron HVC-P2 (Human Vision Components B5T-007001) to IoT Platform managed by FUJITSU ( http://jp.fujitsu.com/solutions/cloud/k5/function/paas/iot-platform/ ).

Hardware Configuration

/media/uploads/dkato/composition_hvcp2_demo.jpg

When executing this sample program, please configure the above H/W as shown below:
/media/uploads/dkato/composition_hvcp2_demo_2.jpg

Also, please close JP3 of GR-PEACH as follows:
/media/uploads/RyoheiHagimoto/usb.jpg

Application Preconfiguration

Build Procedure

  • Import this sample program onto mbed Compiler
  • In GR-PEACH_HVC-P2_IoTPlatform_http/IoT_Platform/iot_platform.cpp, please replace <ACCESS CODE> with the access code you set up on IoT Platform. For details on how to set up Access Code, please refer to the above Application Setup. Then, please delete the line beginning with #error macro.

Access Code configuration

#define ACCESS_CODE <Access CODE>
#error "You need to replace <Access CODE for your resource> with yours"
  • In GR-PEACH_HVC-P2_IoTPlatform_http/IoT_Platform/iot_platform.cpp, please replace <Base URI>, <Tenant ID> and <Path-to-Resource> below with yours and delete the line beginning with #error macro. For details on <Base URI> and <Tenant ID>, please contact FUJITSU LIMITED. Also, for details on <Path-to-Resource>, please refer to the above Application Setup.

URI configuration

std::string put_uri_base("<Base URI>/v1/<Tenant ID>/<Path-to-Resource>.json");
#error "You need to replace <Base URI>, <Tenant ID> and <Path-to-Resource> with yours"
**snip**
std::string get_uri("<Base URI>/v1/<Tenant ID>/<Path-to-Resource>/_past.json");
#error "You need to replace <Base URI>, <Tenant ID> and <Path-to-Resource> with yours"
  • Compile the program. If compilation is successfully terminated, the binary should be downloaded on your PC.
  • Plug Ethernet cable into GR-PEACH
  • Plug micro-USB cable into the port which lies on the next to RESET button. If GR-PEACH is successfully recognized, it appears as the USB flash disk named mbed as show below:
    /media/uploads/HinoNaka/mbed_drive.png
  • Copy the downloaded binary to mbed drive
  • Press RESET button on GR-PEACH in order to run the program. If it's successfully run, you can see the following message on terminal:
    /media/uploads/HinoNaka/iotplatform_log.png

Format of Data to be sent to IoT Platform

In this program, the cognitive data sent from HVC-P2 is serialized in the following JSON format and send it to IoT Platform:

  • Face detection data

{
    "RecodeType": "HVC-P2(face)"
    "id": <GR-PEACH ID>-<Sensor ID>"
    "FaceRectangle": {
        "Height": xxxx,
        "Left": xxxx,
        "Top": xxxx,
        "Width": xxxx,
    },
    "Gender": "male" or "female",
    "Scores": {
        "Anger": zzz,
        "Hapiness": zzz,
        "Neutral": zzz,
        "Sadness": zzz,
        "Surprise": zzz
    }
}

xxxx: Top-left coordinate, width and height of the rectangle circumscribing the detected face in LCD display coordinate system
zzz: the value indicating the expression estimated from the detected face

//

  • Body detection data

{
    "RecodeType": "HVC-P2(body)"
    "id": <GR-PEACH ID>-<Sensor ID>"
    "BodyRectangle": {
        "Height": xxxx,
        "Left": xxxx,
        "Top": xxxx,
        "Width": xxxx,
    }
}

xxxx: Top-left coordinate, width and height of the rectangle circumscribing the detected body in LCD display coordinate system

概要

本プログラムは、オムロン社製HVC-P2 (Human Vision Components B5T-007001)で収集した各種認識データを、富士通社のIoT Platform ( http://jp.fujitsu.com/solutions/cloud/k5/function/paas/iot-platform/ ) に送信するサンプルプログラムです。

ハードウェア構成

/media/uploads/dkato/composition_hvcp2_demo.jpg

本プログラムを動作させる場合、上記H/Wを下図のように接続してください。 /media/uploads/dkato/composition_hvcp2_demo_2.jpg

また下図に示すGR-PEACHのJP3をショートしてください。 /media/uploads/RyoheiHagimoto/usb.jpg

アプリケーションの事前設定

ビルド手順

  • 本サンプルプログラムをmbed Compilerにインポートします
  • 下記に示すGR-PEACH_HVC-P2_IoTPlatform_http/IoT_Platform/iot_platform.cpp中の<ACCESS CODE> を、IoT Platform上で設定したアクセスコードで上書きしてください。<Access Code>の設定方法につきましては、上述のApplication Setupを参照願います。また #errorマクロで始まる行を削除してください。

Access Code configuration

#define ACCESS_CODE <Access CODE>
#error "You need to replace <Access CODE for your resource> with yours"
  • 下記に示すGR-PEACH_HVC-P2_IoTPlatform_http/IoT_Platform/iot_platform.cpp中の<Base URI><Tenant ID>、および<Path-to-Resource>>を適当な値に置換えるとともに、#errorマクロで始まる行を削除してください。ここで、<Base URI>、 <Tenant ID>の詳細につきましては富士通社へご確認願います。また<Path-to-Resource>>につきましては、Application Setupの項を参照ください。

URI configuration

std::string put_uri_base("<Base URI>/v1/<Tenant ID>/<Path-to-Resource>.json");
#error "You need to replace <Base URI>, <Tenant ID> and <Path-to-Resource> with yours"
(中略)
std::string get_uri("<Base URI>/v1/<Tenant ID>/<Path-to-Resource>/_past.json");
#error "You need to replace <Base URI>, <Tenant ID> and <Path-to-Resource> with yours"
  • プログラムをコンパイルします。コンパイルが正常終了すると、バイナリがお使いのPCにダウンロードされます。
  • GR-PEACHのRJ-45コネクタにEthernetケーブルを差し込んでください。
  • USBA - Micro USBケーブルを、GR-PEACHのRESETボタンの隣に配置されたMicro USBポートに差し込んでください。GR-PEACHが正常に認識されると、下図に示すようにGR-PEACHがmbedという名称のUSBドライブとして認識されます。
    /media/uploads/HinoNaka/mbed_drive.png
  • ダウンロードしたバイナリをmbedドライブにコピーします。
  • RESETボタンを押下してプログラムを実行します。正常に実行された場合、下記に示すメッセージがターミナル上に表示されます。
    /media/uploads/HinoNaka/iotplatform_log.png

送信データフォーマット

本プログラムでは、HVC-P2が収集した認識データを下記のJSONフォーマットにシリアライズし、IoT Platformへ送信します。

  • Face detection data

{
    "RecodeType": "HVC-P2(face)"
    "id": <GR-PEACH ID>-<Sensor ID>"
    "FaceRectangle": {
        "Height": xxxx,
        "Left": xxxx,
        "Top": xxxx,
        "Width": xxxx,
    },
    "Gender": "male" or "female",
    "Scores": {
        "Anger": zzz,
        "Hapiness": zzz,
        "Neutral": zzz,
        "Sadness": zzz,
        "Surprise": zzz
    }
}

xxxx: LCD表示座標系における検出した顔に外接する矩形の左上座標・幅・高さ
zzz: 検出した顔から推定した各種感情を示す数値

//

  • Body detection data

{
    "RecodeType": "HVC-P2(body)"
    "id": <GR-PEACH ID>-<Sensor ID>"
    "BodyRectangle": {
        "Height": xxxx,
        "Left": xxxx,
        "Top": xxxx,
        "Width": xxxx,
    }
}

xxxx: LCD表示座標系における検出した人体に外接する矩形の左上座標・幅・高さ
Revision:
0:813a237f1c50
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/NTPClient/NTPClient.cpp	Thu Sep 07 18:15:54 2017 +0900
@@ -0,0 +1,219 @@
+/* NTPClient.cpp */
+/* Copyright (C) 2012 mbed.org, MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of this software
+ * and associated documentation files (the "Software"), to deal in the Software without restriction,
+ * including without limitation the rights to use, copy, modify, merge, publish, distribute,
+ * sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all copies or
+ * substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+ * BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+ * DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+//Debug is disabled by default
+#if 0
+//Enable debug
+#define __DEBUG__
+#include <cstdio>
+#define DBG(x, ...) std::printf("[NTPClient : DBG]"x"\r\n", ##__VA_ARGS__); 
+#define WARN(x, ...) std::printf("[NTPClient : WARN]"x"\r\n", ##__VA_ARGS__); 
+#define ERR(x, ...) std::printf("[NTPClient : ERR]"x"\r\n", ##__VA_ARGS__); 
+
+#else
+//Disable debug
+#define DBG(x, ...) 
+#define WARN(x, ...)
+#define ERR(x, ...) 
+
+#endif
+
+#include "NTPClient.h"
+
+#include "UDPSocket.h"
+
+#include "mbed.h" //time() and set_time()
+
+#define NTP_PORT 123
+#define NTP_CLIENT_PORT 0 //Random port
+#define NTP_TIMESTAMP_DELTA 2208988800ull //Diff btw a UNIX timestamp (Starting Jan, 1st 1970) and a NTP timestamp (Starting Jan, 1st 1900)
+
+NTPClient::NTPClient() : m_sock()
+{
+
+
+}
+
+#if MBED_MAJOR_VERSION >= 5
+
+#include <SocketAddress.h>
+#include <EthernetInterface.h>
+//extern EthernetInterface ei_;
+EthernetInterface ei_;
+
+class Endpoint {
+    SocketAddress sa_;
+public:
+    Endpoint() : sa_() {}
+    int set_address(const char* host, uint16_t port)
+    {
+        sa_.set_port(port);
+        return nsapi_create_stack(&ei_)->gethostbyname(host, &sa_);
+    }
+    const char* get_address() const { return sa_.get_ip_address(); }
+    operator const SocketAddress&() { return sa_; }
+    operator SocketAddress*() { return &sa_; }
+};
+
+class LegacyUS {
+    UDPSocket us_;
+public:
+    LegacyUS() : us_() {}
+    int bind(int port)
+    {
+        us_.open(&ei_);
+        return us_.bind(port);
+    }
+    void set_blocking(bool blocking, unsigned int)
+    {
+        us_.set_blocking(blocking);
+    }
+    int close() { return us_.close(); }
+    int sendTo(Endpoint& remote, char* packet, int length)
+    {
+        return us_.sendto(remote, packet, length);
+    }
+    int receiveFrom(Endpoint& remote, char* buffer, int length)
+    {
+        int ret = us_.recvfrom(remote, buffer, length);
+        if( ret == NSAPI_ERROR_WOULD_BLOCK )
+        {
+            wait(0.1);
+            ret = 0;
+        }
+        return ret;
+    }
+};
+#define m_sock (*((LegacyUS*)&m_sock))
+
+#endif  // MBED_MAJOR_VERSION >= 5
+
+NTPResult NTPClient::setTime(const char* host, uint16_t port, uint32_t timeout)
+{
+#ifdef __DEBUG__
+  time_t ctTime;
+  ctTime = time(NULL);
+  DBG("Time is set to (UTC): %s", ctime(&ctTime));
+#endif
+
+  //Create & bind socket
+  DBG("Binding socket");
+  m_sock.bind(0); //Bind to a random port
+  
+  m_sock.set_blocking(false, timeout); //Set not blocking
+
+  struct NTPPacket pkt;
+
+  //Now ping the server and wait for response
+  DBG("Ping");
+  //Prepare NTP Packet:
+  pkt.li = 0; //Leap Indicator : No warning
+  pkt.vn = 4; //Version Number : 4
+  pkt.mode = 3; //Client mode
+  pkt.stratum = 0; //Not relevant here
+  pkt.poll = 0; //Not significant as well
+  pkt.precision = 0; //Neither this one is
+
+  pkt.rootDelay = 0; //Or this one
+  pkt.rootDispersion = 0; //Or that one
+  pkt.refId = 0; //...
+
+  pkt.refTm_s = 0;
+  pkt.origTm_s = 0;
+  pkt.rxTm_s = 0;
+  pkt.txTm_s = htonl( NTP_TIMESTAMP_DELTA + time(NULL) ); //WARN: We are in LE format, network byte order is BE
+
+  pkt.refTm_f = pkt.origTm_f = pkt.rxTm_f = pkt.txTm_f = 0;
+
+  Endpoint outEndpoint;
+  
+  if( outEndpoint.set_address(host, port) < 0)
+  {
+    m_sock.close();
+    return NTP_DNS;
+  }
+  
+  //Set timeout, non-blocking and wait using select
+  int ret = m_sock.sendTo( outEndpoint, (char*)&pkt, sizeof(NTPPacket) );
+  if (ret < 0 )
+  {
+    ERR("Could not send packet");
+    m_sock.close();
+    return NTP_CONN;
+  }
+
+  //Read response
+  Endpoint inEndpoint;
+  // Set the inEndpoint address property
+  inEndpoint.set_address(outEndpoint.get_address(), 0);
+  DBG("Pong");
+  do
+  {
+    ret = m_sock.receiveFrom( inEndpoint, (char*)&pkt, sizeof(NTPPacket) ); //FIXME need a DNS Resolver to actually compare the incoming address with the DNS name
+    if(ret < 0)
+    {
+      ERR("Could not receive packet");
+      m_sock.close();
+      return NTP_CONN;
+    }
+  } while( strcmp(outEndpoint.get_address(), inEndpoint.get_address()) != 0 );
+
+  if(ret < sizeof(NTPPacket)) //TODO: Accept chunks
+  {
+    ERR("Receive packet size does not match");
+    m_sock.close();
+    return NTP_PRTCL;
+  }
+
+  if( pkt.stratum == 0)  //Kiss of death message : Not good !
+  {
+    ERR("Kissed to death!");
+    m_sock.close();
+    return NTP_PRTCL;
+  }
+
+  //Correct Endianness
+  pkt.refTm_s = ntohl( pkt.refTm_s );
+  pkt.refTm_f = ntohl( pkt.refTm_f );
+  pkt.origTm_s = ntohl( pkt.origTm_s );
+  pkt.origTm_f = ntohl( pkt.origTm_f );
+  pkt.rxTm_s = ntohl( pkt.rxTm_s );
+  pkt.rxTm_f = ntohl( pkt.rxTm_f );
+  pkt.txTm_s = ntohl( pkt.txTm_s );
+  pkt.txTm_f = ntohl( pkt.txTm_f );
+
+  //Compute offset, see RFC 4330 p.13
+  uint32_t destTm_s = (NTP_TIMESTAMP_DELTA + time(NULL));
+  int64_t offset = ( (int64_t)( pkt.rxTm_s - pkt.origTm_s ) + (int64_t) ( pkt.txTm_s - destTm_s ) ) / 2; //Avoid overflow
+  DBG("Sent @%ul", pkt.txTm_s);
+  DBG("Offset: %lld", offset);
+  //Set time accordingly
+  set_time( time(NULL) + offset );
+
+#ifdef __DEBUG__
+  ctTime = time(NULL);
+  DBG("Time is now (UTC): %s", ctime(&ctTime));
+#endif
+
+  m_sock.close();
+
+  return NTP_OK;
+}
+
+