You are viewing an older revision! See the latest version
Device Automation with Hexiwear Echo Dot
Team Members : Lai-Kuei Hsueh, Amanvir Singh Sidana, Rahul Dutta Choudhury
Overview¶
This project demonstrates the use of Hexiwear, Mbed and Amazon Echo Dot for IoT applications. The Echo Dot is used for selecting the device that needs to be controlled and Hexiwear is used for controlling the device. Three devices have been used namely, a music player, LED and motor.
Data Flow Diagram¶
Description¶
Communication between Echo Dot and MBED¶
Amazon Echo Dot is used to select the device to be controlled. First, we enter the Smart Device Skill App created by us and then we set the device to be controlled using voice commands. The Amazon Echo Server is configured to send a trigger signal to If This Then That (IFTTT) app's server. A python code which is running on a laptop continuously waits for signals from If This Then That server. When the trigger signal is received, it queries the required details from the server using HTTP GET POST functions. Then, according to the response, data is sent over serial to the mbed to set the device being controlled.
Communication between Hexiwear and MBED¶
Hexiwear's touch buttons and accelerometer have been used. When the user presses touch button, accelerometer readings are recorded and gesture value is computed. In this project, we use four gestures. Each gesture corresponds to a value which is uploaded to the wolksense cloud using Hexiwear App. For this Hexiwear App is installed on a smartphone and the phone is connected to Hexiwear. Also, the smartphone must be connected to the internet. The data is continuously monitored by a C# program running on a laptop. The C# program grabs the trigger input and sends it over serial to the mbed controller. The mbed then performs the required task according to the input received for the device which was set earlier using Echo dot.
Equipment¶
- mbed LPC1768
- Hexiwear and Docking Station
- Amazon Echo Dot
- RS232 to TTL Shifter
- VS 1053 MP3 Decoder
- microSD Socket
- Speaker
- RGB LED
- Power MOSFET
- Motor and Wheel
Wiring¶
Mbed | VS 1053 | microSD | Speaker | RGB LED | Power MOSFET | Motor | Ext Power Supply (5V) | RS232 Breakout |
---|---|---|---|---|---|---|---|---|
VOUT (3.3V) | VCC | VCC | VCC | |||||
GND | GND | GND | GND | JP2-2 GND | GND | GND | ||
p5 | DI | |||||||
p6 | DO | |||||||
p7 | SCK | |||||||
p9 | CS | |||||||
p11 | SI | |||||||
p12 | SO | |||||||
p13 | SCLK | |||||||
p14 | CS | |||||||
p18 | RESET | |||||||
p19 | DREQ | |||||||
p20 | BSYNC | |||||||
p21 | JP2-3 Control | |||||||
p22 | Blue | |||||||
p23 | Green | |||||||
p24 | Red | |||||||
p27 | TX | |||||||
p28 | RX | |||||||
LEFT | Positive | |||||||
GBUF | Negative | |||||||
JP2-1 RAW | 5V | |||||||
JP1-1 | - | |||||||
JP1-2 | + |
Video¶
Code¶
Hexiwear Code
#include "mbed.h" #include "Hexi_KW40Z.h" #include "Hexi_OLED_SSD1351.h" #include "OLED_types.h" #include "OpenSans_Font.h" #include "string.h" #include "FXAS21002.h" #include "FXOS8700.h" #define LED_ON 0 #define LED_OFF 1 #define DECLINATION -4.94 #define PI 3.14159 void StartHaptic(void); void StopHaptic(void const *n); void txTask(void); DigitalOut redLed(LED1,1); DigitalOut greenLed(LED2,1); DigitalOut blueLed(LED3,1); DigitalOut haptic(PTB9); Serial pc(USBTX, USBRX); /* Define timer for haptic feedback */ RtosTimer hapticTimer(StopHaptic, osTimerOnce); /* Instantiate the Hexi KW40Z Driver (UART TX, UART RX) */ KW40Z kw40z_device(PTE24, PTE25); /* Instantiate the SSD1351 OLED Driver */ SSD1351 oled(PTB22,PTB21,PTC13,PTB20,PTE6, PTD15); /* (MOSI,SCLK,POWER,CS,RST,DC) */ /* Instantiate the gyro */ FXAS21002 gyro(PTC11,PTC10); FXOS8700 accel(PTC11, PTC10); FXOS8700 mag(PTC11, PTC10); // Storage for the data from the sensor float gyro_data[3]; float gyro_rms=0.0; float accel_data[3]; float accel_rms=0.0; float mag_data[3]; float mag_rms=0.0; float roll,pitch,heading; float new_roll,new_pitch,new_heading; int del_roll,del_pitch; volatile bool flag=false; Timeout timeout; /*Create a Thread to handle sending BLE Sensor Data */ Thread txThread; /* Text Buffer */ char text[20]; uint8_t battery = 100; uint8_t light = 0; uint16_t humidity = 4500; uint16_t temperature = 2000; uint16_t pressure = 9000; uint16_t x = 0; uint16_t y = 5000; uint16_t z = 10000; /*Get the new value of roll and pitch*/ void get_new_value(){ gyro.acquire_gyro_data_dps(gyro_data); accel.acquire_accel_data_g(accel_data); new_roll = atan2(accel_data[2],accel_data[1]); new_pitch = atan2(-accel_data[0],sqrt(accel_data[1]*accel_data[1] + accel_data[2]*accel_data[2])); new_pitch *= 180.0 / PI; new_roll *= 180.0 / PI; new_roll+=180; } /*Click the button to start it*/ void Start_Detect(){ if(flag==true) return; flag=true; } /****************************Call Back Functions*******************************/ void ButtonRight(void) { StartHaptic(); kw40z_device.ToggleAdvertisementMode(); } void ButtonLeft(void) { StartHaptic(); kw40z_device.ToggleAdvertisementMode(); } void PassKey(void) { StartHaptic(); strcpy((char *) text,"PAIR CODE"); oled.TextBox((uint8_t *)text,0,25,95,18); /* Display Bond Pass Key in a 95px by 18px textbox at x=0,y=40 */ sprintf(text,"%d", kw40z_device.GetPassKey()); oled.TextBox((uint8_t *)text,0,40,95,18); } /***********************End of Call Back Functions*****************************/ /********************************Main******************************************/ int main() { /* Register callbacks to application functions */ kw40z_device.attach_buttonLeft(&ButtonLeft); kw40z_device.attach_buttonRight(&ButtonRight); kw40z_device.attach_passkey(&PassKey); kw40z_device.attach_buttonUp(&Start_Detect);//Click the button to start it gyro.gyro_config(); accel.accel_config(); mag.mag_config(); /* Turn on the backlight of the OLED Display */ oled.DimScreenON(); /* Fills the screen with solid black */ oled.FillScreen(COLOR_BLACK); /* Get OLED Class Default Text Properties */ oled_text_properties_t textProperties = {0}; oled.GetTextProperties(&textProperties); /* Change font color to Blue */ textProperties.fontColor = COLOR_BLUE; oled.SetTextProperties(&textProperties); /* Display Bluetooth Label at x=17,y=65 */ strcpy((char *) text,"BLUETOOTH"); oled.Label((uint8_t *)text,17,65); /* Change font color to white */ textProperties.fontColor = COLOR_WHITE; textProperties.alignParam = OLED_TEXT_ALIGN_CENTER; oled.SetTextProperties(&textProperties); /* Display Label at x=22,y=80 */ strcpy((char *) text,"Tap Below"); oled.Label((uint8_t *)text,22,80); uint8_t prevLinkState = 0; uint8_t currLinkState = 0; txThread.start(txTask); /*Start transmitting Sensor Tag Data */ while (true) { blueLed = !kw40z_device.GetAdvertisementMode(); /*Indicate BLE Advertisment Mode*/ Thread::wait(50); } } /******************************End of Main*************************************/ /* txTask() transmits the sensor data */ void txTask(void){ while (true) { if(flag){ kw40z_device.SendSetApplicationMode(GUI_CURRENT_APP_SENSOR_TAG); gyro.acquire_gyro_data_dps(gyro_data); accel.acquire_accel_data_g(accel_data); roll = atan2(accel_data[2],accel_data[1]); pitch = atan2(-accel_data[0],sqrt(accel_data[1]*accel_data[1] + accel_data[2]*accel_data[2])); pitch *= 180.0 / PI; roll *= 180.0 / PI; roll+=180; Thread::wait(1000); get_new_value(); float cmd; if(abs(new_roll-roll)>abs(new_pitch-pitch)){ if(new_roll-roll>30) cmd=100; else if(new_roll-roll<-30) cmd=200; else cmd=0; }else if(abs(new_roll-roll)<abs(new_pitch-pitch)){ if(new_pitch-pitch>10) cmd=300; else if(new_pitch-pitch<-10) cmd=400; else cmd=0; }else cmd=0; pc.printf("cmd=%f",cmd); kw40z_device.SendTemperature(cmd); Thread::wait(6000);//Make some delay to wait the data upload to the server kw40z_device.SendTemperature(0);//send 0 to the server flag=false; } Thread::wait(500); } } void StartHaptic(void) { hapticTimer.start(50); haptic = 1; } void StopHaptic(void const *n) { haptic = 0; hapticTimer.stop(); }
Mbed Controller Code
#include "mbed.h" #include "VS1002.h" using namespace mbed; /*For Change Task*/ #define MUSIC_PLAYER 's' #define LED 'l' #define MOTOR 'm' volatile char task=MUSIC_PLAYER; /*Serial COM port*/ Serial pc1(USBTX, USBRX); Serial taskport(p28,p27); /*For Music Control*/ VS1002 mp3(p5, p6, p7, p8,"sd",p11, p12 ,p13, p14, p18, p19, p20, p15); char *song_name[6]={"Escape","Summer of 69","Monster", "Leave out all the rest","Kings","Interstellar Docking"}; //Array of song names entered manually int new_song_number=1; //Variable to store the Song Number int volume_set=-5; //Variable to store the Volume int previous_volume; //Variable to store the volume when muted bool pause=false; //Variable to store the status of Pause button bool mute=false; //Variable to store the status of mute button #define NEXT 7 #define PREVIOUS 8 #define PP 9 #define V_DOWN 10 #define V_UP 11 #define MUTE_UNMUTE 12 /*For LED Control*/ PwmOut red(p24); PwmOut green(p23); PwmOut blue(p22); /*For Motor Control*/ PwmOut fanSwitch(p21); /*For Debugging*/ DigitalOut led1(LED1); DigitalOut led2(LED2); /*For Change Task*/ void changeTask() { if(taskport.readable()){ char c=taskport.getc(); pc1.putc(c); task=c; } } /*For Do Different Function*/ void changeAction(){ if(pc1.readable()){ const char arg0=pc1.getc(); if(task==LED){ switch(arg0) { case '1': { red = 1.0; green = 1.0; blue = 1.0; } break; case '2': { red = 0.75; green = 0.75; blue = 0.75; } break; case '3': { red = 0.5; green = 0.5; blue = 0.5; } break; case '4': { red = 0.25; green = 0.25; blue = 0.25; } break; default : break; } led1=!led1; }else if(task==MOTOR){ switch(arg0) { case '1': { fanSwitch = 1.0; //Full Speed } break; case '2': { fanSwitch = 0.3; // Medium Speed } break; case '3': { fanSwitch = 0; // Low Speed } break; default : break; } led2=!led2; }else if(task==MUSIC_PLAYER) { int key_code=0; switch(arg0){ case '1': key_code=V_UP; break; case '2': key_code=V_DOWN; break; case '3': key_code=PP; break; case '4': key_code=NEXT; break; case '5': key_code=PREVIOUS; break; case '6': key_code=MUTE_UNMUTE; break; default : key_code=0; break; } switch(key_code) // Different cases depending on key press { case NEXT: pc1.printf("next\r\n"); new_song_number+=1; // Next song if(new_song_number==7) new_song_number=1; break; case PREVIOUS: pc1.printf("previous\r\n"); new_song_number-=1; // Previous Song if(new_song_number==0) new_song_number=6; break; case PP: pc1.printf("pp\r\n"); pause=!pause; // Pause/Play button break; case V_UP: pc1.printf("v_up\r\n"); volume_set+=10; // Volume Up if(volume_set>=0) volume_set=0; break; case V_DOWN: pc1.printf("v_down\r\n"); volume_set-=10; //Volume Down if(volume_set<-55) volume_set=-55; break; case MUTE_UNMUTE: pc1.printf("mute_unmute\r\n"); mute=!mute; //Mute/Unmute if(mute) { previous_volume=volume_set; // Attenuation of -55 db is small enough to not hear anything volume_set=-55; } else { volume_set=previous_volume; } break; default: ;//pc.cls(); pc1.printf("error"); // exit on error exit(1); } /* Print to LCD the status of Song */ //pc.cls(); if(pause) pc1.printf("Paused "); if(mute) pc1.printf("Muted"); if(!mute && !pause) pc1.printf("Playing"); pc1.printf("\r\n %d %s",new_song_number,song_name[new_song_number-1]); } } } int main () { led1=1; led2=1; pc1.printf("hello\r\n"); mp3._RST = 1; mp3.cs_high(); //chip disabled mp3.sci_initialise(); //initialise MBED mp3.sci_write(0x00,(SM_SDINEW+SM_STREAM+SM_DIFF)); mp3.sci_write(0x03, 0x9800); mp3.sdi_initialise(); pc1.attach(&changeAction, Serial::RxIrq);//Serial interrupt for function code taskport.attach(&changeTask, Serial::RxIrq);// Serial interrupt for Task code while(1) { mp3.play_song(new_song_number); } }
C# code
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading; using System.Threading.Tasks; using System.Net; using System.Net.Http; using System.IO.Ports; namespace final { class Program { static void Main(string[] args) { SerialPort port = new SerialPort("COM33",9600, Parity.None, 8, StopBits.One); if (!port.IsOpen) port.Open(); char[] temp_value,value; temp_value = new char[1]; value = new char[1]; temp_value[0] = ' '; value[0] = ' '; while (true) { string url = "https://api.wolksense.com/api/v4/points"; var wb = new WebClient(); wb.Headers[HttpRequestHeader.ContentType] = "application/json"; wb.Headers[HttpRequestHeader.Authorization] = "re6m846kr42glcvvvfe67psjskrgtocperd382f1e66qapggjinph3st99j52pch"; //Used Wolksense Web API and Fiddler to get the Token var response = wb.DownloadString(url); string s = response; int startIndex = s.IndexOf("\"readingType\":\"T\""); int length = 35; s = s.Substring(startIndex, length); temp_value[0] = value[0]; value[0] = s[s.Length - 1]; if (temp_value[0] != value[0]) { string output = new string(value); port.Write(output); System.Console.WriteLine(s); System.Console.WriteLine(value); } Thread.Sleep(1000); } } } }
Python Code
from flask import Flask, render_template, request, url_for, jsonify,current_app app = Flask(__name__) import serial @app.route('/') def hello_world(): return 'Hello, Worlds!' @app.route('/', methods=['GET', 'POST']) def check(): if request.method == 'POST': print request.data serdev = '/dev/tty.usbserial-AI03QVUR' s = serial.Serial(serdev) s.write(request.data) s.close() return current_app.send_static_file('index.html') else: return 'Good,bye'
Programming Echo Dot¶
This section explains how to set up an account on AWS, and get started with the coding the lambda function, intent schema and sample utterances required for this project. Following these exact steps should give the correct results.
Setting up Account on AWS¶
- If you do not already have an account on AWS, go to Amazon Web Services and create an account.
- Log in to the AWS Management Console and navigate to AWS Lambda.
- Click the region drop-down in the upper-right corner of the console and select either US East (N. Virginia) or EU (Ireland). Note: Lambda functions for Alexa skills must be hosted in either the US East (N. Virginia) or EU (Ireland) region.
- If you have no Lambda functions yet, click Get Started Now. Otherwise, click Create a Lambda Function.
- To start with sample code in Node.js or Python, select one of the Alexa Skills Kit blueprints. For Node.js select a blank blueprint.
- When prompted to configure triggers, click the box and select Alexa Skills Kit, then click Next.
- The code in the 'Lambda Code' section at the end of the page should be used as the the the inline code:
- Insert the following code in "Lambda Function Code" and "Configure Test Event" respectively:
- Click Create function to save your new function.
Test Event
{ "session": { "sessionId": "amzn1.echo-api.session.[unique-value-here]", "application": { "applicationId": "amzn1.ask.skill.[unique-value-here]" }, "attributes": {}, "user": { "userId": "amzn1.ask.account.[unique-value-here]" }, "new": true }, "request": { "type": "IntentRequest", "requestId": "amzn1.echo-api.request.[unique-value-here]", "locale": "en-US", "timestamp": "2016-12-12T02:28:38Z", "intent": { "name": "LEDControlIntent" } }, "version": "1.0" }
Lambda Function
'use strict'; var https = require('https'); var http = require('http'); const WundergroundApiKey = '15b2686aeaa92f56', WundergroundCity = 'Atlanta', WundergroundState = 'GA'; // Route the incoming request based on type (LaunchRequest, IntentRequest, // etc.) The JSON body of the request is provided in the event parameter. exports.handler = function (event, context) { try { console.log("event.session.application.applicationId=" + event.session.application.applicationId); /** * Uncomment this if statement and populate with your skill's application ID to * prevent someone else from configuring a skill that sends requests to this function. */ // if (event.session.application.applicationId !== "amzn1.echo-sdk-ams.app.05aecccb3-1461-48fb-a008-822ddrt6b516") { // context.fail("Invalid Application ID"); // } if (event.session.new) { onSessionStarted({requestId: event.request.requestId}, event.session); } if (event.request.type === "LaunchRequest") { onLaunch(event.request, event.session, function callback(sessionAttributes, speechletResponse) { context.succeed(buildResponse(sessionAttributes, speechletResponse)); }); } else if (event.request.type === "IntentRequest") { onIntent(event.request, event.session, function callback(sessionAttributes, speechletResponse) { context.succeed(buildResponse(sessionAttributes, speechletResponse)); }); } else if (event.request.type === "SessionEndedRequest") { onSessionEnded(event.request, event.session); context.succeed(); } } catch (e) { context.fail("Exception: " + e); } }; /** * Called when the session starts. */ function onSessionStarted(sessionStartedRequest, session) { console.log("onSessionStarted requestId=" + sessionStartedRequest.requestId + ", sessionId=" + session.sessionId); // add any session init logic here } /** * Called when the user invokes the skill without specifying what they want. */ function onLaunch(launchRequest, session, callback) { console.log("onLaunch requestId=" + launchRequest.requestId + ", sessionId=" + session.sessionId); var cardTitle = "Hello, World!" var speechOutput = "Hello Master, what is your command?" callback(session.attributes, buildSpeechletResponse(cardTitle, speechOutput, "", true)); } /** * Called when the user specifies an intent for this skill. */ function onIntent(intentRequest, session, callback) { console.log("onIntent requestId=" + intentRequest.requestId + ", sessionId=" + session.sessionId); var intent = intentRequest.intent, intentName = intentRequest.intent.name; // dispatch custom intents to handlers here if (intentName == 'LEDControlIntent') { LEDControl(intent, session, callback); } else if (intentName == 'MotorControlIntent') { MotorControl(intent, session, callback); } else if (intentName == 'MusicControlIntent') { MusicControl(intent, session, callback); } else { throw "Invalid intent"; } } /** * Called when the user ends the session. * Is not called when the skill returns shouldEndSession=true. */ function onSessionEnded(sessionEndedRequest, session) { console.log("onSessionEnded requestId=" + sessionEndedRequest.requestId + ", sessionId=" + session.sessionId); // Add any cleanup logic here } function LEDControl(intent, session, callback) { var body=''; var jsonObject2 = {'value1':'l'} var optionspost = { host: 'maker.ifttt.com', path: '/trigger/MusicPlayer/with/key/bGtHbgue1Hvx7hYsHjeQXQ', method: 'POST', headers: { 'Content-Type': 'application/json', } }; var reqPost = https.request(optionspost, function(res) { console.log("statusCode: ", res.statusCode); res.on('data', function (chunk) { body += chunk; }); // context.succeed('Thanks Marcus'); }); reqPost.write(JSON.stringify(jsonObject2)); reqPost.end(); callback(session.attributes, buildSpeechletResponseWithoutCard("LED control on", "", "true")); console.log(new Date()); } function MusicControl(intent, session, callback) { var body=''; var jsonObject2 = {'value1':'s'} var optionspost = { host: 'maker.ifttt.com', path: '/trigger/MusicPlayer/with/key/bGtHbgue1Hvx7hYsHjeQXQ', method: 'POST', headers: { 'Content-Type': 'application/json', } }; var reqPost = https.request(optionspost, function(res) { console.log("statusCode: ", res.statusCode); res.on('data', function (chunk) { body += chunk; }); // context.succeed('Thanks Marcus'); }); reqPost.write(JSON.stringify(jsonObject2)); reqPost.end(); callback(session.attributes, buildSpeechletResponseWithoutCard("Music control on", "", "true")); console.log(new Date()); } function MotorControl(intent, session, callback) { var body=''; var jsonObject2 = {'value1':'m'} var optionspost = { host: 'maker.ifttt.com', path: '/trigger/MusicPlayer/with/key/bGtHbgue1Hvx7hYsHjeQXQ', method: 'POST', headers: { 'Content-Type': 'application/json', } }; var reqPost = https.request(optionspost, function(res) { console.log("statusCode: ", res.statusCode); res.on('data', function (chunk) { body += chunk; }); // context.succeed('Thanks Marcus'); }); reqPost.write(JSON.stringify(jsonObject2)); reqPost.end(); callback(session.attributes, buildSpeechletResponseWithoutCard("Motor control on", "", "true")); console.log(new Date()); } // ------- Helper functions to build responses ------- function buildSpeechletResponse(title, output, repromptText, shouldEndSession) { return { outputSpeech: { type: "PlainText", text: output }, card: { type: "Simple", title: title, content: output }, reprompt: { outputSpeech: { type: "PlainText", text: repromptText } }, shouldEndSession: shouldEndSession }; } function buildSpeechletResponseWithoutCard(output, repromptText, shouldEndSession) { return { outputSpeech: { type: "PlainText", text: output }, reprompt: { outputSpeech: { type: "PlainText", text: repromptText } }, shouldEndSession: shouldEndSession }; } function buildResponse(sessionAttributes, speechletResponse) { return { version: "1.0", sessionAttributes: sessionAttributes, response: speechletResponse }; }
Setting up Intent Schema and Sample Utterances¶
- Sign into the Amazon Developer portal to set up the intent schema and sample utterances for your lambda functuon code
- Navigate to Alexa -> Alexa Skills Kit -> Add New Skill and follow the instructions
- Following is the intent schema for this project
Intent Schema
{ "intents": [ { "intent": "MusicControlIntent" }, { "intent": "MotorControlIntent" }, { "intent": "LEDControlIntent" } ] }
The 3 sample utterances given below were used for this project. However, adding new utterances with similar meanings will give a better range of commands to the user and make the code more robust
Sample Utterances
MusicControlIntent Open Music Control MotorControlIntent Open Speed Control LEDControlIntent Open Lights Control
- In configuration, enter the ARN of your lambda function and select North America
- The code can be checked in the Test section by typing the configured sample utterances.