Bob Stone / Mbed 2 deprecated BlinkTalk Featured

A portable, hands-free, zero-effort blink-controlled speech system, inspired by the voice system of Stephen Hawking. Uses mbed, Neurosky Mindwave Mobile headset, BlueSMiRF modem and Emic2 speech board.

Dependencies:   SPI_TFT TFT_fonts mbed

/media/uploads/RorschachUK/_scaled_p6150001.jpg

Ever since I got the Parallax / Grand Idea Studio Emic2 sound board I've been wanting to find some way to mimic the hands-free typing / speech system used by Stephen Hawking. And when I figured out how to get the Neurosky Mindwave Mobile headset to identify blinks, I realised I could use blinking as the single minimal-effort input into the system to control a keyboard menuing system.

/media/uploads/RorschachUK/_scaled_p6150002.jpg

The system sweeps through rows until I blink, then sweeps along the row to select a character or action with another blink. If I let it go past the end, it cancels and carries on sweeping rows. Since we're also getting eSense (attention/meditation) and brainwave data from the headset, I've included small barcharts at the top of the screen to show those - perhaps concentrating on typing will show up in the concentration levels.

See my other projects for more on how to setup the BlueSMiRF for auto-connecting to the NeuroSky headset.

Hardware:

  • Neurosky Mindwave Mobile headset - sends serial data packets over BlueTooth
  • BlueSMiRF Silver Mate - receives data over BlueTooth from headset and relays over serial to mbed
  • Parallax / Grand Idea Studio Emic2 speech synthesis - speaks text sent over serial from the mbed
  • MikroElektronika TFT Proto - 320x240 TFT display with HX8374 controller driven by SPI
  • Sparkfun level shifter - translates Emic2 5V serial to 3.3V serial for mbed

/media/uploads/RorschachUK/blinktalkfritz.jpg

main.cpp@0:e2daaf858e13, 2013-06-15 (annotated)

Committer:
RorschachUK
Date:
Sat Jun 15 19:43:44 2013 +0000
Revision:
0:e2daaf858e13
First commit of BlinkTalk

Who changed what in which revision?

UserRevisionLine numberNew contents of line
RorschachUK 0:e2daaf858e13 1 /* BlinkTalk - Bob Stone June 2013
RorschachUK 0:e2daaf858e13 2 * Hands-free control of a speech synthesis system by blinking.
RorschachUK 0:e2daaf858e13 3 *
RorschachUK 0:e2daaf858e13 4 * This project implements a proof of concept speech system actuated by a single input
RorschachUK 0:e2daaf858e13 5 * - this could be a button, but for a more fun application I am going to use blinks as
RorschachUK 0:e2daaf858e13 6 * detected by an EEG headset, with a row & column time-sweep keyboard, inspired by
RorschachUK 0:e2daaf858e13 7 * (but not the same as) the speech system used by Prof Stephen Hawking, mainly because the
RorschachUK 0:e2daaf858e13 8 * Emic2 voice board includes the DECTalk 'PerfectPaul' voice familiar to all as Hawking.
RorschachUK 0:e2daaf858e13 9 *
RorschachUK 0:e2daaf858e13 10 * In Hawking's actual system, an infrared sensor mounted on his glasses detects a movement
RorschachUK 0:e2daaf858e13 11 * of his cheek muscle, selecting letters from a keyboard sweeping rows and columns, with a
RorschachUK 0:e2daaf858e13 12 * predictive text system handling word completion and suggested follow-on words. In
RorschachUK 0:e2daaf858e13 13 * our version we will detect a blink using a Neurosky Mindwave Mobile headset and use it to
RorschachUK 0:e2daaf858e13 14 * stop and select a cursor sweeping a keyboard grid.
RorschachUK 0:e2daaf858e13 15 *
RorschachUK 0:e2daaf858e13 16 * Hardware:
RorschachUK 0:e2daaf858e13 17 * Neurosky Mindwave Mobile headset - sends serial data packets over BlueTooth
RorschachUK 0:e2daaf858e13 18 * BlueSMiRF Silver Mate - receives data over BlueTooth from headset and relays over serial to mbed
RorschachUK 0:e2daaf858e13 19 * Parallax / Grand Idea Studio Emic2 speech synthesis - speaks text sent over serial from the mbed
RorschachUK 0:e2daaf858e13 20 * MikroElektronika TFT Proto - 320x240 TFT display with HX8374 controller driven by SPI
RorschachUK 0:e2daaf858e13 21 * Sparkfun level shifter - translates Emic2 5V serial to 3.3V serial for mbed
RorschachUK 0:e2daaf858e13 22 *
RorschachUK 0:e2daaf858e13 23 * Connections:
RorschachUK 0:e2daaf858e13 24 * mbed BlueSMiRF LevelShift Emic2 Speaker TFT-PROTO
RorschachUK 0:e2daaf858e13 25 * GND GND GND/GND GND GND
RorschachUK 0:e2daaf858e13 26 * VOUT(3.3V) VCC LV 3.3V
RorschachUK 0:e2daaf858e13 27 * VU (5V) HV 5V
RorschachUK 0:e2daaf858e13 28 * p9 (TX) RX-I
RorschachUK 0:e2daaf858e13 29 * p10(RX) TX-O
RorschachUK 0:e2daaf858e13 30 * p11(MOSI) SDI
RorschachUK 0:e2daaf858e13 31 * p12(MISO) SDO
RorschachUK 0:e2daaf858e13 32 * p13(SCLK) SCL
RorschachUK 0:e2daaf858e13 33 * p14 CS
RorschachUK 0:e2daaf858e13 34 * p15 RST
RorschachUK 0:e2daaf858e13 35 * p27(RX) LV:RXO
RorschachUK 0:e2daaf858e13 36 * p28(TX) LV:TXI
RorschachUK 0:e2daaf858e13 37 * HV:RXI SOUT
RorschachUK 0:e2daaf858e13 38 * HV:TXO SN
RorschachUK 0:e2daaf858e13 39 * SP+ +
RorschachUK 0:e2daaf858e13 40 * SP- -
RorschachUK 0:e2daaf858e13 41 *
RorschachUK 0:e2daaf858e13 42 * Borrows from http://developer.neurosky.com/docs/doku.php?id=mindwave_mobile_and_arduino
RorschachUK 0:e2daaf858e13 43 * Display library from Peter Drescher: http://mbed.org/cookbook/SPI-driven-QVGA-TFT
RorschachUK 0:e2daaf858e13 44 */
RorschachUK 0:e2daaf858e13 45 #include "mbed.h"
RorschachUK 0:e2daaf858e13 46 #include "SPI_TFT.h"
RorschachUK 0:e2daaf858e13 47 #include "Arial12x12.h"
RorschachUK 0:e2daaf858e13 48 #include "Arial28x28.h"
RorschachUK 0:e2daaf858e13 49 #include <string>
RorschachUK 0:e2daaf858e13 50
RorschachUK 0:e2daaf858e13 51 //Peripherals
RorschachUK 0:e2daaf858e13 52 Serial blueSmirf(p9, p10); //bluetooth comms (TX, RX)
RorschachUK 0:e2daaf858e13 53 Serial voice(p28,p27); //Emic2 text to speech voice synth
RorschachUK 0:e2daaf858e13 54 SPI_TFT screen(p11, p12, p13, p14, p15, "TFT"); //display
RorschachUK 0:e2daaf858e13 55
RorschachUK 0:e2daaf858e13 56 //control variables
RorschachUK 0:e2daaf858e13 57 int quality=0;
RorschachUK 0:e2daaf858e13 58 bool connected=false;
RorschachUK 0:e2daaf858e13 59 bool started=false;
RorschachUK 0:e2daaf858e13 60 int row=-1;
RorschachUK 0:e2daaf858e13 61 int col=-1;
RorschachUK 0:e2daaf858e13 62 Timer initialDelay;
RorschachUK 0:e2daaf858e13 63
RorschachUK 0:e2daaf858e13 64 //Keyboard - rows to display
RorschachUK 0:e2daaf858e13 65 #define ROWS 5
RorschachUK 0:e2daaf858e13 66 #define COLS 10
RorschachUK 0:e2daaf858e13 67 //keys or text to display
RorschachUK 0:e2daaf858e13 68 string keys[ROWS][COLS] = {
RorschachUK 0:e2daaf858e13 69 {"1","2","3","4","5","6","7","8","9","0"},
RorschachUK 0:e2daaf858e13 70 {"Q","W","E","R","T","Y","U","I","O","P"},
RorschachUK 0:e2daaf858e13 71 {"A","S","D","F","G","H","J","K","L","!"},
RorschachUK 0:e2daaf858e13 72 {"Z","X","C","V","B","N","M",",",".","?"},
RorschachUK 0:e2daaf858e13 73 {"Space","\"SAY!\"","Delete"}
RorschachUK 0:e2daaf858e13 74 };
RorschachUK 0:e2daaf858e13 75 //positions to draw a line at the end of a column
RorschachUK 0:e2daaf858e13 76 int keysColPos[ROWS][COLS] = {
RorschachUK 0:e2daaf858e13 77 {36,67,98,129,160,191,222,253,284,315},
RorschachUK 0:e2daaf858e13 78 {36,67,98,129,160,191,222,253,284,315},
RorschachUK 0:e2daaf858e13 79 {36,67,98,129,160,191,222,253,284,315},
RorschachUK 0:e2daaf858e13 80 {36,67,98,129,160,191,222,253,284,315},
RorschachUK 0:e2daaf858e13 81 {100,210,315}
RorschachUK 0:e2daaf858e13 82 };
RorschachUK 0:e2daaf858e13 83 //number of elements in each row
RorschachUK 0:e2daaf858e13 84 int keysRowSize[ROWS] = {10, 10, 10, 10, 3};
RorschachUK 0:e2daaf858e13 85
RorschachUK 0:e2daaf858e13 86 //text being typed
RorschachUK 0:e2daaf858e13 87 string typingBuffer("");
RorschachUK 0:e2daaf858e13 88
RorschachUK 0:e2daaf858e13 89 //*****************************
RorschachUK 0:e2daaf858e13 90 //User routines to process data
RorschachUK 0:e2daaf858e13 91 //*****************************
RorschachUK 0:e2daaf858e13 92
RorschachUK 0:e2daaf858e13 93 /** Say some text out loud
RorschachUK 0:e2daaf858e13 94 *
RorschachUK 0:e2daaf858e13 95 * @param text The text to say out loud
RorschachUK 0:e2daaf858e13 96 */
RorschachUK 0:e2daaf858e13 97 void say(string text)
RorschachUK 0:e2daaf858e13 98 {
RorschachUK 0:e2daaf858e13 99 voice.printf("S%s\n", text);//send command to speak to the Emic2
RorschachUK 0:e2daaf858e13 100 while(!voice.readable()) //wait for Emic2 to return ':'
RorschachUK 0:e2daaf858e13 101 ; //do nothing whilst it's still speaking
RorschachUK 0:e2daaf858e13 102 voice.getc(); //pop the ':' response from the stream
RorschachUK 0:e2daaf858e13 103 }
RorschachUK 0:e2daaf858e13 104
RorschachUK 0:e2daaf858e13 105 /** Maps a value from one scale to another
RorschachUK 0:e2daaf858e13 106 *
RorschachUK 0:e2daaf858e13 107 * @param value Value we're trying to scale
RorschachUK 0:e2daaf858e13 108 * @param min,max The range that value came from
RorschachUK 0:e2daaf858e13 109 * @param newMin,newMax The new range we're scaling value into
RorschachUK 0:e2daaf858e13 110 * @returns value mapped into new scale
RorschachUK 0:e2daaf858e13 111 */
RorschachUK 0:e2daaf858e13 112 int map(int value, int min, int max, int newMin, int newMax)
RorschachUK 0:e2daaf858e13 113 {
RorschachUK 0:e2daaf858e13 114 if (min==max)
RorschachUK 0:e2daaf858e13 115 return newMax;
RorschachUK 0:e2daaf858e13 116 else
RorschachUK 0:e2daaf858e13 117 return newMin + (newMax-newMin) * (value-min) / (max-min);
RorschachUK 0:e2daaf858e13 118 }
RorschachUK 0:e2daaf858e13 119
RorschachUK 0:e2daaf858e13 120 /** Returns a 16-bit RGB565 colour from three 8-bit component values.
RorschachUK 0:e2daaf858e13 121 *
RorschachUK 0:e2daaf858e13 122 * @param red,green,blue primary colour channel values expressed as 0-255 each
RorschachUK 0:e2daaf858e13 123 * @returns 16-bit RGB565 colour constructed as RRRRRGGGGGGBBBBB
RorschachUK 0:e2daaf858e13 124 */
RorschachUK 0:e2daaf858e13 125 int RGBColour(int red, int green, int blue)
RorschachUK 0:e2daaf858e13 126 {
RorschachUK 0:e2daaf858e13 127 //take most-significant parts of red, green and blue and bit-shift into RGB565 positions
RorschachUK 0:e2daaf858e13 128 return ((red & 0xf8) << 8) | ((green & 0xfc) << 3) | ((blue & 0xf8) >> 3);
RorschachUK 0:e2daaf858e13 129 }
RorschachUK 0:e2daaf858e13 130
RorschachUK 0:e2daaf858e13 131 /** Returns a colour mapped on a gradient from one colour to another.
RorschachUK 0:e2daaf858e13 132 *
RorschachUK 0:e2daaf858e13 133 * @param value Value we're trying to pick a colour for
RorschachUK 0:e2daaf858e13 134 * @param min,max Scale that value belongs in
RorschachUK 0:e2daaf858e13 135 * @param minColour,maxColour start and end colours of the gradient we're choosing from (16-bit RGB565)
RorschachUK 0:e2daaf858e13 136 * @returns colour that's as far along the gradient from minColour to maxColour as value is between min and max (16-bit RGB565)
RorschachUK 0:e2daaf858e13 137 */
RorschachUK 0:e2daaf858e13 138 int getMappedColour(int value, int min, int max, int minColour, int maxColour)
RorschachUK 0:e2daaf858e13 139 {
RorschachUK 0:e2daaf858e13 140 // TFT screen colours are 16-bit RGB565 i.e. RRRRRGGGGGGBBBBB
RorschachUK 0:e2daaf858e13 141 int minRed = (minColour & 0xf800) >> 11; //bitmask for 5 bits red
RorschachUK 0:e2daaf858e13 142 int maxRed = (maxColour & 0xf800) >> 11;
RorschachUK 0:e2daaf858e13 143 int minGreen = (minColour & 0x7e0) >> 5; //bitmask for 6 bits green
RorschachUK 0:e2daaf858e13 144 int maxGreen = (maxColour & 0x7e0) >> 5;
RorschachUK 0:e2daaf858e13 145 int minBlue = minColour & 0x1f; // bitmask for 5 bits blue
RorschachUK 0:e2daaf858e13 146 int maxBlue = maxColour & 0x1f;
RorschachUK 0:e2daaf858e13 147 int valRed = map(value, min, max, minRed, maxRed);
RorschachUK 0:e2daaf858e13 148 int valGreen = map(value, min, max, minGreen, maxGreen);
RorschachUK 0:e2daaf858e13 149 int valBlue = map(value, min, max, minBlue, maxBlue);
RorschachUK 0:e2daaf858e13 150 int valColour = ((valRed & 0x1F) << 11) | ((valGreen & 0x3F) << 5) | (valBlue & 0x1F);
RorschachUK 0:e2daaf858e13 151 return valColour;
RorschachUK 0:e2daaf858e13 152 }
RorschachUK 0:e2daaf858e13 153
RorschachUK 0:e2daaf858e13 154 /** Displays a bar graph showing 'value' on a scale 'min' to 'max', where coords (x0,y0) are at 'min' and (x1,y1) are at 'max'.
RorschachUK 0:e2daaf858e13 155 *
RorschachUK 0:e2daaf858e13 156 * @param x0,y0 coordinates of the 'min' end of the bargraph
RorschachUK 0:e2daaf858e13 157 * @param x1,y1 coordinates of the 'max' end of the bargraph
RorschachUK 0:e2daaf858e13 158 * @param isHorizontal If true, bar graph will be drawn with horizontal bars
RorschachUK 0:e2daaf858e13 159 * @param value Value of the bar, with bars drawn from min up to value, remaining 'backColour' from there to max
RorschachUK 0:e2daaf858e13 160 * @param min,max Scale of the bar graph that value should be found within
RorschachUK 0:e2daaf858e13 161 * @param minColour,maxColour colours at the min and max ends of the bar, drawn in a gradient between the two (16-bit RGB565)
RorschachUK 0:e2daaf858e13 162 * @param backColour background colour of the bar graph (16-bit RGB565)
RorschachUK 0:e2daaf858e13 163 */
RorschachUK 0:e2daaf858e13 164 void displayBarGraph(int x0, int y0, int x1, int y1, bool isHorizontal, int value, int min, int max, int minColour, int maxColour, int backColour)
RorschachUK 0:e2daaf858e13 165 {
RorschachUK 0:e2daaf858e13 166 int valColour;
RorschachUK 0:e2daaf858e13 167 if (isHorizontal) {
RorschachUK 0:e2daaf858e13 168 if (x1>x0) {
RorschachUK 0:e2daaf858e13 169 for (int i = x0; i < x1; i+=5) {
RorschachUK 0:e2daaf858e13 170 if (map(i, x0, x1, min, max) > value)
RorschachUK 0:e2daaf858e13 171 valColour = backColour;
RorschachUK 0:e2daaf858e13 172 else
RorschachUK 0:e2daaf858e13 173 valColour = getMappedColour(i, x0, x1, minColour, maxColour);
RorschachUK 0:e2daaf858e13 174 screen.fillrect(i, y0, i+3, y1, valColour);
RorschachUK 0:e2daaf858e13 175 }
RorschachUK 0:e2daaf858e13 176 } else {
RorschachUK 0:e2daaf858e13 177 for (int i = x1; i < x0; i+=5) {
RorschachUK 0:e2daaf858e13 178 if (map(i, x0, x1, min, max) > value)
RorschachUK 0:e2daaf858e13 179 valColour = backColour;
RorschachUK 0:e2daaf858e13 180 else
RorschachUK 0:e2daaf858e13 181 valColour = getMappedColour(i, x0, x1, minColour, maxColour);
RorschachUK 0:e2daaf858e13 182 screen.fillrect(i-3, y0, i, y1, valColour);
RorschachUK 0:e2daaf858e13 183 }
RorschachUK 0:e2daaf858e13 184 }
RorschachUK 0:e2daaf858e13 185 } else {
RorschachUK 0:e2daaf858e13 186 if (y1>y0) {
RorschachUK 0:e2daaf858e13 187 for (int i = y0; i < y1; i+=5) {
RorschachUK 0:e2daaf858e13 188 if (map(i, y0, y1, min, max) > value)
RorschachUK 0:e2daaf858e13 189 valColour = backColour;
RorschachUK 0:e2daaf858e13 190 else
RorschachUK 0:e2daaf858e13 191 valColour = getMappedColour(i, y0, y1, minColour, maxColour);
RorschachUK 0:e2daaf858e13 192 screen.fillrect(x0, i, x1, i+3, valColour);
RorschachUK 0:e2daaf858e13 193 }
RorschachUK 0:e2daaf858e13 194 } else {
RorschachUK 0:e2daaf858e13 195 for (int i = y1; i < y0; i+=5) {
RorschachUK 0:e2daaf858e13 196 if (map(i, y0, y1, min, max) > value)
RorschachUK 0:e2daaf858e13 197 valColour = backColour;
RorschachUK 0:e2daaf858e13 198 else
RorschachUK 0:e2daaf858e13 199 valColour = getMappedColour(i, y0, y1, minColour, maxColour);
RorschachUK 0:e2daaf858e13 200 screen.fillrect(x0, i-3, x1, i, valColour);
RorschachUK 0:e2daaf858e13 201 }
RorschachUK 0:e2daaf858e13 202 }
RorschachUK 0:e2daaf858e13 203 }
RorschachUK 0:e2daaf858e13 204 }
RorschachUK 0:e2daaf858e13 205
RorschachUK 0:e2daaf858e13 206 /** Draw a keyboard based on the supplied array of cells
RorschachUK 0:e2daaf858e13 207 *
RorschachUK 0:e2daaf858e13 208 * @param cells the cells to draw
RorschachUK 0:e2daaf858e13 209 * @param rowHighlight off if -1, else highlights the numbered row
RorschachUK 0:e2daaf858e13 210 * @param colHighlight off if -1, else highlights the cell in row/col
RorschachUK 0:e2daaf858e13 211 */
RorschachUK 0:e2daaf858e13 212 void drawKeyboard(int rowSize[ROWS], string cells[ROWS][COLS], int colPos[ROWS][COLS], int rowHighlight, int colHighlight)
RorschachUK 0:e2daaf858e13 213 {
RorschachUK 0:e2daaf858e13 214 int lineColour = RGBColour(0x20,0xFF,0xE0);
RorschachUK 0:e2daaf858e13 215 screen.foreground(RGBColour(0xE0,0xC0,0x10));
RorschachUK 0:e2daaf858e13 216 screen.set_font((unsigned char*) Arial28x28);
RorschachUK 0:e2daaf858e13 217 for (int i=0; i<ROWS; i++) {
RorschachUK 0:e2daaf858e13 218 int yPos=111+i*32;
RorschachUK 0:e2daaf858e13 219 for (int j=0; j< rowSize[i]; j++) {
RorschachUK 0:e2daaf858e13 220 if (j>0)
RorschachUK 0:e2daaf858e13 221 screen.locate(colPos[i][j-1]+5,84+i*32);
RorschachUK 0:e2daaf858e13 222 else
RorschachUK 0:e2daaf858e13 223 screen.locate(10,84+i*32);
RorschachUK 0:e2daaf858e13 224 screen.printf("%s",cells[i][j]);
RorschachUK 0:e2daaf858e13 225 screen.line(colPos[i][j],yPos-31,colPos[i][j],yPos, lineColour);
RorschachUK 0:e2daaf858e13 226 }
RorschachUK 0:e2daaf858e13 227 screen.line(5, yPos, 315, yPos, lineColour);
RorschachUK 0:e2daaf858e13 228 }
RorschachUK 0:e2daaf858e13 229 screen.line(5, 79, 315, 79, lineColour);
RorschachUK 0:e2daaf858e13 230 screen.line(5, 79, 5, 239, lineColour);
RorschachUK 0:e2daaf858e13 231 if (rowHighlight >= 0) {
RorschachUK 0:e2daaf858e13 232 if (colHighlight >= 0) { //highlight a cell
RorschachUK 0:e2daaf858e13 233 if (colHighlight==0)
RorschachUK 0:e2daaf858e13 234 screen.rect(5,79+rowHighlight*32,colPos[rowHighlight][0],111+rowHighlight*32,RGBColour(0xFF,0x40,0x40));
RorschachUK 0:e2daaf858e13 235 else
RorschachUK 0:e2daaf858e13 236 screen.rect(colPos[rowHighlight][colHighlight-1],79+rowHighlight*32,colPos[rowHighlight][colHighlight],111+rowHighlight*32,RGBColour(0xFF,0x40,0x40));
RorschachUK 0:e2daaf858e13 237 } else { // highlight whole row
RorschachUK 0:e2daaf858e13 238 screen.rect(5,79+rowHighlight*32,315,111+rowHighlight*32,RGBColour(0xFF,0x40,0x40));
RorschachUK 0:e2daaf858e13 239 }
RorschachUK 0:e2daaf858e13 240 }
RorschachUK 0:e2daaf858e13 241 }
RorschachUK 0:e2daaf858e13 242
RorschachUK 0:e2daaf858e13 243 void drawText()
RorschachUK 0:e2daaf858e13 244 {
RorschachUK 0:e2daaf858e13 245 screen.rect(5,35,315,74,RGBColour(0x20,0xFF,0xE0));
RorschachUK 0:e2daaf858e13 246 screen.locate(10,40);
RorschachUK 0:e2daaf858e13 247 screen.foreground(RGBColour(0xE0,0xC0,0x10));
RorschachUK 0:e2daaf858e13 248 screen.set_font((unsigned char*) Arial28x28);
RorschachUK 0:e2daaf858e13 249 screen.printf("%s_ ", typingBuffer);
RorschachUK 0:e2daaf858e13 250 }
RorschachUK 0:e2daaf858e13 251
RorschachUK 0:e2daaf858e13 252 /** This will be called if you blink.
RorschachUK 0:e2daaf858e13 253 */
RorschachUK 0:e2daaf858e13 254 void blinked(void)
RorschachUK 0:e2daaf858e13 255 {
RorschachUK 0:e2daaf858e13 256 //draw blink indicator
RorschachUK 0:e2daaf858e13 257 if (quality == 0) {
RorschachUK 0:e2daaf858e13 258 screen.fillrect(313, 13, 317, 17, White);
RorschachUK 0:e2daaf858e13 259 }
RorschachUK 0:e2daaf858e13 260 //select row or cell
RorschachUK 0:e2daaf858e13 261 if (col == -1)
RorschachUK 0:e2daaf858e13 262 col=-2;
RorschachUK 0:e2daaf858e13 263 else {
RorschachUK 0:e2daaf858e13 264 string s = keys[row][col];
RorschachUK 0:e2daaf858e13 265 if (s.compare("Space") == 0)
RorschachUK 0:e2daaf858e13 266 s=" ";
RorschachUK 0:e2daaf858e13 267 if (s.compare("Delete") == 0) {
RorschachUK 0:e2daaf858e13 268 s="";
RorschachUK 0:e2daaf858e13 269 if (typingBuffer.length() > 0) {
RorschachUK 0:e2daaf858e13 270 typingBuffer = typingBuffer.substr(0, typingBuffer.length() -1);
RorschachUK 0:e2daaf858e13 271 }
RorschachUK 0:e2daaf858e13 272 } else if (s.compare("\"SAY!\"") == 0) {
RorschachUK 0:e2daaf858e13 273 say(typingBuffer);
RorschachUK 0:e2daaf858e13 274 screen.fillrect(5,35,315,74,Black);
RorschachUK 0:e2daaf858e13 275 typingBuffer="";
RorschachUK 0:e2daaf858e13 276 } else {
RorschachUK 0:e2daaf858e13 277 typingBuffer.append(s);
RorschachUK 0:e2daaf858e13 278 }
RorschachUK 0:e2daaf858e13 279 row = -1;
RorschachUK 0:e2daaf858e13 280 col = -1;
RorschachUK 0:e2daaf858e13 281 drawText();
RorschachUK 0:e2daaf858e13 282 }
RorschachUK 0:e2daaf858e13 283 }
RorschachUK 0:e2daaf858e13 284
RorschachUK 0:e2daaf858e13 285 /** This will be called when processed eSense data comes in, about once a second.
RorschachUK 0:e2daaf858e13 286 *
RorschachUK 0:e2daaf858e13 287 * @param poorQuality will be 0 if connections are good, 200 if connections are useless, and somewhere in between if connection dodgy.
RorschachUK 0:e2daaf858e13 288 * @param attention processed percentage denoting focus and attention. 0 to 100
RorschachUK 0:e2daaf858e13 289 * @param meditation processed percentage denoting calmness and serenity. 0 to 100
RorschachUK 0:e2daaf858e13 290 * @param timeSinceLastPacket time since last packet processed, in milliseconds.
RorschachUK 0:e2daaf858e13 291 */
RorschachUK 0:e2daaf858e13 292 void eSenseData(int poorQuality, int attention, int meditation, int timeSinceLastPacket)
RorschachUK 0:e2daaf858e13 293 {
RorschachUK 0:e2daaf858e13 294 //quality indicator
RorschachUK 0:e2daaf858e13 295 quality=poorQuality;
RorschachUK 0:e2daaf858e13 296 if (poorQuality == 200)
RorschachUK 0:e2daaf858e13 297 screen.fillrect(313, 3, 317, 7, Red);
RorschachUK 0:e2daaf858e13 298 else if (poorQuality == 0)
RorschachUK 0:e2daaf858e13 299 screen.fillrect(313, 3, 317, 7, Green);
RorschachUK 0:e2daaf858e13 300 else
RorschachUK 0:e2daaf858e13 301 screen.fillrect(313, 3, 317, 7, Yellow);
RorschachUK 0:e2daaf858e13 302
RorschachUK 0:e2daaf858e13 303 //minimal eSense bars up at the top of the screen
RorschachUK 0:e2daaf858e13 304 screen.set_font((unsigned char*) Arial12x12);
RorschachUK 0:e2daaf858e13 305 if (attention > 0) {
RorschachUK 0:e2daaf858e13 306 displayBarGraph(200, 5, 310, 15, true, attention, 0, 100, RGBColour(0x10,0x00,0x00), RGBColour(0xFF,0x00,0x00), 0x00);
RorschachUK 0:e2daaf858e13 307 screen.locate(135, 6);
RorschachUK 0:e2daaf858e13 308 screen.foreground(Red);
RorschachUK 0:e2daaf858e13 309 screen.printf("Att: %d ",attention);
RorschachUK 0:e2daaf858e13 310 }
RorschachUK 0:e2daaf858e13 311 if (meditation > 0) {
RorschachUK 0:e2daaf858e13 312 displayBarGraph(200, 18, 310, 28, true, meditation, 0, 100, RGBColour(0x00,0x10,0x00), RGBColour(0x00,0xFF,0x00), 0x00);
RorschachUK 0:e2daaf858e13 313 screen.locate(128, 19);
RorschachUK 0:e2daaf858e13 314 screen.foreground(Green);
RorschachUK 0:e2daaf858e13 315 screen.printf("Med: %d ",meditation);
RorschachUK 0:e2daaf858e13 316 }
RorschachUK 0:e2daaf858e13 317 //clear blink indicator
RorschachUK 0:e2daaf858e13 318 screen.fillrect(313, 13, 317, 17, Black);
RorschachUK 0:e2daaf858e13 319 //Safe to start yet?
RorschachUK 0:e2daaf858e13 320 if (initialDelay.read() == 0)
RorschachUK 0:e2daaf858e13 321 initialDelay.start();
RorschachUK 0:e2daaf858e13 322 else if (initialDelay.read() > 5) {
RorschachUK 0:e2daaf858e13 323 started=true;
RorschachUK 0:e2daaf858e13 324 initialDelay.stop();
RorschachUK 0:e2daaf858e13 325 }
RorschachUK 0:e2daaf858e13 326 }
RorschachUK 0:e2daaf858e13 327
RorschachUK 0:e2daaf858e13 328 /** This will be called when processed meter reading data arrives, about once a second.
RorschachUK 0:e2daaf858e13 329 * This is a breakdown of frequencies in the wave data into 8 named bands, these are:
RorschachUK 0:e2daaf858e13 330 * 0: Delta (0.5-2.75 Hz)
RorschachUK 0:e2daaf858e13 331 * 1: Theta (3.5-6.75 Hz)
RorschachUK 0:e2daaf858e13 332 * 2: Low-Alpha (7.5-9.25 Hz)
RorschachUK 0:e2daaf858e13 333 * 3: High-Alpha (10-11.75 Hz)
RorschachUK 0:e2daaf858e13 334 * 4: Low-Beta (13-16.75 Hz)
RorschachUK 0:e2daaf858e13 335 * 5: High-Beta (18-29.75 Hz)
RorschachUK 0:e2daaf858e13 336 * 6: Low-Gamma (31-39.75 Hz)
RorschachUK 0:e2daaf858e13 337 * 7: High-Gamma (41-49.75 Hz)
RorschachUK 0:e2daaf858e13 338 *
RorschachUK 0:e2daaf858e13 339 * @param meter array of meter data for different frequency bands
RorschachUK 0:e2daaf858e13 340 * @param meterMin array of minimum recorded samples of each band
RorschachUK 0:e2daaf858e13 341 * @param meterMax arrat if naximum recorded samples of each band
RorschachUK 0:e2daaf858e13 342 */
RorschachUK 0:e2daaf858e13 343 void meterData(int meter[8], int meterMin[8], int meterMax[8])
RorschachUK 0:e2daaf858e13 344 {
RorschachUK 0:e2daaf858e13 345 //first good signal?
RorschachUK 0:e2daaf858e13 346 if (!connected) {
RorschachUK 0:e2daaf858e13 347 connected=true;
RorschachUK 0:e2daaf858e13 348 screen.fillrect(0,0,319,30,Black); //clear the Waiting to connect msg
RorschachUK 0:e2daaf858e13 349 initialDelay.reset();
RorschachUK 0:e2daaf858e13 350 initialDelay.start();
RorschachUK 0:e2daaf858e13 351 }
RorschachUK 0:e2daaf858e13 352 //minimal meter bars up at the top of the screen
RorschachUK 0:e2daaf858e13 353 for (int j=0; j<8; j++) {
RorschachUK 0:e2daaf858e13 354 displayBarGraph(5 + j * 13, 30, 16 + j * 13, 3, false, meter[j], meterMin[j], meterMax[j], RGBColour(0, j*2, 0x10), RGBColour(0, j*32, 0xFF), 0x00);
RorschachUK 0:e2daaf858e13 355 }
RorschachUK 0:e2daaf858e13 356 //Hijack this routine for menu movement
RorschachUK 0:e2daaf858e13 357 if (started) {
RorschachUK 0:e2daaf858e13 358 if (col==-1) {
RorschachUK 0:e2daaf858e13 359 row++;
RorschachUK 0:e2daaf858e13 360 if (row>=ROWS)
RorschachUK 0:e2daaf858e13 361 row=0;
RorschachUK 0:e2daaf858e13 362 } else if (col==-2) {
RorschachUK 0:e2daaf858e13 363 col=0;
RorschachUK 0:e2daaf858e13 364 } else {
RorschachUK 0:e2daaf858e13 365 col++;
RorschachUK 0:e2daaf858e13 366 if (col>=keysRowSize[row])
RorschachUK 0:e2daaf858e13 367 col=-1;
RorschachUK 0:e2daaf858e13 368 }
RorschachUK 0:e2daaf858e13 369 drawKeyboard(keysRowSize, keys, keysColPos, row, col);
RorschachUK 0:e2daaf858e13 370 }
RorschachUK 0:e2daaf858e13 371 }
RorschachUK 0:e2daaf858e13 372
RorschachUK 0:e2daaf858e13 373 /** This will be called when wave data arrives.
RorschachUK 0:e2daaf858e13 374 * There will be a lot of these, 512 a second, so if you're planning to do anything
RorschachUK 0:e2daaf858e13 375 * here, don't let it take long. Best not to printf this out as it will just choke.
RorschachUK 0:e2daaf858e13 376 *
RorschachUK 0:e2daaf858e13 377 * param wave Raw wave data point
RorschachUK 0:e2daaf858e13 378 */
RorschachUK 0:e2daaf858e13 379 void waveData(int wave)
RorschachUK 0:e2daaf858e13 380 {
RorschachUK 0:e2daaf858e13 381 }
RorschachUK 0:e2daaf858e13 382
RorschachUK 0:e2daaf858e13 383 //*****************
RorschachUK 0:e2daaf858e13 384 //End User routines
RorschachUK 0:e2daaf858e13 385 //*****************
RorschachUK 0:e2daaf858e13 386
RorschachUK 0:e2daaf858e13 387 //System routines to obtain and parse data
RorschachUK 0:e2daaf858e13 388
RorschachUK 0:e2daaf858e13 389 /** Simplify serial comms
RorschachUK 0:e2daaf858e13 390 */
RorschachUK 0:e2daaf858e13 391 unsigned char ReadOneByte()
RorschachUK 0:e2daaf858e13 392 {
RorschachUK 0:e2daaf858e13 393 int ByteRead;
RorschachUK 0:e2daaf858e13 394
RorschachUK 0:e2daaf858e13 395 while(!blueSmirf.readable());
RorschachUK 0:e2daaf858e13 396 ByteRead = blueSmirf.getc();
RorschachUK 0:e2daaf858e13 397
RorschachUK 0:e2daaf858e13 398 return ByteRead;
RorschachUK 0:e2daaf858e13 399 }
RorschachUK 0:e2daaf858e13 400
RorschachUK 0:e2daaf858e13 401 /** Main loop, sets up and keeps listening for serial
RorschachUK 0:e2daaf858e13 402 */
RorschachUK 0:e2daaf858e13 403 int main()
RorschachUK 0:e2daaf858e13 404 {
RorschachUK 0:e2daaf858e13 405 //Video setup
RorschachUK 0:e2daaf858e13 406 screen.claim(stdout); // send stdout to the TFT display
RorschachUK 0:e2daaf858e13 407 screen.background(Black); // set background to black
RorschachUK 0:e2daaf858e13 408 screen.foreground(White); // set chars to white
RorschachUK 0:e2daaf858e13 409 screen.cls(); // clear the screen
RorschachUK 0:e2daaf858e13 410 screen.set_orientation(1);
RorschachUK 0:e2daaf858e13 411 screen.set_font((unsigned char*) Arial12x12);
RorschachUK 0:e2daaf858e13 412 screen.locate(5,5);
RorschachUK 0:e2daaf858e13 413 screen.printf("Waiting to connect...");
RorschachUK 0:e2daaf858e13 414
RorschachUK 0:e2daaf858e13 415 drawText();
RorschachUK 0:e2daaf858e13 416 drawKeyboard(keysRowSize, keys, keysColPos, -1, -1);
RorschachUK 0:e2daaf858e13 417
RorschachUK 0:e2daaf858e13 418 //Voice setup
RorschachUK 0:e2daaf858e13 419 voice.baud(9600);
RorschachUK 0:e2daaf858e13 420 voice.printf("\n");
RorschachUK 0:e2daaf858e13 421 while (!voice.readable())
RorschachUK 0:e2daaf858e13 422 ;
RorschachUK 0:e2daaf858e13 423 wait(0.01);
RorschachUK 0:e2daaf858e13 424 voice.getc();
RorschachUK 0:e2daaf858e13 425 say("Welcome to Blink talk.");
RorschachUK 0:e2daaf858e13 426
RorschachUK 0:e2daaf858e13 427 Timer t; //packet timer
RorschachUK 0:e2daaf858e13 428 t.start();
RorschachUK 0:e2daaf858e13 429 Timer blinkTimer; //used for detecting blinks
RorschachUK 0:e2daaf858e13 430 int time;
RorschachUK 0:e2daaf858e13 431 int generatedChecksum = 0;
RorschachUK 0:e2daaf858e13 432 int checksum = 0;
RorschachUK 0:e2daaf858e13 433 int payloadLength = 0;
RorschachUK 0:e2daaf858e13 434 int payloadData[64] = {0};
RorschachUK 0:e2daaf858e13 435 int poorQuality = 0;
RorschachUK 0:e2daaf858e13 436 int attention = 0;
RorschachUK 0:e2daaf858e13 437 int meditation = 0;
RorschachUK 0:e2daaf858e13 438 int wave = 0;
RorschachUK 0:e2daaf858e13 439 int meter[8] = {0};
RorschachUK 0:e2daaf858e13 440 int meterMin[8];
RorschachUK 0:e2daaf858e13 441 int meterMax[8];
RorschachUK 0:e2daaf858e13 442 for (int j = 0; j < 8; j++) {
RorschachUK 0:e2daaf858e13 443 meterMin[j]=99999999;
RorschachUK 0:e2daaf858e13 444 meterMax[j]=-99999999;
RorschachUK 0:e2daaf858e13 445 }
RorschachUK 0:e2daaf858e13 446 bool eSensePacket = false;
RorschachUK 0:e2daaf858e13 447 bool meterPacket = false;
RorschachUK 0:e2daaf858e13 448 bool wavePacket = false;
RorschachUK 0:e2daaf858e13 449
RorschachUK 0:e2daaf858e13 450 blueSmirf.baud(57600);
RorschachUK 0:e2daaf858e13 451 blinkTimer.reset();
RorschachUK 0:e2daaf858e13 452
RorschachUK 0:e2daaf858e13 453 while(1) {
RorschachUK 0:e2daaf858e13 454 // Look for sync bytes
RorschachUK 0:e2daaf858e13 455 if(ReadOneByte() == 170) {
RorschachUK 0:e2daaf858e13 456 if(ReadOneByte() == 170) {
RorschachUK 0:e2daaf858e13 457 //Synchronised to start of packet
RorschachUK 0:e2daaf858e13 458 payloadLength = ReadOneByte();
RorschachUK 0:e2daaf858e13 459 if(payloadLength > 169) //Payload length can not be greater than 169
RorschachUK 0:e2daaf858e13 460 return;
RorschachUK 0:e2daaf858e13 461
RorschachUK 0:e2daaf858e13 462 generatedChecksum = 0;
RorschachUK 0:e2daaf858e13 463 for(int i = 0; i < payloadLength; i++) {
RorschachUK 0:e2daaf858e13 464 payloadData[i] = ReadOneByte(); //Read payload into memory
RorschachUK 0:e2daaf858e13 465 generatedChecksum += payloadData[i];
RorschachUK 0:e2daaf858e13 466 }
RorschachUK 0:e2daaf858e13 467
RorschachUK 0:e2daaf858e13 468 checksum = ReadOneByte(); //Read checksum byte from stream
RorschachUK 0:e2daaf858e13 469 generatedChecksum = 255 - (generatedChecksum & 0xFF); //Take one's compliment of generated checksum
RorschachUK 0:e2daaf858e13 470
RorschachUK 0:e2daaf858e13 471 if(checksum == generatedChecksum) {
RorschachUK 0:e2daaf858e13 472 //Packet seems OK
RorschachUK 0:e2daaf858e13 473 poorQuality = 200;
RorschachUK 0:e2daaf858e13 474 attention = 0;
RorschachUK 0:e2daaf858e13 475 meditation = 0;
RorschachUK 0:e2daaf858e13 476 wave = 0;
RorschachUK 0:e2daaf858e13 477 for(int i = 0; i < payloadLength; i++) { // Parse the payload
RorschachUK 0:e2daaf858e13 478 switch (payloadData[i]) {
RorschachUK 0:e2daaf858e13 479 case 2: //quality
RorschachUK 0:e2daaf858e13 480 i++;
RorschachUK 0:e2daaf858e13 481 poorQuality = payloadData[i];
RorschachUK 0:e2daaf858e13 482 eSensePacket = true;
RorschachUK 0:e2daaf858e13 483 break;
RorschachUK 0:e2daaf858e13 484 case 4: //attention
RorschachUK 0:e2daaf858e13 485 i++;
RorschachUK 0:e2daaf858e13 486 attention = payloadData[i];
RorschachUK 0:e2daaf858e13 487 eSensePacket = true;
RorschachUK 0:e2daaf858e13 488 break;
RorschachUK 0:e2daaf858e13 489 case 5: //meditation
RorschachUK 0:e2daaf858e13 490 i++;
RorschachUK 0:e2daaf858e13 491 meditation = payloadData[i];
RorschachUK 0:e2daaf858e13 492 eSensePacket = true;
RorschachUK 0:e2daaf858e13 493 break;
RorschachUK 0:e2daaf858e13 494 case 0x80: //wave
RorschachUK 0:e2daaf858e13 495 wave = payloadData[i+2] * 256 + payloadData[i+3];
RorschachUK 0:e2daaf858e13 496 //We also want to try to detect blinks via analysing wave data
RorschachUK 0:e2daaf858e13 497 time = blinkTimer.read_ms();
RorschachUK 0:e2daaf858e13 498 if (wave > 32767) wave -= 65535; //cope with negatives
RorschachUK 0:e2daaf858e13 499 if (wave>200 && time == 0) {
RorschachUK 0:e2daaf858e13 500 blinkTimer.start();
RorschachUK 0:e2daaf858e13 501 } else if (wave<-90 && time > 10 && time < 350) {
RorschachUK 0:e2daaf858e13 502 blinkTimer.stop();
RorschachUK 0:e2daaf858e13 503 blinkTimer.reset();
RorschachUK 0:e2daaf858e13 504 blinked();
RorschachUK 0:e2daaf858e13 505 } else if (time>500) {
RorschachUK 0:e2daaf858e13 506 blinkTimer.stop();
RorschachUK 0:e2daaf858e13 507 blinkTimer.reset();
RorschachUK 0:e2daaf858e13 508 }
RorschachUK 0:e2daaf858e13 509 i = i + 3;
RorschachUK 0:e2daaf858e13 510 wavePacket = true;
RorschachUK 0:e2daaf858e13 511 break;
RorschachUK 0:e2daaf858e13 512 case 0x83: //meter readings for different frequency bands
RorschachUK 0:e2daaf858e13 513 for (int j=0; j<8; j++) {
RorschachUK 0:e2daaf858e13 514 //documentation is inconsistent about whether these values are big-endian or little-endian,
RorschachUK 0:e2daaf858e13 515 //and claims both in different places. But wave data is big-endian so assuming that here.
RorschachUK 0:e2daaf858e13 516 meter[j] = payloadData[i+j*3+2]*65536 + payloadData[i+j*3+3]*256 + payloadData[i+j*3+4];
RorschachUK 0:e2daaf858e13 517 if (quality==0) {
RorschachUK 0:e2daaf858e13 518 if (meter[j]<meterMin[j])
RorschachUK 0:e2daaf858e13 519 meterMin[j]=meter[j];
RorschachUK 0:e2daaf858e13 520 if (meter[j]>meterMax[j])
RorschachUK 0:e2daaf858e13 521 meterMax[j]=meter[j];
RorschachUK 0:e2daaf858e13 522 }
RorschachUK 0:e2daaf858e13 523 }
RorschachUK 0:e2daaf858e13 524 meterPacket = true;
RorschachUK 0:e2daaf858e13 525 i = i + 25;
RorschachUK 0:e2daaf858e13 526 break;
RorschachUK 0:e2daaf858e13 527 default:
RorschachUK 0:e2daaf858e13 528 break;
RorschachUK 0:e2daaf858e13 529 } // switch
RorschachUK 0:e2daaf858e13 530 } // for loop
RorschachUK 0:e2daaf858e13 531
RorschachUK 0:e2daaf858e13 532 //Call routines to process data
RorschachUK 0:e2daaf858e13 533 if(eSensePacket) {
RorschachUK 0:e2daaf858e13 534 eSenseData(poorQuality, attention, meditation, t.read_ms());
RorschachUK 0:e2daaf858e13 535 eSensePacket = false;
RorschachUK 0:e2daaf858e13 536 }
RorschachUK 0:e2daaf858e13 537 if (meterPacket) {
RorschachUK 0:e2daaf858e13 538 meterData(meter, meterMin, meterMax);
RorschachUK 0:e2daaf858e13 539 t.reset();
RorschachUK 0:e2daaf858e13 540 meterPacket=false;
RorschachUK 0:e2daaf858e13 541 }
RorschachUK 0:e2daaf858e13 542 if (wavePacket) {
RorschachUK 0:e2daaf858e13 543 waveData(wave);
RorschachUK 0:e2daaf858e13 544 wavePacket=false;
RorschachUK 0:e2daaf858e13 545 }
RorschachUK 0:e2daaf858e13 546 } else {
RorschachUK 0:e2daaf858e13 547 // Checksum Error
RorschachUK 0:e2daaf858e13 548 } // end if else for checksum
RorschachUK 0:e2daaf858e13 549 } // end if read 0xAA byte
RorschachUK 0:e2daaf858e13 550 } // end if read 0xAA byte
RorschachUK 0:e2daaf858e13 551 } //end while
RorschachUK 0:e2daaf858e13 552 }