Authors

Connor Brothers, Richard Li

Objective

The objective of the project is to detect the body parts (namely hands) of the group members using a Pi camera with OpenCV library functions, and use the position of the detected objects to send movement commands to a robot. The controller is powered by a Pi 3b, and the robot is powered with the LPC 1768, and the robot and controller are connected with a HC-05 Bluetooth module master-slave pair.

Parts List

Pi Controller:

• Raspberry Pi 3b
• Micro SD Card with Raspbian
• Pi Camera 2.1 with cable
• 5V power source
• Pi Cobbler and breakout board
• HC-05 Bluetooth module

Robot

• Shadow Robot kit
• Dual H-Bridge
• battery pack
• 4 AA batteries
• Mbed LPC 1768
• 2 DC motors
• barrel jack adapter
• HC-05 Bluetooth module

Hardware Setup

Pi 3b

If the SD card does not already have Raspbian installed, follow the NOOBS installation tutorial at https://www.raspberrypi.org/documentation/installation/noobs.md.

Installing OpenCV

For the controller, the OpenCV library will have to be installed. A tutorial can be found at https://www.learnopencv.com/install-opencv-4-on-raspberry-pi/, although OpenCV frequently changes and the tutorial may be outdated by the time this document is read, so new tutorials would be preferable.

Important note: The OpenCV Contrib library and examples do not have to be installed for the controller to work, and compiling them will drastically increase the amount of time needed to compile the source code! When running the CMake command, replace the text in the tutorial with

cmake -D CMAKE_BUILD_TYPE=RELEASE \ -D CMAKE_INSTALL_PREFIX==/usr/local \ -D INSTALL_C_EXAMPLES=OFF \ -D INSTALL_PYTHON_EXAMPLES=OFF \ -D WITH_TBB=ON \ -D WITH_V4L=ON \ -D OPENCV_PYTHON3_INSTALL_PATH=$cwd/OpenCV-$cvVersion-py3/lib/python3.5/site-packages \ -D BUILD_EXAMPLES=OFF ..

to reduce time spent on this section. Compilation may take several hours, so be prepared to wait.

Connecting the Camera:

A tutorial for the camera is available at https://projects.raspberrypi.org/en/projects/getting-started-with-picamera.

If the pi is unable to take a picture, enter vcgencmd get_camera into the terminal. If a camera is detected, then it is possibly defective. Otherwise, the issue is with the Pi.

Enabling the serial port

The serial login console uses the same serial port that will be used for the HC-05, so it should be disabled first. From the terminal, enter #sudo raspi-config#, select interfacing, serial, and disable the login console and enable serial hardware.

Assembling the robot kit

Note: Be aware of motor orientation when assembling the robot.

Configuring HC-05 modules

Wiring (for this step only!)

A tutorial for configuring HC-05 modules is available at https://howtomechatronics.com/tutorials/arduino/how-to-configure-pair-two-hc-05-bluetooth-module-master-slave-commands/.

The below program can be used to enter AT command mode when using a terminal emulator with the Mbed. Note that the mbed library included may cause errors when compiling, so replace it with a newer mbed library from another project. Be sure that the program is loaded onto the Mbed before the HC-05 is powered on.

Wiring

Controller

HC-05

Robot

HC-05

H-Bridge

Left Motor

Right Motor

Code

Robot Code

Pi 3b Code

from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2
import numpy
import serial

camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
camera.rotation = 180
#camera.image_effect = 'none'
ser = serial.Serial(port='/dev/serial0', baudrate = 38400, parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE, bytesize=serial.EIGHTBITS, timeout=1)
rawCapture = PiRGBArray(camera, size=(640, 480))
low_1 = numpy.array([155,40,5], dtype = "uint8")
high_1 = numpy.array([179,255,200], dtype = "uint8")
low_2 = numpy.array([0,40,5], dtype = "uint8")
high_2 = numpy.array([19,255,200], dtype = "uint8")

time.sleep(0.1)
 
for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
    image = frame.array
    hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    img_mask = cv2.inRange(hsv_image, low_1, high_1)
    img_mask2 = cv2.inRange(hsv_image, low_2, high_2)
    img_mask = cv2.bitwise_or(img_mask, img_mask2, img_mask)
    masked_img = cv2.bitwise_and(image, image, mask = img_mask)
    thing = numpy.sum(img_mask)
    if thing > 30:
        M = cv2.moments(img_mask)
        cX = int(M["m10"] / M["m00"])
        cY = int(M["m01"] / M["m00"])
        cv2.circle(image, (cX, cY), 3, (255, 255, 255), -1)
        if cX < 100:
            ser.write("l")
        elif cX > 540:
            ser.write("r")
        elif cY < 120:
            ser.write("b")
        elif cY > 320:
            ser.write("f")
        else:
            ser.write('s')
    else:
        ser.write('s')
    #combined = numpy.concatenate((image, masked_img), axis = 1)
    cv2.imshow("stuff",image)
    key = cv2.waitKey(1) & 0xFF
    rawCapture.truncate(0)
 
    
    if key == ord("q"):
        break

Camera Demonstration: Detected hand position is at white dot