Friday, 12 February 2016

Robert Burke, Sprint 1 - Extended vision system

Team:    QRM
Author:  Robert Burke
Date:      08/02/2016
Post:      #2

Action item review:
My action items for  last week were to implement an extended version of the basic vision system interactions to control a pan and tilt servo mechanism, using threads and interprocess communication (IPC) and remote programming of raspberry pi using laptop.

The basic functionality of the vision system was not extended as planned due to the limitations encountered with the basic vision system in terms of processing and FPS. In order to incorporate the basic vision script into a multi threaded script with my other team members code the basic vision script functionality was reduced to improve performance while being executed as a thread concurrently with the audio and servo control threads.

The displayed image and all displayed graphics were therefore removed as well as eye tracking functionality, to prototype the face tracking with audio and servo control threading. The display image is also removed reducing the work the RPI has to do.

The servo control code, audio code and vision code were then combined into threads in a single .py script by Micheal. The servo’s were fastened to a bread board and RPI camera mounted to the tilt servo. SInce the continuous image capture is no longer displayed on the screen, feedback from the system to indicate a face is detected was implemented by Micheal using digital output pins of the RPI and LED’s.

The servos stay in a central position until a face is detected with the RPI camera at which point the green LED turns on indicating a face detection and the servos move to center the image frame on the face. As the servo moves an audio file plays to mask the servo noise. Where the volume of the audio is a function of the distance of the servo movement and the servo step size being executed.

Full code available at,

Action discussion:
The vision system functionality was incorporated into a thread by Michael, and audio and vision code was contributed online using Google drive allowing each member of the team to upload and modify the main code. Below the vision system functionality of the code within the multithreaded code are outlined where audio and servo threads are omitted.

The RPI camera variables and cascade classifier are now initialised as global variables. Before the RPI camera is initialised within an initialisation function along with the inter process communications (IPC) mutex lock and event functions.

The vision thred itself is much like the basic vision system code where the frames are captured in a continuous loop, before being colour converted, operated on by the cascade classifier and the center of the face calculated.

An IF statement now checks the face array for a detected face and determines the center of the face, setting it to the centre of the frame if not, maintaining and returning to central position if no face detected. Digital outputs are now also sent to the GPIO pins for both cases as visual feedback to the user.

As the graphics were found to be an advantage when debugging movements etc, the graphics are now surrounded by an if statement which allows the user to only display graphics during debug mode. The image capture is again truncated to allow for the next frame to be processed before the mutex event is set allowing the servo thread to be executed.

Various methods of remotely programming the RPi were researched and three main methods were identified. Access command line, remotely transfer files through SSH or display RPI desktop on PC using VNC all of which have pro’s and con’s. Unfortunately time and commitments to other projects the not permit future implementation of these methods.

Useful Links:
  • Remote Programming of RaspberryPi using PyCharm
  • PiCamera Documentation

  • OpenCV/Python tutorials

  • Pi camera image capture methods

  • Object detection

  • Face tracking poject

  • RPI remote access methods

  • Remote access of rpi command line using KiTTY

  • Display RPI GUI on windows pc:

No comments:

Post a Comment