Project showcase
Head Tracking for Wireless 3D First Person Vision

Head Tracking for Wireless 3D First Person Vision © GPL3+

A remote controlled vehicle with stereo cameras on a head-tracked pan-tilt unit and wireless 3D first person control via an Oculus Rift.

  • 9,244 views
  • 7 comments
  • 58 respects

Components and supplies

Necessary tools and machines

09507 01
Soldering iron (generic)
3drag
3D Printer (generic)

Apps and online services

About this project

What is it about?

In remote vehicle operation, video feedback provides the operator with real-time spatial and environmental information. Stereoscopic video feedback can improve upon standard single camera systems by providing the operator with details about scale, depth and motion in a rapid and intuitive form. This supplementary information is especially important when operating in unfamiliar or hazardous environments.

Why did I make it?

This project was for a university mechatronic engineering assignment.

The scope of the design was to test the effects on vehicle control when using a 3D visual telepresence system to navigate surroundings. An effective design would deliver heightened interactivity, control, navigation and spatial awareness to the driver, whilst minimizing the negative effects of motion sickness.

How does it work?

A 3D first person vision system has been developed, which captures stereoscopic perspectives with a pair of small cameras and displays them to a user’s corresponding eyes via a head mounted display. The head mounted display simultaneously captures positional and rotational values of the user’s head, using this data to manipulate a pan-tilt unit on which the cameras are mounted. The effect is delivery of a minimal latency 3D perspective of the remote controlled vehicles surroundings, where the field of view can be mechanically adjusted by rotating the head in the desired direction (see first attached video).

Testing the Pan-Tilt unit - Arduino Uno, XBee wireless tx/rx and Oculus Rift DK2

There are three main modules that make up the project:

  • Video Capture and Communication (blue)
  • Headtracker Data Communication (green)
  • Pan-Tilt unit (PTU) with Controller (yellow)

An Arduino Uno served as a remote (vehicle mounted) platform for manipulating the PTU, via two digital servos embedded within a 3D printed enclosure. At a minimum the design required at least two Pulse Width Modulated signal outputs, an RS232 UART/USART Serial Controller, and additional I/O pins for additional sensors/actuators (such as dynamic stability). The Arduino was selected over a conventional microcontroller on the basis of:

  • High level language with pre-compiled libraries,
  • On-board power regulator, bootloader, etc
  • Ease of interface to external devices, such as XBee wireless serial devices
  • Processing speed and memory control not critical to minimize latency, when combined with already substantial video latency.

Results

Various different hypothesis related to motion sickness, control, cognitive memory retention and depth perception were all tested from the prototype design, but when focusing on hardware performance, the Arduino Uno was a simple and reliable platform.

The main target criteria for a successful design was anticipated as;

  • A wide enough field of view (>100 degrees)
  • Adequate video resolution
  • Accurate and responsive tracking
  • A high refresh rate
  • Low Latency

From these, the main issue observed was that to do with minimizing video latency, that is the time from moving your head to receiving the corresponding image to your eyes. The cameras, wireless video communication and display methods created a latency up to 133 milliseconds when stationary, with an additional 26 ms generated via the mechanical response of the head-tracking, serial comm's and mechanical adjustment of the PTU. This 26 ms of data processing/signal generation is insignificant when compared to that of the video latency.

In replicating this project, there are many issues regarding video equipment I would reconsider to increase image quality whilst minimizing latency. However, regarding the elements relating to the Arduino, I would consider a smaller packaged board (such as Arduino Pro Mini) and faster operating servo motors as the only limitation experienced was in the rate of PTU adjustment, not in speed of processing and I/O capacity.

Thank you, Arduino, for meeting my needs in what would otherwise be a daunting project.

Code

Arduino Servo Control from XBee serial receiveArduino
Flashed to Arduino Uno
#include <Servo.h>
Servo yawservo;  // create servo object to control a servo
Servo pitchservo;  // create servo object to control a servo

String inputString = "";         // a string to hold incoming data
boolean yawComplete = false;  // whether the string is complete
boolean pitchComplete = false;  // whether the string is complete

void setup() {
  // initialize serial:
  Serial.begin(9600); // reserve 200 bytes for the inputString:
  inputString.reserve(200);
  yawservo.attach(9);  // attaches the servo on pin 9 to the servo object
  pitchservo.attach(10);  // attaches the servo on pin 9 to the servo object
}

void loop() {
  serialEvent(); //call the function // print the string when a newline arrives:
  if (pitchComplete) {
    pitchservo.write(inputString.toInt());                  // sets the servo position according to the scaled value 
    delay(1); 
    inputString = "";
    pitchComplete = false;
  }

    if (yawComplete) {
    yawservo.write(inputString.toInt());                  // sets the servo position according to the scaled value 
    delay(1); 
    inputString = "";
    yawComplete = false;
  }
}

void serialEvent() {
  while (Serial.available()) {
    // get the new byte:
    char inChar = (char)Serial.read();
    if (inChar == 'x')
    {
      pitchComplete = true;
      break;
    }
        if (inChar == 'y')
    {
      yawComplete = true;
      break;
    }
    inputString += inChar;
  }
}

Comments

Similar projects you might like

Otto DIY+ Arduino Bluetooth Robot Easy to 3D Print

Project tutorial by Team Otto builders

  • 48,250 views
  • 117 comments
  • 162 respects

How to Make a Remote Controlled Robotic Hand with Arduino

Project showcase by Gabry295

  • 35,363 views
  • 5 comments
  • 110 respects

Minimal MIDI Drum Kit with 3D Printer

Project tutorial by ryokosaka

  • 9,881 views
  • 1 comment
  • 29 respects

Human Head to Robot Head

Project tutorial by jegatheesan

  • 3,401 views
  • 5 comments
  • 8 respects

Linear Motion Plotter

Project in progress by Kramick Saha

  • 2,699 views
  • 2 comments
  • 8 respects

PlayStation Remote Controlled Wireless Car

Project showcase by Jasmeet Singh

  • 2,586 views
  • 1 comment
  • 6 respects
Add projectSign up / Login