Project in progress

Person Detection with TensorFlow and Arduino © GPL3+

Using the TensorFlow Lite library, we can flash tiny machine learning models on an Arduino to detect humans from a camera.

  • 9,961 views
  • 2 comments
  • 83 respects

Components and supplies

Necessary tools and machines

Apps and online services

About this project

Intro

One of the exciting aspects of running machine learning models on embedded hardware is the ability to use low power devices with greater contextual awareness to trigger high energy devices only when it is valuable to do so.

To learn more about this vision, check out Pete Warden's blog.

As a proof-of-concept, we want to use the low-power Arduino Nano 33 BLE Sense and an ArduCam Mini 2MP, along with the TensorFlow Lite library, to trigger a relay to turn on/off when a person is recognized.

Basic wiring

Follow the following wiring diagram to connect your Arduino Nano 33 BLE Sense to the ArduCam Mini 2MP. This diagram also shows how to add a lipo battery after you've flashed the Arduino.

Flashing the model

A few examples of tiny models have been released on the official TensorFlow repo, the micro_speech model being the most widely shown example. We wanted to flash the vision (person_detection) demo, however as of the time of this post being written, not all of the resources are available. From the micro_speech example, we saw that these Arduino zip packages were being stored in the same place as Google's nightly TensorFlow builds. When you navigate to this link, you'll see a document tree with links to all kinds of nightly builds:

Doing a quick word search through the document tree, we were able to locate the missing package! (Update: TF team is working on updating the missing link)

Download this package link and rename it to person_detection.zip. This will be the package we upload as a library to our Arduino IDE as the official instructions describe.

After uploading this zip package, we want to make a couple more modifications for things to run smoothly. With your text editor of choice, find where the Arduino library installed and open the library.properties file for editing

# In our case, our Arduino library is in ~/Arduino/libraries
$ cd ~/Arduino/libraries/person_detection
$ vim library.properties

Change the first line that declares the name of the library to TensorFlowLite:person_detection. Your file should look like this:

name=TensorFlowLite:person_detection
version=1.14-ALPHA
author=TensorFlow Authors
maintainer=Pete Warden <petewarden@google.com>
sentence=Allows you to run machine learning models locally on your device.
paragraph=This library runs TensorFlow machine learning models on microcontrollers, allowing you to build AI/ML applications powered by deep learning and neural networks. With the included examples, you can recognize speech, detect people using a camera, and recognise "magic wand" gestures using an accelerometer. The examples work best with the Arduino Nano 33 BLE Sense board, which has a microphone and accelerometer.
category=Data Processing
url=https://www.tensorflow.org/lite/microcontrollers/overview
ldflags=-lm
includes=TensorFlowLite.h

This makes it visible in the Arduino IDE examples menu. Finally, navigate to the examples/ directory in the same place and rename the subdirectory to person_detection. All these changes resolve the necessary differences to get the demo running.

From here, you can follow the official instructions to download the libraries for the ArduCam and JPEG decoding.

To run the example, navigate to Files -> Examples -> TensorflowLite:person_detection and select the person_detection sample script. Make sure you have your Arduino BLE 33 sense connected to your computer and it has been selected for flashing on the Arduino IDE. Flash the example and it should start up immediately. The on board LED will flash blue to indicate that an image has been captured and inference has been completed. The light will turn red if it did not detect a human and turn green when it has. Inference takes ~19 seconds since the person detection model is fairly large for the device.

Using the Arduino as a Smart Switch

Microcontrollers are very energy efficient, some being able to operate on coin cell batteries for years! With recent efforts to minimize machine learning models to run on embedded devices, we can build smart "switches" that can be invoked to power or trigger other devices that need more power to operate. With this person detection model running on the Arduino BLE sense, we can trigger a high energy consuming device like a Donkeycar using the sombrero shield. We'll use a relay connected to the Arduino and switch it on/off depending on inference results.

Wire up the relay, pi, and arduino following the wiring diagram below:

Now simply add a few lines to the person_detection.ino example to look at the inference results and switch the relay.

// at the top of the script, initialize a variable for the relay signal
int relayOut = 10;

// in the setup() loop, initialize the relayOut pin as output
void setup(){
/* .... */
pinMode(relayOut, OUTPUT);
}

// in the void() loop:
void loop() {
// Get image from provider.
if (kTfLiteOk != GetImage(error_reporter, kNumCols, kNumRows, kNumChannels,
input->data.uint8)) {
error_reporter->Report("Image capture failed.");
}
// Run the model on this input and make sure it succeeds.
if (kTfLiteOk != interpreter->Invoke()) {
error_reporter->Report("Invoke failed.");
}
TfLiteTensor* output = interpreter->output(0);
// Process the inference results.
uint8_t person_score = output->data.uint8[kPersonIndex];
uint8_t no_person_score = output->data.uint8[kNotAPersonIndex];
RespondToDetection(error_reporter, person_score, no_person_score);

// Add the following lines:
if (person_score >= no_person_score){
digitalWrite(relayOut, HIGH);
}
else {
digitalWrite(relayOut, LOW);
}

}

Extending our Smart Switch using BLE

There may be situations where a high powered device could benefit from logic inferred elsewhere. For example, say you wanted a power hungry rover to start up when a person is detected in another section of a large warehouse. A way to extend the perception of our smart relay switch is through BLE.

BLE is a low power consuming wireless communication protocol designed to send small amounts of data. In this example, we set up an Arduino Nano 33 BLE Sense with the ArduCam attached to it as a control, sending a high alert when a person is detected. In the person_detection script, along with other BLE initialization steps, we add the logic to send a high alert when the person_score is high enough.

if (peripheral.connected()) {
if (sendsignal > 0){
alertLevelChar.writeValue((byte)0x02);
Serial.println("Wrote high alert");
}
else{
alertLevelChar.writeValue((byte)0x00);
Serial.println("Wrote low alert");
}
}

A second Arduino Nano 33 BLE Sense will serve as a peripheral device that will listen for that alert and switch a relay attached to it.

while (central.connected()) {
//Serial.println("Getting Alert Level:");
if (alertLevelChar.written()){
if (alertLevelChar.value()){
digitalWrite(relay, HIGH);
Serial.println("Set relay to HIGH");
} else {
digitalWrite(relay, LOW);
Serial.println("Set relay to LOW");
}
}
}

Code

Person Detection with BLEC/C++
This script uses the Arduino BLE library to send a high alert to a peripheral bluetooth device when a person has been detected.
/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/
#include <ArduinoBLE.h>
#include <TensorFlowLite.h>

#include "main_functions.h"

#include "detection_responder.h"
#include "image_provider.h"
#include "model_settings.h"
#include "person_detect_model_data.h"
#include "tensorflow/lite/experimental/micro/kernels/micro_ops.h"
#include "tensorflow/lite/experimental/micro/micro_error_reporter.h"
#include "tensorflow/lite/experimental/micro/micro_interpreter.h"
#include "tensorflow/lite/experimental/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"

// Globals, used for compatibility with Arduino-style sketches.
namespace {
tflite::ErrorReporter* error_reporter = nullptr;
const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;

// An area of memory to use for input, output, and intermediate arrays.
constexpr int kTensorArenaSize = 70 * 1024;
static uint8_t tensor_arena[kTensorArenaSize];
}  // namespace

// The name of this function is important for Arduino compatibility.
void setup() {
  // Set up logging. Google style is to avoid globals or statics because of
  // lifetime uncertainty, but since this has a trivial destructor it's okay.
  // NOLINTNEXTLINE(runtime-global-variables)
  static tflite::MicroErrorReporter micro_error_reporter;
  error_reporter = &micro_error_reporter;

  // Map the model into a usable data structure. This doesn't involve any
  // copying or parsing, it's a very lightweight operation.
  model = tflite::GetModel(g_person_detect_model_data);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    error_reporter->Report(
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        model->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  // Pull in only the operation implementations we need.
  // This relies on a complete list of all the ops needed by this graph.
  // An easier approach is to just use the AllOpsResolver, but this will
  // incur some penalty in code space for op implementations that are not
  // needed by this graph.
  //
  // tflite::ops::micro::AllOpsResolver resolver;
  // NOLINTNEXTLINE(runtime-global-variables)
  static tflite::MicroMutableOpResolver micro_mutable_op_resolver;
  micro_mutable_op_resolver.AddBuiltin(
      tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
      tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
  micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_CONV_2D,
                                       tflite::ops::micro::Register_CONV_2D());
  micro_mutable_op_resolver.AddBuiltin(
      tflite::BuiltinOperator_AVERAGE_POOL_2D,
      tflite::ops::micro::Register_AVERAGE_POOL_2D());

  // Build an interpreter to run the model with.
  static tflite::MicroInterpreter static_interpreter(
      model, micro_mutable_op_resolver, tensor_arena, kTensorArenaSize,
      error_reporter);
  interpreter = &static_interpreter;

  // Allocate memory from the tensor_arena for the model's tensors.
  TfLiteStatus allocate_status = interpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    error_reporter->Report("AllocateTensors() failed");
    return;
  }

  // Get information about the memory area to use for the model's input.
  input = interpreter->input(0);
  /* BLE initialize */
  BLE.begin();
  //BLE.scanForUuid("1801");
}

// The name of this function is important for Arduino compatibility.
void loop() {
  // Get image from provider.
  if (kTfLiteOk != GetImage(error_reporter, kNumCols, kNumRows, kNumChannels,
                            input->data.uint8)) {
    error_reporter->Report("Image capture failed.");
  }

  // Run the model on this input and make sure it succeeds.
  if (kTfLiteOk != interpreter->Invoke()) {
    error_reporter->Report("Invoke failed.");
  }

  TfLiteTensor* output = interpreter->output(0);

  // Process the inference results.
  uint8_t person_score = output->data.uint8[kPersonIndex];
  uint8_t no_person_score = output->data.uint8[kNotAPersonIndex];
  RespondToDetection(error_reporter, person_score, no_person_score);
  /* Send inference via bluetooth */
  BLE.scanForUuid("1801");
  delay(500); // delay to find uuid
  BLEDevice peripheral = BLE.available();
  Serial.println(peripheral);
  if (peripheral){
    if (peripheral.localName() != "PersonDetectionMonitor") {
      return;
    }
    BLE.stopScan();
    if (person_score > no_person_score){
      sendAlert(peripheral,2);
    }
    else{
      sendAlert(peripheral,0);
    }
    //peripheral.disconnect();
  }
  else{
    Serial.println("Peripheral not available");
  }
}

void sendAlert(BLEDevice peripheral, int sendsignal) {
  // connect to the peripheral
  Serial.println("Connecting ...");

  if (peripheral.connect()) {
    Serial.println("Connected");
  } else {
    Serial.println("Failed to connect!");
    return;
  }

  // discover peripheral attributes
  Serial.println("Discovering attributes ...");
  if (peripheral.discoverAttributes()) {
    Serial.println("Attributes discovered");
  } else {
    Serial.println("Attribute discovery failed!");
    peripheral.disconnect();
    return;
  }
  // retrieve the alert level characteristic
  BLECharacteristic alertLevelChar = peripheral.characteristic("2A06");

   if (!alertLevelChar) {
    Serial.println("Peripheral does not have alert level characteristic!");
    peripheral.disconnect();
    return;
  } else if (!alertLevelChar.canWrite()) {
    Serial.println("Peripheral does not have a writable alert level characteristic!");
    peripheral.disconnect();
    return;
  }

  if (peripheral.connected()) {
    if (sendsignal > 0){
      alertLevelChar.writeValue((byte)0x02);
      Serial.println("Wrote high alert");
    }
    else{
      alertLevelChar.writeValue((byte)0x00);
      Serial.println("Wrote low alert");
    }
  }
  peripheral.disconnect();
  Serial.println("Peripheral disconnected");
}
Peripheral DeviceC/C++
This script uses the Arduino BLE library to connect and listen for an alert triggered by the person detection device to switch a relay on/off
#include <ArduinoBLE.h>
#include "Arduino.h"

// Relay on D8
int relay = 8;

 // BLE PersonDetection Service
BLEService PersonDetectionService("1801");

// BLE Alert Level Characteristic
BLEByteCharacteristic alertLevelChar("2A06",  // standard 16-bit characteristic UUID
    BLERead | BLEWrite); // remote clients will be able to get notifications if this characteristic changes


void setup() {
  Serial.begin(9600);    // initialize serial communication
  while (!Serial);

  pinMode(LED_BUILTIN, OUTPUT); // initialize the built-in LED pin to indicate when a central is connected
  //pinMode(LEDR, OUTPUT);
  //pinMode(LEDG, OUTPUT);
  //pinMode(LEDB, OUTPUT);
  //pinMode(relay, OUTPUT);

  //digitalWrite(LEDR, LOW);
  //digitalWrite(LEDG, LOW);
  //digitalWrite(LEDB, LOW);
  digitalWrite(relay, LOW);
  
  // begin initialization
  if (!BLE.begin()) {
    Serial.println("starting BLE failed!");

    while (1);
  }

  /* Set a local name for the BLE device
     This name will appear in advertising packets
     and can be used by remote devices to identify this BLE device
     The name can be changed but maybe be truncated based on space left in advertisement packet
  */
  BLE.setLocalName("PersonDetectionMonitor");
  BLE.setAdvertisedService(PersonDetectionService); // add the service UUID
  PersonDetectionService.addCharacteristic(alertLevelChar); // add the alert level characteristic
  BLE.addService(PersonDetectionService); // Add the battery service
  alertLevelChar.writeValue((byte)0x00);
  
  /* Start advertising BLE.  It will start continuously transmitting BLE
     advertising packets and will be visible to remote BLE central devices
     until it receives a new connection */

  // start advertising
  BLE.advertise();

  Serial.println("Bluetooth device active, waiting for connections...");
}

void loop() {
  // wait for a BLE central
  BLEDevice central = BLE.central();

  // if a central is connected to the peripheral:
  if (central) {
    Serial.print("Connected to central: ");
    // print the central's BT address:
    Serial.println(central.address());
    // turn on the LED to indicate the connection:
    digitalWrite(LED_BUILTIN, HIGH);

    // while the central is connected:
    while (central.connected()) {
      //Serial.println("Getting Alert Level:");
      if (alertLevelChar.written()){
        if (alertLevelChar.value()){
          Serial.println("Got high alert");
          digitalWrite(relay, HIGH);
          Serial.println("Set relay to HIGH");
        } else{
          Serial.println("Got low alert");
          digitalWrite(relay, LOW);
          Serial.println("Set relay to LOW");
        }
      }
    }
    // when the central disconnects, turn off the LED:
    digitalWrite(LED_BUILTIN, LOW);
    Serial.print("Disconnected from central: ");
    Serial.println(central.address());
  }
}

Comments

Similar projects you might like

Farmaid: Plant Disease Detection Robot

by Teamato

  • 17,520 views
  • 13 comments
  • 79 respects

Technologies: Intel Pattern Matching vs TensorFlow

by 7 developers

  • 883 views
  • 0 comments
  • 8 respects

Jrobot Self Drive Powered by TensorFlow Lite

Project showcase by joechen

  • 10,545 views
  • 13 comments
  • 69 respects

Anomaly Detection & Temperature Monitoring Using Bolt IoT

Project tutorial by ashish anand

  • 3,214 views
  • 2 comments
  • 19 respects

A Sudoku Solver's Robot

Project tutorial by msana

  • 7,372 views
  • 7 comments
  • 29 respects

Joy Robot (Robô Da Alegria)

Project tutorial by Igor Fonseca Albuquerque

  • 3,986 views
  • 5 comments
  • 30 respects
Add projectSign up / Login