Project tutorial

J.A.R.V.I.S. : A Virtual Home Assistant © Apache-2.0

J.A.R.V.I.S. is a personal home automation assistant for controlling electrical home appliances integrated with an augmented reality app.

  • 19,924 views
  • 12 comments
  • 149 respects

Components and supplies

Necessary tools and machines

09507 01
Soldering iron (generic)
Hy gluegun
Hot glue gun (generic)

Apps and online services

About this project

Why tackle the Smart Home problem?

Nowadays, people have smartphones with them all the time. So it makes sense to use these to control home appliances. Presented here is a home automation system using a simple Android app which you can use to control electrical appliances with voice commands. Using the app you can control all the appliances like TV, fans, light etc. It’s like a Siri which can be controlled using voice (command given by user). You can give the command to switch on or off the devices (like light, fan etc) and also manipulate them like fan speed, light intensity. Commands are sent via a Bluetooth module to Arduino. So there is no need for you to get up to switch on or switch off the device while watching a movie or doing some important work.

What’s new?

We are also using an AR-augmented reality app (self made) to control the appliances and manipulate them. If we want to switch on the electrical appliance we just need to focus the camera and it will start it and using a virtual button we can switch it off .
If we want to know the electricity unit consumed by our appliances along with the cost of units we just need to send a text message to the device (Arduino) through our phone. The Arduino would be connected to the network through a GSM module which would reply with another text message containing the electricity unit and the total cost consumed by the appliances. This will help the user to get update on the electricity bill and they can decide whether to use the appliances on a regular routine or reduce it usage to save electricity.

Why did we decide to make it?

Working

The Digital Energy Meter

In the energy meter, a LDR (Light Dependent Resistor) is used to take the blink speed according to the load. The blink per second value is fed into the Arduino for further calculation units and the value of energy consumed. These are sent to the user via the GSM module.The user can then comprehend which appliance is consuming more power and take action accordingly.

The AI Virtual Assistant

The AI Virtual Assistant works using an android app which is installed on the user’s android phone. The user activates the agent using the wake up command, ‘Listen up Jarvis’.
The android app is connected to the Arduino 101 via Bluetooth which is inturn connected to the relay and hence the electrical appliances.

The flow of signals is as follows:

  • 1. Voice Input to Android App
  • 2. Processing on Android App
  • 3. Sending encrypted data to Arduino via Bluetooth.
  • 4. Processing on Arduino 101
  • 5. Digital signals to relay

Pins connections:

  • Pin 8 -- > PulseIn [Digital Input]
  • Pin 6 -- > IN1 -- > Fan Output
  • Pin 4 -- > IN2 -- > Light Output
  • Pin 5 -- > IN3 -- > Coffee Maker

Connections of Energy meter:

  • Connect the meter to the supply.
  • Cover the energy meter with a black tape, preventing the LDR inside the energy meter from outside light.
  • Connect the neutral wire directly to the loads and the phase wire to the relay.
  • Take the wires from the LDR outside the meter to the counter circuit.
  • Now take the output from the counter circuit to the assigned pin 8 pulsepin as in the code.
  • Connect the TX, RX pin of the Arduino 101 to RX, TX pin of the GSM module.
  • Now power up the boards after burning the code.

Circuit Diagram

Hardware and Equipment used:

  • Inside Energy meter: LDR Light Dependent Resistor
  • GSM module
  • Counter circuit: potentiometer, IC LM358N
  • Arduino 101
  • BlueTooth-HC05 module
  • 4 channel 12V 10A relay control board module with optocoupler for PIC AVR ARM

Programming Platforms

  • · For programming the Arduino 101, we have used the default Arduino IDE.
  • · We have developed our Android app using Android Studio (JAVA and XML).
  • · Database connectivity is done using Google's Firebase.
  • · For augmented reality we have used Unity along with the Vuforia SDK.

Implementation:

  • All the components are giving readings to the Arduino, where we are converting these readings into character code in order to send the bytes efficiently and quickly to the computer. Inside a never ending loop, every reading corresponds to a character.
  • A code should run on the computer in order to receive the data that is being sent by arduino via serial ports.
  • For developing the android app we are using android studio and to convert voice to text we are using the standard google api.
  • For speech recognition we are using acoustic-phonetic speech parameters for speaker-independent speech recognition in the android app.
  • We have the AI (artificial intelligence) virtual assistant which uses deep learning techniques like CNN (Convolutional Neural Network) using python to implement the natural language processing.
  • For the augmented reality app we are using Unity IDE and VuforiaSDK (It is an Augmented Reality Software Developments Kit (SDK) for mobile devices that enables the creation of Augmented Reality applications).

Videos

Conversations with personal AI Assistant

Conversations with personal AI Assistant

Controlling appliances using personal AI Assistant

Controlling appliances using personal AI Assistant

Controlling appliances using Augmented Reality

Digital energy reading and cost of appliances connected to J.A.R.V.I.S.
1. Augmented Reality App Development : https://library.vuforia.com/articles/Solution/Compiling-a-Simple-Unity-Project
2. Android App Development : https://developer.android.com/training/index.html
3. ChatBot using neural network : http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/

Custom parts and enclosures

Augmented Reality Android App for speech impaired people
Instructions :

1. Download and install the app
2. Make sure that the phone is connected to the internet
3. Start the app and point it to an appliance which has been configured with Vuforia (the default Vuforia marker is used here)
4. The appliance should start
5. To turn off the appliance, click on the virtual button that appears on the screen whenever phone is pointed towards the device
jarvisaugmented_I2zWAu2jGG.apk

Schematics

Overall Schematics and circuit daigram
Presentation 07 wx3h8wntmy
J.A.R.V.I.S. Android App
Kindly note: Active Internet Connection on phone is required at all times
Instructions :
1. Install the app on an android phone
2. After setting up the hardware and running it successfully, start the app
3. For using Virtual Assistant (Voice), say the keyword, 'Listen Up JARVIS'. This should activate the agent
3.1 Speak your command clearly
3.2 The agent should provide a response
4. To start the augmented reality app, click on the augment button; a blank screen should appear
4.1 Open the Jarvis Augmented App on a different phone (NOTE : The J.A.R.V.I.S. android app should be on a different phone)
4.2 Proceed to instructions given in JarvisAugmented for implementing augmented reality

NOTE : 1. If app is stuck at any point, try restarting the app
2. If a message similar to, 'Could not open Microphone' appears on the screen, restart the app
3. This app is implemented using HC05 Bluetooth module to increase the scope of the project. Hence, HC05 Bluetooth Module is recommended
app-release_eo3LK2AsLV.apk

Code

Arduino code for digital energy reading and cost calculationArduino
This code implements the energy units consumed by an appliance connected to the Arduino.
Instructions :
1. Make the connections and insert a mobile number in the Arduino code
2. Upload the code to an Arduino 101
3. Run the code
4. Send an SMS from any mobile to the mobile number specified in the code in the format #anytext*
5. A reply containing the units consumed and the cost would be received
#include <LiquidCrystal.h>
LiquidCrystal lcd(7,6,5,4,3,2);

int led=13; // declering LED in 13th pin
#define pulsein 8 //defining the pulse input pin 8
unsigned int pusle_count=0; //Declaring Pulse count type as integer
float units=0; //Declaring units as Float
unsigned int rupees=0; //declaring rupees types as unsigned integer

unsigned int temp=0,i=0; //declaring temp& i as unsigned integer

char str[70],flag1=0,flag2=0; // character length as string and declaring flags


void setup() 
{
  lcd.begin(16,2);  // initializing LCD type as 16x2
  Serial.begin(9600); // initializing serial baud rate as 9600
  pinMode(led, OUTPUT);
  pinMode(pulsein, INPUT);
   digitalWrite(pulsein, HIGH);
  lcd.setCursor(0,0);
  lcd.print("Wireless Energy");
  lcd.setCursor(0,1);
  lcd.print("      Meter    ");
  delay(2000);
  lcd.clear();
  lcd.print("By   ");
  delay(2000);
  lcd.clear();
  lcd.print("GSM Initilizing...");
  gsm_init();
  lcd.clear();
  lcd.print("System Ready");
  Serial.println("AT+CNMI=2,2,0,0,0");
  init_sms();
  send_data("System Ready");
  send_sms();
  delay(1000);
  digitalWrite(led, LOW);
  lcd.clear();
}

void loop()    
{
    serialEvent();  // checking serial event
    units=rupees/5.0;  // calculation of unit
    lcd.setCursor(0,0);
    lcd.print("Units:");
    lcd.print(units);
    lcd.print("      ");
    lcd.setCursor(0,1);
    lcd.print("Rupees:");
    lcd.print(rupees);
    lcd.print("      ");
    read_pulse();
    
    if(temp==1)
    {
          check();
    }
}

void serialEvent() // serial event for receiving and checking the incoming sms
{
  while(Serial.available()) 
  {
    if(Serial.find("#"))
    {
      digitalWrite(led, HIGH);
      delay(1000);
      digitalWrite(led, LOW);
      while (Serial.available()) 
      {
      char inChar=Serial.read();
      str[i++]=inChar;
      if(inChar=='*')
      {
        temp=1;
        return;
      } 
      } 
    }
   }
 }

void init_sms() //initializing & configuration of sending message
{
   Serial.println("AT+CMGF=1");
   delay(200);
   Serial.println("AT+CMGS=\"INSERT MOBILE NUMBER HERE\"");
   delay(200);
}

void send_data(String message)
{
  Serial.println(message);
  delay(200);
}

void send_sms()
{
  Serial.write(26);
}

//Reading Pulse from Energy Meter
void read_pulse()
{
    if(!digitalRead(pulsein))
    {
      digitalWrite(led, HIGH);
      units++;
      rupees=units*5;
      while(!digitalRead(pulsein));
      digitalWrite(led,LOW);
    
    }
}

//Checking the status of Message 
void check()
{
     
     if(!(strncmp(str,"units",5)))
      {
        Serial.println("AT+CMGF=1");
   delay(200);
   Serial.println("AT+CMGS=\"INSERT MOBILE NUMBER HERE\"");
   delay(200);
       lcd.setCursor(0,1);
       send_data("Energy Meter Unit:");
       Serial.println(units);
       send_data("\nRupees:");
       Serial.println(rupees);
       delay(200);
       Serial.write(26);
       flag1=1;
       flag2=1;
       temp=0;
      }
     }



void message_sent()
{
  lcd.clear();
  lcd.print("Message Sent.");
  delay(1000);
}

void gsm_init()
{
  lcd.clear();
  lcd.print("Finding Module..");
  boolean at_flag=1;
  while(at_flag)
  {
    Serial.println("AT");
    while(Serial.available()>0)
    {
      if(Serial.find("OK"))
      at_flag=0;
    }
    delay(1000);
  }

  lcd.clear();
  lcd.print("Module Connected..");
  delay(1000);
  lcd.clear();
  lcd.print("Disabling ECHO");
  boolean echo_flag=1;
  while(echo_flag)
  {
    Serial.println("ATE0");
    while(Serial.available()>0)
    {
      if(Serial.find("OK"))
      echo_flag=0;
    }
    delay(1000);
  }

  lcd.clear();
  lcd.print("Echo OFF");
  delay(1000);
  lcd.clear();
  lcd.print("Finding Network..");
  boolean net_flag=1;
  while(net_flag)
  {
    Serial.println("AT+CPIN?");
    while(Serial.available()>0)
    {
      if(Serial.find("+CPIN: READY"))
      net_flag=0;
    }
    delay(1000);
  }
  lcd.clear();
  lcd.print("Network Found..");
  delay(1000);
  lcd.clear();
}
Arduino implementing JARVIS connection to relay using ArduinoArduino
This is a code implementing serial Bluetooth transfer of characters between the android app and the Arduino.
# define CH3 7
# define CH4 4
# define CH5 5
# define CH6 6

 void setup()  
 {  
  Serial.begin(9600);  
  pinMode(CH3, OUTPUT);
  pinMode(CH4, OUTPUT);
  pinMode(CH5, OUTPUT);
  pinMode(CH6, OUTPUT);
   //Provide power to both LEDs
   
   
   //Turn OFF any power to the Relay channels
   digitalWrite(CH3, HIGH); 
   digitalWrite(CH4, HIGH); 
   digitalWrite(CH5, HIGH); 
   digitalWrite(CH6, HIGH); 
   delay(2000); //Wait 2 seconds before starting sequence
  
 }  
 void loop()  
 {  
       
       
   delay(100);
   
   if(Serial.available())
   {
     
  //Serial.println(sum1);
    
    char c = Serial.read();  
    Serial.print(c);
   if(c=='b')
    {
      //coffee off
    }
   else if(c=='c')//light on
    {digitalWrite(CH4  ,LOW);
       delay(100);}
   else if(c=='d')//fan on
    {digitalWrite(CH6,LOW);
       delay(100);}
   else if(c=='k')//coffee on
    {digitalWrite(CH5,LOW);
       delay(100);}
   else if(c=='g')//coffee off
    {digitalWrite(CH5,HIGH);
       delay(100);}
   else if(c=='e')//light off
    {digitalWrite(CH4,HIGH);
       delay(100);}
     else if(c=='f')//fan off
    {digitalWrite(CH6,HIGH);
       delay(100);}
     else if(c=='z')//goodbye
    {digitalWrite(CH6,HIGH);
     digitalWrite(CH4,HIGH);
     digitalWrite(CH3,HIGH); 
     digitalWrite(CH5,HIGH);
     delay(500);
     digitalWrite(CH5,LOW);
     delay(500);
     digitalWrite(CH5,HIGH);
     delay(500);
     digitalWrite(CH5,LOW);
     delay(500);
     digitalWrite(CH5,HIGH);      
     delay(10000);
     }  
    
  }  
 } 
 
  

Comments

Similar projects you might like

Flying Drone Mjolnir With Hacked RC Transmitter

Project in progress by Allen Pan

  • 6,896 views
  • 0 comments
  • 28 respects

[App Inventor IoT ]LED fading (PWM)

Project tutorial by CAVEDU Education

  • 269 views
  • 0 comments
  • 5 respects

Remote System Monitor

Project tutorial by Stephen Harrison

  • 2,611 views
  • 0 comments
  • 5 respects

Integrate Node-Red with RAK811 LoRA node using the TTN

Project tutorial by Naresh krish

  • 200 views
  • 0 comments
  • 3 respects

Fox Advisor

Project tutorial by Celia Garrido Hidalgo

  • 1,157 views
  • 0 comments
  • 5 respects

Soil Control Grid: From The Ground To The Cloud

Project tutorial by Andrea De Gaetano

  • 130 views
  • 0 comments
  • 2 respects
Add projectSign up / Login