DIY Gesture Controlled Arduino Based Air Mouse using Accelerometer

Ever wondered how our world is moving towards the immersive reality. We are continuously finding new ways and methods to interact with our surrounding using virtual reality, mixed reality, augmented reality etc. New devices are coming out every day with these fast pacing technologies to impress us by their new interactive technologies.

These immersive technologies are used in gaming, interactive activities, entertainment and many other applications. In this tutorial, we are going to know about such an interactive method which gives you a new way to interact with your system instead of using a boring mouse. Our gaming geeks must be knowing that few years back Nintendo a gaming company sells out an idea of a 3D interactive method to interact with their consoles with the help of a handheld controller known as Wii controller. It uses the accelerometer to locate your gestures for a game and send it to the system wirelessly. If you want to know more about this technology you can check out their patent EP1854518B1, this will give you a complete idea of how this technology works.

 

Inspired by this idea we are going to make an "Air mouse", to interact with systems just by moving the console in the air, but instead of using 3-dimensional coordinate references, we are only going to use 2-dimensional coordinate references so we can imitate the actions of the computer mouse since the mouse works in two dimensions X and Y.

The concept behind this Wireless 3D Air Mouse is very simple, we will use an accelerometer to get the value of the acceleration of the actions and motions of the “Air mouse” along the x and y-axis, and then based on the values of the accelerometer we will control the mouse cursor and perform certain actions with the help of the python software drivers running on computer.

 

Pre-requisites

• Arduino Nano (any model)
• Accelerometer ADXL335 Module
• Bluetooth HC-05 Module
• Push buttons
• Python Installed computer

To learn more about installing python in computer follow the previous tutorial on Arduino-Python LED Controlling.

 

Circuit Diagram

 

To control your computer with the movements of your hand you need an accelerometer that gives out the acceleration along the X and Y-axis and to make the whole system wireless a Bluetooth module is used to transfer the signal wirelessly to your system.

Here an ADXL335 accelerometer is used, it's a MEMS-based triple axis module outputting the acceleration along X, Y, and Z-axis but as told previously for controlling the mouse we would be only needing the acceleration only along the X and Y-axis. Learn more about using ADXL335 accelerometer with Arduino with our previous projects:

• Arduino Based Vehicle Accident Alert System using GPS, GSM and Accelerometer
• Ping Pong Game using Arduino and Accelerometer
• Accelerometer Based Hand Gesture Controlled Robot using Arduino
• Earthquake Detector Alarm using Arduino

 

Here the Xout and Yout pin of accelerometer are connected to the Analog, A0, and A1 pins of Arduino and for transmitting the signals from the Arduino to the system Bluetooth module HC-05 is used here, since the Bluetooth works over the Tx and Rx pin connections, so we use software serial pins D2 and D3. It is connected using Software serial because if we connect the Bluetooth with hardware serial and start getting the readings over the python console it would show errors for the mismatch baud rate as the Bluetooth would be communicating with the python on its own baud rate. Learn more about using Bluetooth module by going through various Bluetooth based projects using different microcontrollers including Arduino.

Here we have used three push buttons - one for triggering the Air mouse, and other two for left and right click as shown in the below image:

 

Process Flow for the Air Mouse

The flow chart shows the process flow of Arduino based Air Mouse:

 

1. The system continuously checks for the mechanical trigger to be pressed until it is not pressed we can work out normally with the computers mouse.

2. When the system detects button press, the control for the mouse transferred to the air mouse.

3. As the trigger button is pressed the system starts to transfer the readings of the mouse to the computer. The system reading consists of the accelerometer readings, and the readings for the left and right click.

4.  The system readings consist of the data stream of 1 byte or 8 bits, in which the first three bits consist of the X-coordinates, second three bits consist of the Y-coordinates, the second last bit is the status bit for getting the status of the left click of the mouse and the last bit is the status bit for getting the status of the right click.

5. The value of the first three bits i.e. X-coordinate can range from 100<=Xcord<=999, whereas the value for the Y-coordinate can range from the 100<=Ycord<=800. The values for the right click and left click are the binary values either 0 or 1 in which 1 indicates the click is made and 0 that click is not made by the user.

6. To not let the bouncing of the button affect the position of the cursor a known delay of 4 seconds is kept after every click of the trigger button of the mouse.

7. For the right and left click in the air mouse, we have to first press either left or right pushbutton, and after that, we have to press the trigger button to move to the position of the air mouse where we want.

 

Programming the Arduino for Air Mouse

The Arduino should be programmed to read the acceleration values in the X and Y-axis. The complete program is given at the end, below are the important snippets from the code.

 

Setting up the global variables

As previously said we will hook up the Bluetooth module with the software serial pins. So to set the software serial we need to declare the library of software serial and set up the pins for Tx and Rx. In Arduino Nano and Uno Pin 2 and 3 can work as a software serial. Next, we declare the Bluetooth object from the software serial library to set up the pin for the Tx and Rx.

#include <SoftwareSerial.h>
const int rxpin = 2, txpin = 3;
SoftwareSerial bluetooth(rxpin, txpin);
const int x=A0;
const int y=A1;
int xh, yh;
int xcord, ycord;
const int trigger = 5;
int lstate = 0;
int rstate = 0;
const int lclick = 6;
const int rclick = 7;
const int led = 8;

 

Void setup()

In the setup function, we are going to set the variables to tell the program whether they will act as input or output. The trigger button would be set up as input pull-up, and the left and right clicks are declared as input and set up as High to make them act as input pullups.

Also set the baud rate for the serial and Bluetooth communication to 9600.

  void setup()
{
  pinMode(x,INPUT);
  pinMode(y,INPUT);
  pinMode(trigger,INPUT_PULLUP)
  pinMode(lclick,INPUT);
  pinMode(rclick,INPUT);
  pinMode(led, OUTPUT);
  digitalWrite(lclick,HIGH);
  digitalWrite(rclick,HIGH);
  Serial.begin(9600);
  bluetooth.begin(9600);
}

 

Void loop()

As we would need trigger button to tell when we need to send the system the data stream, so we set up the whole code inside the while loop which will continuously monitor the digital state of the pull-up trigger, as it goes low it will pass it further for the processing.

 

As we have attached a LED to let us know the status of the system for when the trigger button is pressed, we initially set the led to low outside the while loop as it's default condition and high inside the while loop which will light up the led whenever the trigger button is pressed.

To read the status of the left and right click button we have globally declared two variables lclick and rclick whose values initially were set up to 0.

And in the loop, set the value of those variables according to the digital status of the left and right click button to check whether the buttons are pressed or not.

We would be reading the values of the X and Y out pins of the accelerometer using the analogRead function and would map those values to the screen size to get the mouse pointer moving across the whole screen. Since the screen size is the pixels in the screen, we need to set it up accordingly and as we need the output value to be three digits we have deliberately set up the range for the X as 100<=X<=999 and similarly the value for the Y as 100<=Y<=800. Remember, the pixels are being read from the top left corner i.e. the top left corner has the value (0,0), but since we have declared three digits for the x and y our values would be read from the point (100,100).

Further, print the value of the coordinates and the status of the click over the serial and Bluetooth with the help of Serial.print and bluetooth.print functions they help in getting the values on serial monitor and over your system via Bluetooth.

At last, due to the bouncing of a button a single value may be repeated which would cause a mouse cursor to linger over a single position, so to get rid of this we have to add this delay.

void loop()
{
digitalWrite(led,LOW);
  while(digitalRead(trigger)==LOW)
  {
    digitalWrite(led, HIGH);
    lstate = digitalRead(lclick);
    rstate = digitalRead(rclick);
    xh=analogRead(x);
    yh=analogRead(y);
    xcord=map(xh,286,429,100,999);
    ycord=map(yh,282,427,100,800);
    Serial.print(xcord);
    Serial.print(ycord);
    if (lstate == LOW)
    Serial.print(1);
    else
    Serial.print(0);
    if (rstate == LOW)
    Serial.print(1);
    else
    Serial.print(0);
    bluetooth.print(xcord);
    bluetooth.print(ycord);
    if (lstate == LOW)
    bluetooth.print(1);
    else
    bluetooth.print(0);
    if (rstate == LOW)
    bluetooth.print(1);
    else
    bluetooth.print(0);
    delay(4000);
   } }

 

Python Driver Script for Computer

As of now, we have completed with the hardware and its firmware part, now to get the air mouse working we need to have a driver script which could decode the signals from the air mouse into the cursor movements, so for this, we have chosen Python. Python is a scripting language and by scripting here we mean that it helps us to get the control of the other program, as here we are controlling the mouse cursor.

So open your python shell and get the following libraries installed using below commands:

pip install serial
pip install pyautogui

 

The serial is a library for python which helps us to get the data from serial interfaces such as com ports and also lets us manipulate it whereas pyautogui is library for python to get control over the GUI features, in this case, mouse.

 

Now let's get to the code for the drivers, the first thing we need to do is importing the serial and pyautogui libraries, and then from the serial library, we have to set the com port for the communication with a baud rate of 9600, the same as Bluetooth.serial is operating at. For this you have to get connected the Bluetooth module to your system and then in system settings you have to check out which com port is it connected to.

Next thing is to read the serial communication from the Bluetooth to the system and to keep it going continuously keep rest of code in a continuous loop with the help of while 1.

As said previously that Arduino is sending out 8 bits, first 6 for the coordinates and the last two for the status of the click buttons. So read all the bits with the help of ser.read and set up its length to 8 bits.

Next, divide the bits for the cursor coordinates and clicks by slicing them over, and then further slice down the cursor bits into X and Y coordinates separately. Same goes for the left and right click.

Now from the communication, we are getting a byte string and we need to convert it into an integer so that they can fit in for the coordinates, we do this by decoding them and then typecasting them into integers.

Now to move the cursor we use the pyautogui moveto function, which takes as arguments those integer coordinates and moves the cursor to that position.

Next check for the clicks, we do this by using the last two bits and pyautogui's click function, its default click is left one, however we can set it into right by declaring button value to right, we can also define the number of clicks to set it off to a double click by setting up the clicks parameter to 2.

Below is the complete Python code to be run on the computer:

import serial
import pyautogui
ser=serial.Serial('com3',9600)
while 1:
    k=ser.read(8)
    cursor=k[:6]
    click=k[6:]
    x=cursor[:3]
    y=cursor[3:]
    l=click[0]
    r=click[1]
    xcor=int(x.decode('utf-8'))
    ycor=int(y.decode('utf-8'))
    pyautogui.moveTo(xcor,ycor)
    if l==49:
        pyautogui.click(clicks=2)
    elif r==49:
        pyautogui.click(button='right', clicks=2)

 

Testing the Arduino Air Mouse

So for operating the Air Mouse attach a power source to it. It can be from the Arduino Nano USB slot or from the 5v regulated power supply using 7805 IC. Then run the python driver script by setting the com port that your Bluetooth is connected to. As the script runs you would see a time lag in the blinking of the Bluetooth it means it's connected to your system. Then for operating it click the trigger button and you would see the position of the coordinates would change and if you want the left or right click, then first press left or right pushbutton and trigger button together, you would see the action of the click at a changed location of the cursor.

 

Check the detailed working Video below.