A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas For Hearing Impaired and Non-Verbal People

Volume 9, Issue 4, April – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24APR246
A Canvas of Air and Signs: Integrating

Voice Activated Hand Sign Recognition and Air
Canvas for Hearing Impaired and Non-Verbal People
Pratham Kumar1; Rishav Sharma2; Shruti3; Jaya Shree4; Virat Tiwari5; Varun Mishra6; Soumya Upadhyay7
12345
Student, Department of Information Technology, College of Engineering Roorkee, Uttarakhand, India
6,7
Faculty, Department of Information Technology, College of Engineering Roorkee, Uttarakhand, India
Abstract:- Moving hands in air and watching your screen of hand gesture recognition will surely be a great help to the
writes for you and at the same time making different Hearing Impaired and Non-Verbal community. Hand gesture
Signs with hands and the system displays it as well as recognition can be integrated into communication devices,
speak what Sign you are making.This seems to be very such as smartphones, to provide Hearing Impaired
futuristic approach towards the realm of Image individuals with alternative means of communication.
Processing and Gesture Recognition . In this paper, we
present a very interesting and a novel approach towards  Computer Vision
an interactive learning platform where one can draw the Computer vision is a technology or a field of Artificial
content on screen while moving their hand in air and can Intelligence that enables Machines to Look through the
also use hand sign language to communicate with an ease World as the humans do by enabling the computer to identify
with the Hearing Impaired and Dumb Community. Our the objects and humans through images and videos. Its basic
system combines both technologies to create a smooth and functionality covers Acquiring an image , Processing the
engaging experience for users. It can be used in image , understanding the image.
interactive art setups or virtual reality setups. Air canvas
enables users to draw and manipulate digital content in  MediaPipe
mid-air with object tracking using Computer Vision and MediaPipe is an open source framework which is used
Mediapipe framework, while hand gesture recognition for making perception pipelines to perform time series data
allows for real-time interpretation of Hand Signs to like images , videos, etc. It was developed by google for real
perform actions or commands within the system. This time analysis of videos and audio on Youtube. Mediapipe
Model not only recognizes the sign but also speaks it loud Provides a strong Toolkit for building applications related to
using pyttsx3 a text-to-speech conversion Library, face detection , hand tracking , etc and more.
ensuring a good communication between a normal
human and people with Non-Verbal and Hearing  Gesture Recognition
Impaired disability. To enhance the performance of the Gesture Recognition is a field of research in Computer
model We validate the model with a real dataset trained Science and Technology that tries to detect or recognize and
by us. This training was essential for refining the interpret Gestures made by Humans with the use of Machine
accuracy and efficiency of the model. learning mathematical Algorithms. These gestures can be
made with hands , fingers , face , etc.The main objective of
Keywords:- Air Canvas, Image Processing, Gesture gesture recognition is to avoid the use of traditional input
Recognition, Real-Time, Virtual Reality, Computer Vision, devices like mouse , keyboard , etc.
Mediapipe, Pyttsx3, Dataset.
 Pyttsx3
I. INTRODUCTION Pyttsx3 is a Text-to-Speech library in python makes
your program to speak text Aloud. With Pyttsx3 you can
In recent years, advances in interactive technologies easily convert text into voice in your Python Projects . This
have revolutionized the way humans interact with computers makes it a useful tool for creating applications that requires
and digital content. Among these technologies, Air Canvas voice activated / enabled outputs like voice assistant , etc.
and Hand Gesture Recognition gives a new way to humans to
get themselves engaged in such an interactive and intuitive II. LITERATURE REVIEW
user interface. Air canvas system enables to draw, sketch and
manipulate the digital content by only just waving their hands As many researches have been carried out on both Air
in the air. This system eliminates the use of styluses or any canvas and Hand Sign Detection and both of them have their
other tablets and hardware devices. This system has an significance in their fields seperately. But none of researches
outstanding integration of hand gesture recognition with air have ever been made out which shows the integration of the
canvas that enables computers to interpret and respond to same. Some researches that have been carried out before on
gestures made by the user's hands, opening up new both the areas are mentioned below .
possibilities for natural and intuitive interaction. The feature
IJISRT24APR246 www.ijisrt.com 172

Air Canvas : Draw in Air by Sayli More , Prachi techniques , and an Elliptical Kernal is subjected in the series
Mhatre,Shruti Pakhare, Surekha Khot [1] in which they of events for further procedures.
developed a canvas for communicating a concept or
presenting an idea in real world space.With air canvas, they III. METHODOLOGY
accomplished a sans hand drawing that utilises open cv to
recognise the client’s point finger. By doing so,lines can be Our integrated system primarily focuses on
drawn any place the clients want.By using color implementing air canvas and hand gesture recognition
tracking,contour detection, algorithmic applications doesn’t involve complex algorithms, it utilizes
optimization ,trackbars modules for the proposed system. various techniques.
Air Canvas through Object Detection using open cv in  Open CV –

python by Harshit Rajput,Mudit Sharma,Twesha OpenCV (Open-Source Computer Vision Library) is an
Mehrotra,Tanya Maurya[2] which enables users to draw in open-source computer vision and machine learning software
mid air using a stylus on a virtual canvas.The System library, It helps in achieving functionalities such as image
incorporates Object Detection techniques in OpenCV to track and video analysis, object detection and tracking, facial
the stylus’s position and real time drawing .They trained a recognition and machine learning algorithms. We have used
Haar cascade classifier to detect the marker in the video it in our system to capture the real time video frames as we
stream.When the marker is detected,they used it's position to have to capture the hand’s movement for both our features.
track the stylus and update the canvas in real time.
 Hand Landmark Detection –
Air Canvas by Aniket Sandbhor , Prasad Rane , Media Pipe is an open- source framework developed by
prathamesh Shirole , Pawan Phapale[3] in which they utilised Google for building cross-platform multimodal perception
camera and the screen for reading inputs and displaying pipelines. It is used for the following.
outputs.They used their hand fingers to draw required shapes
on the screen. To challenge the potential of traditional  Hand Landmark Detection:
writing methods .It is very well carried out by using python Media Pipe is employed in our project to detect and
libraries named as OpenCV and Mediapipe which are ready localize key points i.e. landmarks on the user’s hand in real-
to used ML solutions for recognition and tracking. time. This landmark detection helps us to track the position
of the hand and fingers of the user for both gesture
Real-Time Sign Language Recognition System For Deaf recognition and for drawing control based on finger position.
And Dumb People by Furkan , Ms. Nidhi Sengar [4] in which
the research proposes a system that utilizes algorithms for an  Real Time Performance:
application which helps in recognizing the various Indian The efficient implementation of Media Pipe enables
signs named as Indian Real Time Sign Language.The system users to experience the real time hand landmark detection and
worked on 9 classes which came out to be 95% accurate with tracking providing a smooth and responsive experience
their images captured on every possible angle and tested on without significant delays.
45 different types of output. Image Acquisition, Feature
Extraction, Orientation Detection and Gesture Recognition  Colour Management –
were the four major algorithms the proposed system Numerical Python is a python library that supports
consisted. multi-dimensional arrays and matrices. It helps in performing
wide range of mathematical functions that includes arithmetic
Hand Gesture Recognition and Voice Conversion for operations, trigonometric functions, exponential and
Deaf and Dumb by R.Anusha , K.Dhanalakshmi , S. logarithmic functions, etc.
Shravanthi , G. Hymanjali, T. Hemalatha [5] have propose a
system in which they proposed a novel Convolutional neural  We had Utilized NumPy Majorly in Air Canvas
network (CNN) using multi-channels of video streams , Application for the Following Parameters:
including colour information , depth clue , and body joints
positions are used as input to the CNN in order to make it  Drawing Operations:
happened. They converted the identified gesture’s text to NumPy is used to update the array with the
speech . corresponding pixel values to reflect the drawn strokes.
Hand Gesture Recognition for Deaf and Dumb People  Colour Management:
by Mahesh Kumar.D [6] have proposed a system for NumPy is used to facilitate colour management. It
identifying signing languages that is based on American Sign allows us to manipulate the colours.
Language . The datasets they used consist of 2000 images of
American Sign motions , in which 1600 were used for  Drawing Management –
training and 400 for Validating . The dataset divided into
80:20 proportion , 80 for training and 20 for Validating. A  Deque-based Stroke Storage:
CNN model is utilized to forecast hand motions. They used Deques (double-ended queues) is a data structure that is
HSV ( Hue, Saturation , Value) Method for recognising used to store the drawing strokes as a sequence of points. It
Backgrounds from the images. Segmentation , Morphological

allows to add and remove the points, enabling the smooth  Gesture Classification –
experience. Machine Learning Models: Hand gesture recognition
involves training machine learning models on labelled
The Flowchart depicted in (Fig.1) show the gesture data. These models learn to recognize specific
comprehensive view of the working of various algorithms gestures based on the positions and movements of hand
and commands togertherly which lead us to the perfect landmarks.
working of Air Canvas . This diagram meticulously outlines
the sequential workflow and the interdependencies of each  Classification Algorithms –
process, highlighting precision with which they are Classification Models: Various classification algorithms,
orchestrated to achieve the desired outcome. including deep learning models such as CNN, are employed
to classify the hand gestures made by the user.
 Custom Dataset –
As our project has two integrated parts namely air
canvas and hand gesture recognition, the air canvas doesn’t
require any dataset as it do not involve any machine learning
techniques. Dataset used for hand gesture recognition was
made by ourselves only.
 Text to Audio : Pyttsx3 –

Our Hand gesture recognition system has a very helpful
feature that generate the recognized result as an audio.
Fig 1 Flowchart of Air Canvas Fig 2 Flowchart of Hand Sign Detection

The Flowchart depicted in (Fig.2) show the V. CONCLUSION

comprehensive view of the working of various algorithms
and commands togertherly which lead us to the perfect The paper, "A Canvas of Air and Signs," proposed an
working of Air Canvas. innovative system that integrates air canvas technology with
real-time sign language recognition. This first-of-its-kind
IV. RESULT approach creates a dynamic learning platform and pushes the
boundaries of human-computer interaction. Utilizing
The frame capturing process is matched with the sophisticated machine learning and computer vision
movement of the object onto the canvas.Thus,the system is algorithms, the system empowers users to not just manipulate
able to analyse the live video frames of the real time and and create digital content in mid-air, but also to seamlessly
presenting the object movements on the canvas and creating translate sign language into spoken words. This innovation
an interactive user experience.We can use the same object bridges the communication gap for the Hearing Impaired and
detection techniques to detect the marker.When the position mute community, providing them with a powerful tool for
of the marker is detected it can be used for detecting the self-expression and interaction with the broader world. The
position of the pattern that we are making on the canvas and proposed system has the potential to unlock groundbreaking
update the virtual canvas in the live feed. (Fig. 3) educational opportunities and fundamentally reshape how
humans interact with computers.
REFERENCES
[1]. Sayli More , Prachi Mhatre,Shruti Pakhare, Surekha

Khot “ Air Canvas : Draw in Air” Volume : 09,
February 2022.
[2]. Harshit Rajput , Mudit Sharma , Twesha Mehrotra ,
Tanya Maurya “Air Canvas through Object Detection
using open cv in python” Volume : 11, May 2023.
[3]. Aniket Sandbhor , Prasad Rane , prathamesh Shirole ,
Pawan Phapale”Air Canvas” Voume : 11 , April 2023.
[4]. Furkan , Ms. Nidhi Sengar “Real-Time Sign Language
Recognition System For Deaf And Dumb People”
Volume : 9 , June 2021
[5]. R.Anusha , K.Dhanalakshmi , S. Shravanthi , G.
Hymanjali, T. Hemalatha “Hand Gesture Recognition
and Voice Conversion for Deaf and Dumb” Volume :
Fig 3 Output of Air Canvas 3 , August 2022
[6]. Mahesh Kumar.D “Hand Gesture Recognition for
The camera starts capturing the movement of the hand Deaf and Dumb People”Volume : 10, Sep 2022.
enabling it to capture it the live feed. It is done by capture the [7]. Alex Ming Hui Wong , Dae-Ki Kang “Stationary
real time video frame by frame.After capturing the gesture,it Hand Gesture Authentication Using Edit Distance on
recognises the gesture made in the frame and then display it Finger Pointing Direction Interval” Volume : 2016.
in the form of text in the live feed.Lastly,the text displayed is [8]. Prof. S.U. Saoji , Nishtha Dua , Akash Kumar
converted into voice and the user is able to hear the name of Choudhary , Bharat Phogatv “ Air Canvas Application
the sign made by using the hand.(Fig. 4) Using OpenCV And Numpy in Python” Volume : 8 ,
Aug 2021
[9]. Ganesh Gaikwad , Vaibhav Sonawane , Siddhant
Sonawane , Pushpak Khadke “ Virtual Painting”
Volume : 04 , May 2022
[10]. https://teachablemachine.withgoogle.com/
[11]. https://www.ijctr.com
[12]. https://www.researchgate.net
Fig 4 Output of Hand Sign Detection

A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas For Hearing Impaired and Non-Verbal People

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas For Hearing Impaired and Non-Verbal People

Uploaded by

Copyright:

Available Formats

Volume 9, Issue 4, April – 2024 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24APR246

A Canvas of Air and Signs: Integrating

IJISRT24APR246 www.ijisrt.com 172

Air Canvas through Object Detection using open cv in  Open CV –

IJISRT24APR246 www.ijisrt.com 173

 Text to Audio : Pyttsx3 –

Fig 1 Flowchart of Air Canvas Fig 2 Flowchart of Hand Sign Detection

IJISRT24APR246 www.ijisrt.com 174

The Flowchart depicted in (Fig.2) show the V. CONCLUSION

[1]. Sayli More , Prachi Mhatre,Shruti Pakhare, Surekha

Fig 4 Output of Hand Sign Detection

IJISRT24APR246 www.ijisrt.com 175

You might also like