A 23-year old Pakistani entrepreneur has developed an application for people with hearing disabilities. The application, known as Edu-Aid, translates sign language into spoken language.
The software of the application developed by Syed Faizan Hussain makes use of computer vision, natural language processing and voice commands.
In most American Sign Language (ASL) translators, one is completely dependent on hardware, which includes 3D cameras, flex sensors or sign language gloves. Hussain says instead of depending on hardware, he wanted to focus on the software side and develop something which made use of existing hardware, such as cameras available in mobile phones.
Read more: Games for Children With Special Needs
With the help of a single lens, he successfully identified some elementary ASL signs used for communication in a conversation.
The software uses the webcam or mobile camera to capture images of the hand as it gesticulates. It then uses an algorithm to analyze hand movements and tries to find the closest sign available in its library.
Hussain says he initially used an open source library to identify ASL signs. Later, he built a custom library which also included Pakistan Sign Language (PSL) gestures as well. “Some ASL signs are quite different from the PSL signs but basic communication signs are similar,” he says.
At this point, the application does not have full sentences to explain what a gesture means. Hussain explains, “We apply natural language processing and apply proper grammar. Then we make use of the Bing Translate library to read it out loud to the user.”
For example, if someone raises their forefinger and twists its tip on their cheek, the application will compare the gesture with the sign language library and tell the user that the person is asking for candy.
A limitation of the application is that if the colour of the hand matches the environment, it will be harder for the software to analyze gestures. One must have an ultra-red LED to illuminate the ambiance so the camera can actually see the hand clearly.
Edu-Aid claims to be a valuable resource for students with hearing disabilities and their families, as well as those who want to learn a new language.
Hussain is currently working on other projects to solve local problems in Pakistan. He recently received Her Majesty The Queen’s Young Leaders Award for his work to support the use of technology to create solutions to health problems. The award was given to leaders who are focused on “making lasting change in their community and beyond.”