In the spirit of innovation and creativity, my project embarks on an exciting journey to revolutionise communication accessibility for the Maltese Deaf community. My name is Mac Patrick Gauci, and my AI-powered assistive technology is designed to translate spoken English into Maltese Sign Language (LSM) through a dynamic digital avatar, marking a significant leap forward in bridging communication gaps.

Project Overview

Imagine a world where communication barriers dissolve, where technology facilitates seamless interactions between the hearing and hard-of-hearing communities.

This vision drove the development of my project, integrating cutting-edge motion capture, automatic speech recognition (ASR), and natural language processing (NLP) technologies. The result is a pioneering prototype that translates spoken language into sign language, bringing conversations to life through a digital avatar.

The journey began with meticulous planning and setup. Leveraging Python, Visual Studio, and Unity 3D, I created an environment where advanced technologies could converge. The avatar, developed using Mixamo, was animated to reflect the nuances of LSM gestures accurately. As the heart of the system, Whisper provided continuous audio recording and transcription, converting spoken English into text, which was then processed using sophisticated NLP techniques to ensure precise and contextually appropriate translations.

 

Development and Evaluation

The development process unfolded in phases, each critical to the project’s success. Initial setup involved creating a robust development environment and configuring essential tools. Core development saw the integration of Whisper for real-time audio transcription and the application of NLP techniques to prepare text for animation. A significant milestone was establishing seamless communication between Unity and Python, enabling real-time data exchange for fluid animations. Motion capture technology at MCAST’s Applied Research and Innovation Centre played a pivotal role in recording LSM gestures. These gestures were meticulously edited and re-targeted to the digital avatar, ensuring accurate and lifelike sign language representation. The true test of the prototype came during structured user engagement sessions. Members of the Maltese Deaf community and professional interpreters interacted with the technology, providing invaluable feedback. Their insights highlighted the system’s strengths and areas needing improvement, such as sign interpretation accuracy, incorporation of facial expressions, and avatar movement fluidity.

 

Key Findings and Future Directions

 

The feedback was overwhelmingly positive, confirming the prototype’s potential in enhancing communication accessibility. Users appreciated the system’s ability to perform basic translations and its use of semi-realistic 3D animations for LSM gestures. However, they also emphasised the need for improvements to make sign language interpretation more natural and comprehensible. Future iterations of the project will focus on refining these aspects. By enhancing sign accuracy, integrating facial expressions and lip reading, and expanding the participant base for broader feedback, the goal is to create an even more effective tool.

The ultimate aim is to achieve greater realism and naturalness in the avatar’s movements, ensuring the technology meets the diverse needs of the Deaf community.

 

Broader Implications and Conclusion

 

The implications of this research extend far beyond the Maltese Deaf community. It sets the stage for global advancements in assistive technologies, fostering inclusivity and accessibility in various domains such as education, healthcare, and public services. By pushing the boundaries of innovation and creativity, this project aims to contribute to a world where communication is truly universal.

 

In conclusion, my AI-powered assistive technology project marks a significant step towards improving communication accessibility for the Maltese Deaf community. It embodies the essence of innovation and creativity, promising a more inclusive future where technology bridges gaps and brings people closer together