Hearing loss and auditory impairments affect approximately 2.5 million individuals in Indonesia, forming a gap in communication that is undeniably an issue. Sign language is not widely used in the country, and the scarcity of sign language interpreters further compounds this challenge.
Muhammad Alan Nur, founder and CEO of Tulibot, first learned of this issue during his first semester as an undergraduate at a college in Surabaya. While dining at a restaurant in the vicinity, he found it challenging to interact with a waiter with a hearing disability. This experience inspired him to develop innovative solutions that would not only aid understanding between people regardless of hearing ability but also pave an avenue to empower the community using technology.
Developing the smart glasses
Nur and his team at Tulibot initially experimented with a variety of potential solutions that could bridge the communication barrier between those with hearing disabilities and the rest of society. This included integrating smartphones with virtual reality (VR) glasses to provide visual outputs, smart gloves to translate sign language into audio, and more. Eventually, smart glasses were determined to be the most holistic solution.
“Smart glasses provide a distinctive advantage by allowing deaf individuals not only to access subtitles but also to perceive body language and facial expressions during conversations, offering so much more than the conventional smartphone transcription solution,” Nur said.
Tulibot’s smart glasses work in tandem with a smartphone, acting as a secondary monitor. The smartphone records and transcribes speech in real time, before transmitting the transcription that is displayed on the glasses. This process boasts a latency rate that can go as low as 10 milliseconds, depending on connection speed.
Drawing from the open-source Common Voice repository provided by Mozilla, which features a large amount of voice data in a diverse range of languages, Tulibot’s smart glasses can understand the languages that are commonly used in Indonesia, including Sudanese, Javanese, and Bahasa Indonesia. According to Nur, Tulibot refines its dataset on a regular basis, hoping to achieve an even higher level of transcription accuracy.
The smart glasses have been designed to work in almost any scenario, taking into consideration factors such as changes in lighting, compatibility with eyeglasses, and multi-participant situations. These glasses offer a spectrum of settings, including options to modify brightness and adjust the lenses according to the wearer’s vision correction needs.
For the smart glasses to function in scenarios with multiple speakers, Tulibot employs a dual-pronged strategy. The first approach harnesses the power of a cloud-based speaker diarization engine, enabling the smart glasses to discern and distinguish the speakers’ identities, and labeling the subtitles accordingly. As an alternative solution, Tulibot also offers a localized option whereby wearers can utilize an app to invite other individuals with smartphones to join a chat group with features akin to a WhatsApp group. Smartphones in the same group can help to identify and differentiate the speakers from one another.
Making an impact
Following the launch of pilot projects run in collaboration with energy company PT Pertamina, Tulibot has garnered favorable responses from some of its users. These initiatives have led to wider adoption of its smart glasses, including the Cue Café which is managed by the Bitung Caring Deaf community (Kaleb) and supported by PT Pertamina. The cafe hires baristas with hearing disabilities, who have offered positive feedback that the smart glasses are convenient and effective.
A customer of Tulibot, who was a university student, shared an anecdote of the smart glasses’ usefulness in her discussions and academic endeavors, demonstrating the technology’s impact. Such feedback has served as a driving force for the company to persist in its mission of advancing communication for individuals with hearing impairments.
Furthermore, Tulibot is collaborating with the Indonesian government to identify key locations where their solution can yield a significant impact, focusing on communities with a higher prevalence of hearing disabilities.
Future outlook
Moving forward, Tulibot is ambitiously seeking to transform its smart glasses into a lifestyle brand, transcending beyond the initial purpose of serving the deaf and hearing-impaired community. Nur envisions integrating Tulibot’s smart glasses into everyday life, subtitling experiences in real time to make interactions more engaging than ever before.
Another focus of Tulibot is to reduce the weight of its smart glasses. Presently, each pair weighs approximately 79 grams, surpassing the average weight of 25–50 grams for conventional eyewear. Striking an optimal balance between weight and functionality is pivotal, ensuring that Tulibot’s smart glasses are lightweight and comfortable to wear while still accommodating the necessary processing power to serve their intended purpose in daily life.
Tulibot’s primary target market is presently the hearing-impaired Indonesian community, at least until 2024. Beyond this timeframe, the company plans to expand into other Asian countries, distributing its app and smart glasses to communities in these countries, improving accessibility and empowerment for a broader audience.
With preorders streaming in from diverse regions such as India, Singapore, and the UK, Tulibot’s technology appears to be striking a chord across several parts of the globe, hinting at a bright future for more inclusive communication.
Tulibot was a participant in OPPO Inspiration Challenge 2023, an initiative aimed at empowering global entrepreneurs and technology professionals to create innovative solutions and shape a better future for everyone. Winners of the challenge can access various forms of support from OPPO to fast-track their growth.