Apps for Accessibility: Otosense Sound Recognition

Posted by on / 0 Comments

One of the challenges that deaf and hard-of-hearing people deal with on a daily basis is how to tell what’s going on in and around their homes without being able to listen to the sounds. Audio cues from smoke detectors, baby monitors, dripping faucets, and other noises might be things that hearing people take for granted, but they can present a complex issue for people who aren’t able to hear.

Mobile application developers are hoping to solve this problem with sound recognition apps, which can run in the background and alert users to noises in their environment. Otosense Inc. makes an app for Android and iOS that allows users to record sounds, and then receive a variety of alerts when those same noises are detected by the microphone on their mobile device.

The Otosense app comes with two different smoke detector noises already in its library, and a simple interface for adding new sounds. By tapping a button and then triggering the sound, the user can teach Otosense to recognize common noises – for instance, the doorbell, oven timer, or end of the wash cycle. The app uses location services to determine whether the user is at home or away, so it’s possible to set certain alerts, like a doorbell, to work only inside the home – other sounds, such as emergency sirens, can be set up in the app so that the user only receives alerts while outside the home.

Otosense can generate a variety of alerts for each sound added to the library – in addition to push notifications and engaging the flash on the user’s mobile device, the app can also connect to Philips Hue light bulbs and make the lights flash when a sound is detected.

On the surface, sound recognition apps are a simple way to help deaf and hard-of-hearing people use their mobile devices to hear the sounds in their environment. An afternoon testing the Otosense app in a hearing household revealed a concept with a lot of merit, but numerous hurdles still strewn across the path.

For instance, when recording and recognizing sounds, the app requires the background to be more or less silent. A dog barking across the street was enough to muddle a recording of the doorbell, and the sound had to be recorded three times before the app could recognize it successfully. The app does provide a visualization of the current noise levels, which might help users identify a quiet moment in which to record a sound – but the process is still a bit tricky.

The app also appeared to have trouble if ambient noise occurred during or after the trigger sound – for example, if someone was speaking in the room when the doorbell rang, the sound didn’t trigger an alert.

Fortunately, in these cases it’s possible to send feedback to Otosense, which asks “Did I miss something?” The user is then able to select from the library the sound that was missed, and presumably Otosense is able to learn from those instances when the app doesn’t recognize noises as it should. Of course, this functionality may not be as useful for the people the app is designed to help, because it’s hard to pinpoint when the app misses the doorbell if you’re not able to hear it.

The Philips Hue integration, while easy to set up, leaves a bit to be desired as well. There aren’t any options to customize how the alerts show up on your light bulbs — the lights flash once, but aren’t very noticeable. It would be nice to see some additional options, like having the lights flash repeatedly or change color.

Despite its current limitations, Otosense has a lot of potential as a tool for deaf and hard-of-hearing people and as an affordable way to set up alerts in and out of the home. More importantly, we’re beginning to see a new proliferation of apps that are designed to help people with disabilities of all kinds. While there’s no shortage of work to be done, with each update the world becomes a bit more accessible.