Smartphone motion sensors could be used to listen to your phone conversations

Track this: A relatively simple device in your smartphone that counts steps, among other things, also has the capacity to be used as a listening device, according to researchers at the University of Illinois Urbana-Champaign.

Your smartphone’s accelerometer – which is responsible for adjusting the screen’s orientation, among other motion-sensor applications – can capture sound vibrations during a phone conversation, which UIUC researchers were able to partially reconstruct into words using machine learning. That means that accelerometers, which are typically not well regulated by privacy controls, as opposed to say, microphones, could be used by rogue apps to listen into conversations without even asking your permission.

“Today, if you download an app that needs access to your accelerometer, it doesn’t have to ask,” said Tarek Abdelzaher, Sohaib and Sara Abbasi Professor of Computer Science and lead researcher. “The accelerometer has always been regarded as an innocent sensor that can’t do any harm. But we’ve learned that maybe it can.”

Accelerometers are found in everything from smartphones to Fitbits and Apple watches. They measure acceleration caused by gravity, motion, or vibration. In the same way, soundwaves can be measured in certain frequencies by the vibration they cause when they land on an object.

Human speech occupies higher frequencies, while motion and vibration are detected in lower frequencies. For example, the sampling rate for an accelerometer is a mere 200 to 400 samples for second, compared to a microphone that takes about 8,000 samples per second, which provides much more information. Recovering higher frequency signals from lower frequencies is like filling-in blanks; one needs to guess the missing samples, explained Tianshi Wang, one of the researchers.

Abdelzaher’s team overcame these challenges by developing a new neural network that can reconstruct high-bandwidth signals from very low-sampling-rate data.

“Human speech has a special pattern,” Abdelzaher said. “By constraining your interpretation to that special pattern, you can guess higher frequency from low frequency. This won’t work with any random sound, but for some keywords or numbers, it works fairly well.”

The work stemmed in part out of research sponsored by the National Science Foundation. The goal was to study whether it was possible to reduce the cost of sensing and sensor data processing in intelligent Internet of Things (IoT) applications, particularly in scenarios when bandwidth was less than optimal. The team was able to reduce the volume of communications data, which may be useful as the number of IoT devices continues to expand.

The research was presented last week at the UbiComp 2021 conference. In the future, the team may work to develop the research into a super-resolution technique that enhances the resolution of signals (acceleration, sound, images, video, etc…) automatically to recover higher fidelity information from lower-fidelity signals. 

“We increasingly need a communications strategy where devices can perform intelligent functions even in resource-scarce environments,” Abdelzaher said. “But since it turns out that accelerometers can do unexpected things, like recording your credit card information, we want to encourage that people be aware of these vulnerabilities.”