CSL advances mobile augmented reality technology

Anyone who has seen a spy movie remembers their favorite agent donning special eyewear or using a device to scan surrounding objects and extract information about them. Once an action movie fantasy, augmented reality technology is now almost, well, reality.

“You are interacting with the real world, and putting virtual information on top of that,” said Puneet Jain, a Ph.D. student in computer science who has been working on the technology for about two years. He is conducting research and developing the software alongside advisor Romit Roy Choudhury and Justin Manweiler of IBM Research in New York. The project is being funded by NSF, Nokia and Google.

The technology, called OverLay, works inside a smartphone and essentially combines the camera and the sensor data to make sense of the surroundings in real time. Once OverLay knows that the user is looking at an object, say a pair of shoes on display, relevant information about those shoes can pop up immediately on the screen. The technology will be a breakthrough for marketers, who will be able to advertise products 24/7.

Such forms of mobile augmented reality have been a tantalizing topic for researchers since the dawn of smart phones. Jain said that his passion for the project stems from prior experience in developing technology that identifies the location of objects outdoors. Whereas GPS could be utilized outside, geometric understanding of the environment is too erroneous in indoor environments.

Roy Choudhury, an ECE associate professor, noted that “combining vision and sensing achieves the desired outcome. Specifically, as users look at different objects in their environment, the sensor data is used as geometric constraints.”

For instance, OverLay observes that objects A and B are statistically separated by certain angles, objects B and C are typically separated by 10 seconds in time and so on.

“With these observations” Jain said, “we came up with an optimization framework to build a geometry out of our surroundings, and once we built that layout, we were able to drastically speed-up computer vision.”

The uses for the technology extend beyond defense and surveillance, as seen loosely in fictional spy movies. Jain said that it could be beneficial for infrastructure management, museums, shopping and privacy. For instance, an out-of-reach object, such as a broken air conditioner, could be labeled, with specific information about what part needs to be fixed, using one’s phone from a distance. Through augmented reality, museums could provide more interactivity and education about artifacts and exhibits. If photography is prohibited for certain paintings, the camera could automatically become inactive when it’s pointed at these exhibits.

Jain said that in the future, the technology could be deployed by the likes of Google and Amazon, but there are infrastructural, financial and legal obstacles to overcome before a public release of that scale. For the time being, they are working closely with the Illinois Distributed Museum to iron out issues and make mobile augmented reality accessible for a wider audience.

Jain and his colleagues presented their work at HotMobile, the International Workshop on Mobile Computing Systems and Applications, winning Best Demo. A full paper detailing their work is featured this month in ACM MobiSys 2015, an international conference in Florence, Italy.