Augmented reality is an emerging technology that uses handheld devices to superimpose digital data on the real world: If, say, you’re in Paris and point your phone at the Eiffel Tower, the tower’s image would pop up on-screen, along with, perhaps, information about its history or the hours that it’s open. With a TV enhancement called Surround Vision, researchers at MIT Media Lab are bringing the same technology into the living room.

Surround sound technology uses multiple speakers to extend the world of a TV show or movie beyond the edges of the screen: the audience can, in effect, hear what’s happening just off-camera. Surround Vision does something analogous, but with images.

Surround Vision is intended to work with standard, Internet-connected handheld devices. If a viewer wanted to see what was happening off the left edge of the television screen, she could simply point her cell phone in that direction, and an image would pop up on its screen. The technology could also allow a guest at a Super Bowl party, for instance, to consult several different camera angles on a particular play, without affecting what the other guests see on the TV screen.

To get his prototype up and running, MIT graduate student Santiago Alfaro had to attach a magnetometer — a compass — to an existing handheld device and write software that incorporated its data with that from the device’s other sensors. But Alfaro says that devices now on the market, including the most recent version of the iPhone, have magnetometers built in.

Once he rigged up a handheld device with the requisite motion sensors, Alfaro shot video footage of the street in front of the Media Lab from three angles simultaneously. A television set replays the footage from the center camera. If a viewer points a motion-sensitive handheld device directly at the TV, the same footage appears on the device’s screen. But if the viewer swings the device either right or left, it switches to one of the other perspectives. The viewer can, for instance, watch a bus approach on the small screen before it appears on the large screen.

Since many DVD and Blu-ray movies now come with bonus footage of scenes shot from different angles, Alfaro was able to devise demos that allowed the user to switch between the final version of a film and alternate takes.

The researchers plan a series of user studies in the spring and summer, which will employ content developed in conjunction with a number of partners. Since sports broadcasts and other live television shows already feature footage taken from multiple camera angles, they’re a natural fit for the system. Many children’s shows already encourage a kind of audience participation that could be enhanced, and viewers of the type of criminal-forensics shows now popular could use Surround Vision to, say, see what the show’s protagonists are looking at through the microscope lens.

<

Story filed 13.04.10