Software Makes Augmented Reality Photo-realistic

Pokémon Go, a game for mobile devices, has given augmented reality a big boost. Computer scientists from Saarbrücken are now taking this technology to a new level of quality by no longer overlaying the real, digital image with visual information, but changing the image realistically.

Picture gallery

"To put it simply, the color value that a camera assigns to individual pixels during shooting is always the product of reflection and lighting," explains Christian Theobalt, head of a research group at the Max Planck Institute for Computer Science and Saarbrücken Computer science professor at Saarland University. "The problem is, however, that the two components of this calculation are only included indirectly in the image and are therefore not measurable," said Theobalt.

Software works in real time

Together with the two Max Planck researchers Abhimitra Meka and Michael Zollhöfer and Christian Richardt, who works at the University of Bath, he solved this problem. The software developed by the researchers does this in real time for every pixel of an image that comes from a conventional webcam.

The demonstrator is simple to set up, so the result is even more impressive: a young woman stands in a room in front of a mobile whiteboard and a large plant. She is wearing a red T-shirt with a wide, white lettering. A webcam films them, but the T-shirt can be seen on the screen of the connected monitor in blue and therefore in a completely different color. It is the only difference that can be seen between the scene in the room and the scene on the screen. The following video shows further examples:

Mathematical optimization procedures create a realistic picture

"By estimating the lighting and the degree of reflection for each pixel at lightning speed and changing only one of the two factors, the realistic impression remains," explains Theobalt. This estimate is based on mathematical optimization methods. By calculating these in parallel, the changes can be implemented so quickly that it is even possible with live recordings. And the software doesn't stop at colors. If the scientists change the reflectance, other materials can even be faked. In this way, the software can transform a filmed shirt made of cotton into a velvet shirt while it is being broadcast.

Possible applications

"No other method currently does this so quickly and with the help of a simple camera," explains Abhimitra Meka, who developed the software as part of his doctoral thesis at the University of Saarland. The Saarbrücken scientists are therefore excited to see which applications the software will be used for. Computer game industry and fashion are obvious. According to Theobalt, online retailers can also benefit from this: “Imagine you are thinking about a new sofa in a different color. With our program you can check whether the intended color matches the lighting conditions in the living room. You don't even have to get up to do it."

Visualization

Visualize complex 3D data efficiently on all devices

Background: Saarland Informatics Campus

The core of the Saarland Informatics Campus is the field of computer science at the Saarland University. Seven other world-renowned research institutes conduct research on the campus in the immediate vicinity. In addition to the two Max Planck Institutes for Computer Science and Software Systems, these are the German Research Center for Artificial Intelligence (DFKI), the Center for Bioinformatics, the Intel Visual Computing Institute, the Center for IT Security, Privacy and Accountability (CISPA) and the Cluster of Excellence "Multimodal Computing and Interaction". (mz)