The University of Washington algorithm Photo Wake-Up was posted in preprint form on arXiv in December and created a buzz because it can take a person from a 2D photo or a work of art and make them run, walk or jump out of the frame. The system also allows users to view the animation in three dimensions using augmented reality tools. Next week at the Conference on Computer Vision and Pattern Recognition in Long Beach, California, the researchers will be showing results.
Creating a 3-D character using multiple viewpoints is possible but there are challenges in animating a person running out of a photograph, chiefly that the input is only from a single camera position, so part of the person is invisible. With a single photograph it is far more challenging to being someone to life and have them run out of a scene, nor could you use augmented reality to join the photo.
Stephen Curry runs off the court. Credit: University of Washington
That means artistic creative visualization must also come into play. And it's what the computer scientists have done.
To make the magic a reality, Photo Wake-Up starts by identifying a person in an image and making a mask of the body's outline. From there, it matches a 3D template to the subject's body position. Then the algorithm does something surprising: In order to warp the template so that it actually looks like the person in the photo, it projects the 3D person back into 2D.
"It's very hard to manipulate in 3D precisely," said co-author Chung-Yi Weng, a doctoral student in the Allen School. "Maybe you can do it roughly, but any error will be obvious when you animate the character. So we have to find a way to handle things perfectly, and it's easier to do this in 2D."
The applications of Photo Wake-Up are numerous, the team says. The researchers envision this could lead to a new way for gamers to create avatars that actually look like them, a method for visitors to interact with paintings in an art museum -- say sitting down to have tea with Mona Lisa -- or something that lets children to bring their drawings to life. Examples in the research paper include animating the Golden State Warriors' Stephen Curry to run off the court (above), Paul McCartney to leap off the cover of the "Help!" album and Matisse's "Icarus" to leave his frame.
It can even be used in VR, as seen here.
Photo Wake-up also allows users to view the animation in three dimensions using augmented reality tools.University of Washington