Smile (or Not): Photos Can Be Animated to Show Expressions

With the help of an actor and some high-tech motion-capture techniques, computer scientists can now take a still photo of a person's face and animate it. The photos can be animated to express emotions such as happiness, anger or surprise. They can even include details such as teeth when the person in the photo had shown none.

The new photo-manipulation technique is the result of a collaboration between computer scientists at Facebook and Tel Aviv University. The research, published in the journal Associate for Computing Machinery on Nov. 20, provides what the scientists claim is the most realistic manipulations of a portrait or selfie to date. [Optical Illusions: A Gallery of Visual Tricks]

"The most difficult part is to make it look real, or natural looking," said lead author Hadar Averbuch-Elor, a doctoral candidate at Tel Aviv University. "People are extremely sensitive to the most subtle variations in face animation and it's challenging not to fall into the 'uncanny valley,'" she said.

The team starts out by mapping the facial features of someone looking at the camera in a photo. Then, they did the same facial-feature mapping to an actor expressing an emotion in a video, either shot in the lab or taken from a database. The facial movements from the video were then applied to the original photo, animating it into expressing an emotion, according to the research.

Once the researchers got the original photo moving, they fine-tuned the resulting video by smoothing out wrinkles and, if necessary, adding in the actor's teeth and tongue.

What they were left with was a short video of a person making an expression. Even if the person in the photo had never made that face in their entire life, the resulting product made it look as if they had.

To see if the videos were convincing, the researchers showed them to 30 people. They found that 58 percent of the participants thought that a video of someone smiling was real the first time they saw it, and 37 percent thought the same of a video of someone making a surprised expression. Overall, an average of 46 percent of the people said they thought the manipulated videos were real, compared with an average 87 percent of the people who thought the unedited videos were real, according to the paper. Thirteen percent thought that the real videos were fakes.

The animated faces aren't perfect, however. Many of the problems come from either the photo or the actor in the video looking off to the side, because it creates a bizarre contortion in which a part of the face pivots but the rest of the photo continues to look forward, the researchers said. Also, a photo of someone smiling with exposed teeth adds difficulty — if the actor creates an open-mouthed expression, the photo's teeth will stretch out instead of separating.

"Combining this tech with 3D would solve pose issues," Averbuch-Elor told Live Science, though projecting the photo into 3D might lower the image quality. "It would also be cool to combine it with VR to create an interactive avatar from just a single image," she said.

The team said that this technology could be used to animate profile photos on Facebook; clicking the "like" button could someday make your photo smile at the liker, or something to that effect.

It's possible that the technology could one day be used to manipulate photos into a deliberately misleading videos; however, that wasn't a current question for the researchers. But with technologies like this and others, including AI-generated photorealistic landscapes, it could be easier than ever to fake photos.

"When we were creating the technology, the goal was to push the boundaries of what's possible starting from just a single image," Averbuch-Elor said. "We didn't have production plans in mind and still don't — we wanted to create state-of-the-art research."

As face-manipulating technology becomes even more advanced, the line between real and fake could blur and become harder to find.

"We didn't really think about it while we were creating this technology," says Averbuch-Elor, "but like with many other examples, technology can be misused, and it's a scary thing."

Original article on Live Science .