📺 Stream EntrepreneurTV for Free 📺

Microsoft's New AI Can Make Photographs Sing and Talk — and It Already Has the Mona Lisa Lip-Syncing The VASA-1 AI model was not trained on the Mona Lisa but could animate it anyway.

By Sherin Shibu

Key Takeaways

  • Microsoft researchers have come up with a way to turn an image of someone into a video of them lip-syncing to an unrelated audio clip.
  • The framework they came up with is called VASA-1.
  • They wrote that they were "opposed to any behavior to create misleading or harmful contents of real persons."
entrepreneur daily

Microsoft published a research paper this week highlighting a new AI model called VASA-1 that can transform a single picture and audio clip of a person into a realistic video of them lip-syncing — with facial expressions, head movements, and all.

The AI model was trained on AI-generated images from generators like DALL·E-3, which the researchers then layered with audio clips. The results are images-turned-videos of talking faces.

The researchers built on technology from competitors such as Runway and Nvidia, but state in the paper that their method of doing things is higher-quality, more realistic, and "significantly outperforms" existing methods.

Related: Adobe's Firefly Image Generator Was Partially Trained on AI Images From Midjourney

The researchers said the model can take in audio of any length and generate a talking face in accordance with the clip.

The only image that wasn't AI-generated that the researchers experimented with was the Mona Lisa. They made the iconic image lip-sync to Anne Hathaway's "Paparazzi," which starts with the lines "Yo I'm a paparazzi, I don't play no yahtzee."
A screenshot of the video mid-frame. Credit: Entrepreneur

The Mona Lisa was one example of a photo input that the AI model was not trained on — but could manipulate anyway. The model could also transform artistic photos, take in singing audios, and handle speech in languages that weren't English.

The researchers emphasized that the model could work in real-time with a demo video that showed the model instantly animating images with head movements and facial expressions.

Deepfakes, or digitally altered media of a person that could spread misinformation or take someone's likeness without permission, are a risk posed by advanced AI that can generate digital media with relatively few reference points.

Related: Tennessee Passes Law Protecting Musicians From AI Deepfakes

Microsoft addressed that concern generally in the paper, with the researchers stating, "We are opposed to any behavior to create misleading or harmful contents of real persons, and are interested in applying our technique for advancing forgery detection."

The researchers stated that their technique had potentially positive applications too, like improving accessibility and enhancing educational efforts.

Google demoed a similar research project last month, showcasing an AI capable of taking a photo and creating a video from it that the user can then control with their voice. The AI was able to add head movements, blinks, and hand gestures.
Sherin Shibu

Entrepreneur Staff

News Reporter

Sherin Shibu is a business news reporter at Entrepreneur.com. She previously worked for PCMag, Business Insider, The Messenger, and ZDNET as a reporter and copyeditor. Her areas of coverage encompass tech, business, strategy, finance, and even space. She is a Columbia University graduate.

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Side Hustle

These Coworkers-Turned-Friends Started a Side Hustle on Amazon — Now It's a 'Full Hustle' Earning Over $20 Million a Year: 'Jump in With Both Feet'

Achal Patel and Russell Gong met at a large consulting firm and "bonded over a shared vision to create a mission-led company."

Business News

These Are the 10 Most Profitable Cities for Airbnb Hosts, According to a New Report

Here's where Airbnb property owners and hosts are making the most money.

Side Hustle

How to Turn Your Hobby Into a Successful Business

A hobby, interest or charity project can turn into a money-making business if you know the right steps to take.


Want to Be More Productive? Here's How Google Executives Structure Their Schedules

These five tactics from inside Google will help you focus and protect your time.

Starting a Business

This Couple Turned Their Startup Into a $150 Million Food Delivery Company. Here's What They Did Early On to Make It Happen.

Selling only online to your customers has many perks. But the founders of Little Spoon want you to know four things if you want to see accelerated growth.