Facebook AI Gets Better at Describing Photos for Visually Impaired Users

The social network rolled out an update to its automatic alternative text (AAT) technology.
Facebook AI Gets Better at Describing Photos for Visually Impaired Users
Image credit: Rafael Henrique | SOPA Images | LightRocket | Getty Images via PCMag

Free Book Preview Winfluence

Get a glimpse of how to influence your audience’s buying habits using traditional and unconventional influencer marketing techniques.
Reporter at PCMag
3 min read
This story originally appeared on PCMag

In an effort to better accommodate users who are blind or visually impaired, Facebook this week updated its automatic alternative text (AAT) technology.

The feature, introduced in 2016 (and granted the Helen Keller Achievement Award from the American Foundation for the Blind in 2018), relies on object recognition to generate descriptions of photos on demand.

Blind and visually impaired (BVI) users have long relied on individuals to tag images with alternative text, or screen readers to mechanically describe pictures on their News Feed. The next generation of Facebook's AAT, however, makes scrolling through social media much more enjoyable.

"The latest iteration … represents multiple technological advances that improve the photo experience for our users," according to a Facebook AI blog post. The team expanded tenfold the number of concepts AAT can reliably detect and identify, promising more photos with more detailed descriptions, including activities, landmarks, types of animals, and more.

If someone navigating their feed, for instance, stops at a photo of friends posing in front of a famous Italian tourist attraction, the audio caption might say something like "May be a selfie of two people, outdoors, the Leaning Tower of Pisa."

Image Credit: Facebook via PCMag

In an apparent industry first, Facebook even makes it possible to include details of positional location and relative size of elements in a picture. So instead of describing the contents as "May be an image of five people," the site can specify that there are two people in the center and three on the sides. Or, rather than describing a landscape with "May be a house and a mountain," it can determine that the summit is the primary object based on its comparable size.

Related: Elon Musk Tells Followers to Use Signal Messaging App Amid WhatsApp Privacy Update

"Taken together, these advancements help users who are blind or visually impaired better understand what's in photos posted by their family and friends—and in their own photos—by providing more (and more detailed) information," the blog said.

When it launched nearly five years ago, the first version of AAT used human-labeled data to train a neural network; the completed model could recognize 100 common concepts like "tree," "mountain," and "outdoors," and identify faces (with opt-in consent). "But we knew there was more than AAT could do," Facebook said, "and the next logical step was to expand the number of recognizable objects and how we describe them."

Now trained on weakly supervised data in the form of billions of public Instagram images and their hashtags, automatic alternative text is more accurate and culturally and demographically inclusive, able to perceive more than 1,200 concepts. "We want to give our users who are blind or visually impaired as much information as possible about a photo's contents—but only correct information," the company added.

Facebook subsidiary Instagram in 2018 took steps to become more accessible, embracing object recognition technology that automatically identifies items in a photo and creates an audible description. Users are also encouraged to write up to 100 characters of alt text detailing what's in their images.

More from Entrepreneur

Our Franchise Advisors will guide you through the entire franchising process, for FREE!
  1. Book a one-on-one session with a Franchise Advisor
  2. Take a survey about your needs & goals
  3. Find your ideal franchise
  4. Learn about that franchise
  5. Meet the franchisor
  6. Receive the best business resources
Discover the franchise that’s right for you by answering some questions about
  • Which industry you’re interested in
  • Why you want to buy a franchise
  • What your financial needs are
  • Where you’re located
  • And more
Discover a better way to hire freelancers. From business to marketing, sales, finance, design, technology, and more, we have the freelancers you need to tackle your most important work and projects, on-demand.

Latest on Entrepreneur