Facebook AI Gets Better at Describing Photos for Visually Impaired Users

The social network rolled out an update to its automatic alternative text (AAT) technology.

By Stephanie Mlot

Rafael Henrique | SOPA Images | LightRocket | Getty Images via PCMag

This story originally appeared on PCMag

In an effort to better accommodate users who are blind or visually impaired, Facebook this week updated its automatic alternative text (AAT) technology.

The feature, introduced in 2016 (and granted the Helen Keller Achievement Award from the American Foundation for the Blind in 2018), relies on object recognition to generate descriptions of photos on demand.

Blind and visually impaired (BVI) users have long relied on individuals to tag images with alternative text, or screen readers to mechanically describe pictures on their News Feed. The next generation of Facebook's AAT, however, makes scrolling through social media much more enjoyable.

"The latest iteration … represents multiple technological advances that improve the photo experience for our users," according to a Facebook AI blog post. The team expanded tenfold the number of concepts AAT can reliably detect and identify, promising more photos with more detailed descriptions, including activities, landmarks, types of animals, and more.

If someone navigating their feed, for instance, stops at a photo of friends posing in front of a famous Italian tourist attraction, the audio caption might say something like "May be a selfie of two people, outdoors, the Leaning Tower of Pisa."

Image Credit: Facebook via PCMag

In an apparent industry first, Facebook even makes it possible to include details of positional location and relative size of elements in a picture. So instead of describing the contents as "May be an image of five people," the site can specify that there are two people in the center and three on the sides. Or, rather than describing a landscape with "May be a house and a mountain," it can determine that the summit is the primary object based on its comparable size.

Related: Elon Musk Tells Followers to Use Signal Messaging App Amid WhatsApp Privacy Update

"Taken together, these advancements help users who are blind or visually impaired better understand what's in photos posted by their family and friends—and in their own photos—by providing more (and more detailed) information," the blog said.

When it launched nearly five years ago, the first version of AAT used human-labeled data to train a neural network; the completed model could recognize 100 common concepts like "tree," "mountain," and "outdoors," and identify faces (with opt-in consent). "But we knew there was more than AAT could do," Facebook said, "and the next logical step was to expand the number of recognizable objects and how we describe them."

Now trained on weakly supervised data in the form of billions of public Instagram images and their hashtags, automatic alternative text is more accurate and culturally and demographically inclusive, able to perceive more than 1,200 concepts. "We want to give our users who are blind or visually impaired as much information as possible about a photo's contents—but only correct information," the company added.

Facebook subsidiary Instagram in 2018 took steps to become more accessible, embracing object recognition technology that automatically identifies items in a photo and creates an audible description. Users are also encouraged to write up to 100 characters of alt text detailing what's in their images.

Stephanie Mlot

Reporter at PCMag

Stephanie began as a PCMag reporter in May 2012. She moved to New York City from Frederick, Md., where she worked for four years as a multimedia reporter at the second-largest daily newspaper in Maryland. She interned at Baltimore magazine and graduated from Indiana University of Pennsylvania (in the town of Indiana, in the state of Pennsylvania) with a degree in journalism and mass communications.

Related Topics

Editor's Pick

The Dark Side of Pay Transparency — And What to Do If You Find Out You're Being Underpaid
Thinking of a Career Change? Here Are 4 Steps You Can Take to Get There.
A Founder Who Bootstrapped Her Jewelry Business With Just $1,000 Now Sees 7-Figure Revenue Because She Knew Something About Her Customers Nobody Else Did
Everything You Need to Know About Franchise Law

What Is a Brand Personality? Here's How to Develop One.

Connect with your audience on a deeper level by giving and cultivating your brand a personality. Read here how to do so.

Business Ideas

55 Small Business Ideas To Start Right Now

To start one of these home-based businesses, you don't need a lot of funding -- just energy, passion and the drive to succeed.

Starting a Business

How to Craft the Best Benefits Package for a Global Workforce

Attract top talent from across the globe with a benefits package that speaks to anyone, anywhere.

Starting a Business

How To Sell on Etsy in 2023: A Comprehensive Guide

Want to start selling your handmade goods online? This article outlines how to start and grow your business using Etsy.