Who’s at Fault When AI Makes a Serious Mistake? Managers Can’t Agree.

When AI makes a mistake, the blame game begins.

Researchers at the Wharton School at the University of Pennsylvania recently found that managers are divided over who is responsible when AI systems make “consequential” errors. Their recent study found a growing accountability issue as businesses hand more responsibilities to AI.

When an AI bot makes a serious mistake, managers do not agree on who should answer for it. About one-quarter pointed to the employee using the tool, another quarter named that employee’s manager, and a similar share chose the company’s risk team. Around 10% blamed the AI vendor, while about the same percentage were unsure.

The survey included more than 1,000 business leaders, including managers, directors, executives, owners and founders.

Examples of automation gone wrong

Fears about AI systems acting on their own have drawn attention in recent months. However, for businesses, the more immediate danger may be simpler situations. A vague instruction or a small prompt mistake can lead to bigger problems when an AI agent can access live tools such as employee emails or customer data.

Leeron Walter, a vice president at employee-monitoring software company Teramind, recently told Bloomberg that companies are already encountering those simpler kinds of problems. 

Walter cited a case involving a major U.S. retailer where an employee used AI to automate customer emails. A mistake in the bot’s instructions led it to send false refund and cancellation messages to more than 18,000 customers in under six hours, prompting thousands of service requests. 

Walter also described a U.S. consumer electronics company where a marketing employee used AI to set and post product prices. A prompt mistake listed one of the company’s flagship devices at an 85% discount for three hours. More than 700 customers placed orders before the company fixed the error, and its customer-service team spent weeks dealing with the fallout.

Recent AI advances mean the technology can do more

Accountability is becoming more urgent as major tech companies release AI agents that can do more than answer questions. For example, OpenAI introduced Dots late last month. The always-on agents can connect to over 4,000 apps and keep working on projects between user check-ins. OpenAI is also developing workplace versions of AI with their own credentials and access to company systems. 

Meanwhile, Google announced its Gemini agent for enterprise users earlier this month. The tool can plan and complete multistep work across company systems, including drafting content and writing code. 

Meta introduced Muse last month as a personal agent that can manage email and travel booking tasks and continue working on to-dos even after a user leaves the app. The Muse app has also gained early traction, reaching more than 2.5 million downloads in its first two weeks, according to Sensor Tower data shared with CBS News.

AI is already becoming part of the shopping process. IBM’s recent research shows that 41% of consumers use AI assistants to research products, while 31% turn to them to find deals.