Ending Soon! Save 33% on All Access

Preparing for the Inevitable: 5 Steps to Follow When Technology Fails Give your engineers the help they need to make the tough calls at crunchtime.

By Patrick Hill Edited by Dan Bova

Opinions expressed by Entrepreneur contributors are their own.

gilaxia | Getty Images

Amazon's massive internet outage in late March was a reminder that any company offering a public cloud service, however big or small, needs a plan for incident response. Outages are a fact of life; what matters is how you respond when they occur.

Having processes in place is essential, but those processes can't (and shouldn't try to) cover all eventualities. If something unexpected strikes at 3 a.m., your incident response team needs firm guidelines to help them decide how to act in the critical moments that follow.

Related: Why Every Small Business Needs a Backup and Disaster Recovery Plan

At Atlassian, we came up with five values that guide how we respond to incidents and minimize disruption. A lot gets written about "values," but they're more than something nice to hang on the wall. Our engineers look to these values to steer them through tough decisions they have to make under pressure.

Each value maps to a specific component of incident response. I'm sharing them here in the hope they'll be useful to your organization, too.


Value: Atlassian knows before our customers do

A well-designed service will have enough monitoring to detect and flag any issue before it becomes an incident. If your team isn't getting paged about imminent problems before they impact customers, you need to improve your monitoring and alerting.


Value: Escalate, escalate, escalate

The worst thing an engineer can decide is that they don't want to wake someone because it might not be their problem. Nobody should mind getting woken for an incident and finding out they're not needed. But they will mind if they're not woken when they should have been. We're supposed to be on the same team, and teammates support each other.

Related: The Worst Hacks of 2017 -- So Far


Value: Stuff happens; clean it up fast

Customers don't care why your service is down, only that you restore it as fast as possible. Never hesitate to get an incident resolved quickly so you can minimize the impact.

If you're the tech lead and you know you can restore service with a quick restart, but you could also spend time investigating the cause while the service is still down, what should you do? This value guides your answer: Restore now and figure out the cause later; the customer experience comes first.


Value: Always blameless

Incidents are a part of running a service. We all improve by holding teams accountable, not apportioning blame. Human error is never a valid root cause for a major incident. Why was that engineer able to deploy a dev version to production? How did a command-line typo have such a devastating effect?

Assigning blame is never the appropriate response. Figure out what safeguards were missing and put them in place.

Related: 3 Ways to Protect Yourself From a Ransomware Disaster


Value: Never have the same incident twice

Determine the root cause and identify the changes that will prevent that whole class of incidents from happening again. Can the same bug bite elsewhere? What situations could lead to a programmer introducing this bug? Commit to delivering specific changes by specific dates.

With these values in place, the next step is to ensure they're put into practice. We hold monthly meetings where we discuss how they've been implemented and dissect occasions when they weren't. We call people out for following them -- and for not following them. And we've added them to our documentation for incident response.

Service outages are a big deal: the AWS incident affected 54 of the top 100 retailers, and that's in just one industry segment. Your footprint may be a good deal smaller, but the impact of an outage on both you and your customers can be just as disruptive, proportionally speaking. Give your engineers the help they need to make the tough calls at crunchtime. Both they and your customers will thank you for it.

Patrick Hill

SRE Team Lead at Atlassian

Patrick Hill is SRE team lead at Atlassian, provider of team collaboration and productivity software that helps teams organize, discuss and complete shared work. Teams at more than 68,000 organizations use Atlassian products including JIRA, Confluence, HipChat, Trello and Bitbucket. Based in Austin, Texas, Hill helps build teams and processes to ensure consistently high performance and availability of Atlassian's internal and external cloud services.

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Social Media

How To Start a Youtube Channel: Step-by-Step Guide

YouTube can be a valuable way to grow your audience. If you're ready to create content, read more about starting a business YouTube Channel.

Growing a Business

Want to Expand Your Market Overseas? Here's Everything You Need to Know About Global Logistics in 2024

With rising geopolitical tensions and changing market conditions it can be hard for businesses to navigate supply chain logistics even in a post-pandemic world. Here are three tips from the CEO of an international customs brokerage.

Business Ideas

63 Small Business Ideas to Start in 2024

We put together a list of the best, most profitable small business ideas for entrepreneurs to pursue in 2024.

Business News

Elvis Presley's Granddaughter Fights Graceland Foreclosure, Calls Paperwork 'Forgeries'

The 13.8-acre estate was scheduled to be sold in a public foreclosure auction on Thursday. Presley's granddaughter and heir, Riley Keough, is fighting to save Graceland in court.

Business News

Kickstarter Is Opening Up Its Platform to Creators and Making Big Changes to Its Model — Here's What's New

The company noted it is moving beyond traditional crowdfunding and making it easier for businesses to raise more money.