You were supposed to be closing your next customer. Instead, you became the incident manager for the afternoon.
The application was down for 40 minutes. Customers needed an explanation, engineers had failed jobs to work through, and you had to replan the rest of the day. Across a small team, that can easily add up to a day of lost work, sometimes more.
You cannot guarantee there will never be another outage. You can prepare for how your team reacts when one happens.
Keep the discussion close to the work your team does. Think about the customer call you had to postpone, the engineer pulled away from a release and the time spent checking whether a report had arrived. Include those hours when you review the outage. They help you decide which part of recovery needs attention first.
Start with three questions at your next team meeting.
1. How will we know which customers are affected?
Take something a customer does in your application, such as uploading a file or generating a report. If it failed this afternoon, could someone on your team find the request and tell that customer what happened?
Some requests fail straight away. Others are accepted but sit in a queue, waiting to be processed. New requests might work once the application comes back while those earlier customers are still waiting. Your team needs to be able to tell the difference.
For the workflow you chose, follow one request from the customer action through to the result they expect. Note how you would identify it, where you would check its progress and what would count as completion. A customer who can sign in but still cannot get their report needs a different update from someone whose work has finished.
Write down where to look and who has access. If finding a failed job depends on one engineer remembering how to search several systems, work through it together before an outage makes it urgent.
2. Who takes charge, and who keeps customers updated?
Agree who will coordinate the response and who will speak to customers. In a small team, one person may do both, but they need to know it is their job. Have a backup for when they are unavailable.
Keep the current status and decisions in one place so the person answering customers can see what engineering knows without interrupting them for every question. Decide who approves updates and how you will agree when to send the next one.
You can give customers an update before you know the cause. Tell them what is affected, what you are checking and when they will hear from you again. An early suspicion should stay a suspicion until you have evidence. The same goes for a promised recovery time.
Decide what customers need to do while they wait. If you have confirmed that their request is still queued, tell them whether to leave it there or try again. If you have not checked, say that you are checking. Keep that advice with the incident status so another person picking up the conversation can give the same answer.
Give the next update when you said you would, even if the investigation is still under way. Explain what you have checked since the last message and what remains unresolved. That gives customers something useful to pass on to their own colleagues.
3. What needs to be done when the application comes back?
Engineers may still need to retry failed jobs or check that delayed notifications were delivered. Put that work in the incident plan and give each task an owner.
Before retrying an action, check whether it already completed. Running a report again is different from charging a customer or sending an order twice.
Agree what you need to check before calling the incident over. Can a customer complete the affected workflow? Has the team accounted for failed or delayed work, and is the backlog returning to normal? Have affected customers heard from you?
Separate the work that must finish before you close the incident from improvements you will make afterwards. A delayed customer report still needs someone to see it through. A better way to find failed reports belongs in the follow-up work, with an owner and a date to check progress. Keep both visible until they are done.
Once that work is done, record the timeline while everyone remembers it. Choose a few improvements, assign them to someone and follow through on them.
Use the timeline to find where time went. If the team spent most of the afternoon finding affected requests, start there. If engineers kept stopping to answer status questions, review how updates were shared. Work through the revised plan together using the same customer workflow, and check that someone other than its author can follow it.
You invest in software development to move your business forward. Give your cloud infrastructure the same attention.
At your next team discussion, look at the last outage. Compare when the application came back with when your team and customers were back to normal. The work between those two times belongs in your recovery plan.
