Back to the blog

Can your cloud team take a holiday?

A guide to reducing key-person risk in IT, with a practical holiday-cover checklist and a safe handover exercise.

Can your cloud team take a holiday?

An infrastructure engineer recently described coming back from ten days off to a difficult conversation. Their manager said the team struggled when they were away. There would be no additional infrastructure hire; the existing team and managed service provider would have to cover it.

The post is on r/sysadmin, and we can't verify what happened inside the company. It raises a practical question: if your infrastructure lead couldn't take a call tomorrow, how would the rest of the team get on?

That's key-person risk in IT: essential work depends on knowledge, access or judgement held by one person. Putting someone else's name on the cover rota won't do much if they haven't had the training or time to practise.

Find the work that still needs a phone call

Start with one service your customers rely on. Sit down with your technical lead and talk through a recent incident, or an example such as a failed release. Follow what happens from the first alert until customers can use the service again.

A deputy might know how to investigate a fault but still have to wait for someone who's away to approve the next step. Or they might get stuck much earlier, trying to find the right dashboard. Check the provider's part in this too: raising a ticket isn't much use as holiday cover if the next step still needs a decision from the person on leave.

Those are different problems. Writing down which dashboard to use can help a deputy get started. It won't help them get a decision from an absent colleague. Work out who will be available to approve action, and agree with the provider who investigates, who acts and when to bring someone else in.

You don't need to learn the recovery commands yourself. Ask what happens to customers while the team waits for help. That gives you a basis for deciding how much interruption the business can cope with.

Make cover specific enough to check

Ask your team to fill in this short record for each essential service. You can start with the one whose failure would cause customers the most immediate trouble.

  • Service and customer impact: what stops working if this service fails?
  • Primary owner and deputy: who normally handles it, and who covers an absence?
  • Approved access: can the deputy reach the systems they need using their own authorised account?
  • Recovery procedure: where are the instructions, and who maintains them?
  • Decision and escalation: what can the deputy decide, and who responds when they reach that limit?
  • Practice evidence: when did the deputy last rehearse the procedure, and what got in the way?

Keep credentials out of this record. It should point to the approved access process, not become another place to store passwords. Access changes still need the normal security review.

Leave an answer blank if nobody knows it yet. That's a gap you can fund or fix. A confident answer needs a bit more scrutiny: ask when the team last tried it and how it went.

Try the handover before the holiday

Set aside a little time with the deputy and technical lead to talk through a fictional incident. The usual expert can listen, but should let the deputy work through the answers. Keep it as a discussion, without changing a live system.

For example, a release has failed and customers cannot complete checkout. Ask the deputy to find the alert, locate the recovery instructions, explain who can approve a rollback and identify who will update customers. Record where they get stuck. If the answer depends on calling the observer, you have found a task for the handover plan.

A tabletop can expose missing information and unclear responsibilities. It doesn't demonstrate that a recovery procedure works. Arrange any hands-on rehearsal separately in a suitable test environment, with an agreed scope and the normal approvals. Note both the date and the result in the cover record.

Give the team time to close the gaps

Training and rehearsal need space in the delivery plan. Sit down with your technical lead and agree what can move to make room. Otherwise, the handover work can sit on an already full to-do list until the next holiday comes round.

The right fix will depend on the gap. It might be training a colleague, simplifying a fragile procedure, hiring, or agreeing more cover with a support provider. If you're relying on an external team, check what they've agreed and prepared to do, including the hours when help is available.

Agree who'll follow up, and put another run-through in the diary. When you meet again, look at where the deputy managed on their own and where they still needed help. You'll have more to go on when deciding whether another hire or more outside support would help.

You can start with one essential service before the next holiday. Ask its deputy to walk you through a recovery, then give the team time to deal with the gaps.

Source

Organizational Failure, r/sysadmin

All articles

THE NEXT ADVANTAGE IS YOURS

Make CloudOps an
advantage for your business.

Your cloud. Our responsibility.

Talk to a Cloud Expert

Talk to a cloud expert