Runbook

Give on-call engineers the steps to operate a service and recover it.

EngineeringIT & Operations


Service

The name of the service

Owner team

The team that owns the service

On-call contact

@mention the on-call engineer, or the on-call group

Last tested

Overview

Two or three sentences: what the service does, who uses it, and what happens when it stops.

The main parts of the service, and the services that it depends on. Link to the architecture diagram.

Dashboards and alerts

Dashboards: links to the dashboards that show the health of the service

Alert

What it means

First action

The name of the alert

What is wrong, and how users are affected

What to check or do first

Common tasks

A task that on-call engineers do often, for example restart the service

  1. The first step, with the exact command or the place in the console
  2. The next step
  3. How you check that the task worked

Another task, for example add capacity or clear a queue

  1. The first step, with the exact command or the place in the console
  2. The next step
  3. How you check that the task worked

Recovery

Before you start

Recovery can cause downtime or lose data. Tell the incident channel or the team lead what you will do before you do it.

  1. How to find out which part failed
  2. How to restore the service: the exact commands, in order
  3. How to restore the data from a backup, if you must
  4. How to check that the service works again

Escalation

Level

Who

When

level, for example 1

@mention the person, or name the team

when to escalate, for example when the service is down for 30 minutes

allanswered knowledge management