How to Set Up an Incident Management Process in 6 Steps

Incident management is the practice of declaring a service disruption, coordinating the people who respond, restoring normal service, and learning from the event afterward. This guide sets up an incident management process in 6 steps: define severity levels, write the rule for declaring an incident, assign incident roles, keep one live incident document, restore service while preserving evidence, and review the incident.

The steps fit a software team that runs an online service with an on-call rotation. The steps do not cover security breach handling or regulatory notification.

What incident management means

Google’s Site Reliability Engineering book includes a chapter on managing incidents by Andrew Stribblehill. The chapter follows an incident that spirals out of control under ad hoc practices and names the hazards behind it: a sharp focus on the technical problem, poor communication, and freelancing. The chapter bases Google’s incident management system on the Incident Command System, which the chapter credits with clarity and scalability.

The short form incident mgmt names the same practice. In the SRE chapter, the incident management framework separates the response into roles, so everyone involved knows who leads, who changes the system, and who keeps stakeholders informed.

Term	Meaning in this guide	Source
Incident	An event that meets the team’s declaration rule, such as an outage customers can see	SRE book, Managing Incidents
Severity level	A numbered level from SEV-1 to SEV-5, where a lower number is more urgent	PagerDuty Incident Response documentation
Major incident	An incident above SEV-3, which gets a more intensive response	PagerDuty Incident Response documentation
Incident commander	The person who holds the high-level state of the incident	SRE book, Managing Incidents

ITIL and ITSM incident management

ITIL 4 treats incident management as one of its practices, and PeopleCert lists ITIL under IT governance and service management. PeopleCert’s page for the ITIL 4 Practitioner: Incident Management certification describes the practice’s role as restoring normal service operation quickly after a disruption, and describes its processes as a way to minimize service downtime.

An ITSM incident management process and the SRE framework share restoring service as a core goal. The SRE chapter adds the command roles of the Incident Command System, and the 6 steps below combine the two: measurable severity levels and a declaration rule first, then command roles and a live record.

Before you set up incident management

Six inputs decide whether the process works on the first real incident.

The services in scope, and the symptoms customers see when each service fails.
An on-call rotation with a reachable person for every service in scope.
One agreed place for incident communication, such as a dedicated chat channel or a bridge call.
A metric per service that shows user impact, such as the error rate or the share of affected accounts.
A timestamped record of recent changes: deploys, configuration edits, and feature flag edits.
Agreement that the incident commander can call in people from other teams.

How to set up an incident management process in 6 steps

1. Define severity levels with measurable criteria

Severity levels tell responders how hard to push and whom to call in. PagerDuty’s public severity levels page, part of its Incident Response documentation, uses five levels from SEV-1 to SEV-5, where a lower number is more urgent. On that page, a SEV-1 is critical enough to need public notification and contact with executive teams, a SEV-2 is a critical system issue that affects many customers’ use of the product, and a SEV-3 is a stability issue or minor customer impact that service owners handle right away.

The same page says any incident above SEV-3 is a major incident with a more intensive response, recommends severity definitions that are metric driven, and tells responders who cannot choose between two levels to treat the incident as the higher one. The criteria in the table below are illustrative for a web service.

Level	Example criterion	First response
SEV-1	Checkout fails for every customer	Incident commander, status page update, executives informed
SEV-2	Error rate above 5% for many customers	Incident commander and the on-call team
SEV-3	One feature degraded for a minority of customers	Service owner responds right away

2. Write the rule for declaring an incident

A declaration rule removes the debate over whether a problem is big enough. The SRE chapter lists three questions, and a yes to any one of them makes the event an incident on the author’s team: does the fix need a second team, can customers see the outage, and is the issue still unsolved after an hour of concentrated analysis. The chapter also recommends declaring early, and says declaring an incident and closing it after a simple fix is better than starting the incident management framework hours into a growing problem.

Write the rule where every on-call engineer can find it, and let any on-call engineer declare.

3. Assign incident roles, starting with the incident commander

Roles keep responders from working on the same problem in different directions, which the SRE chapter calls freelancing. The chapter separates responsibilities into four roles, so that everyone involved knows their role and stays out of the others’ work.

Role	Job in the SRE chapter	Changes production
Incident command	Holds the high-level state of the incident and keeps the living incident document	No
Operational work	Works with the incident commander and applies operational tools to the incident	Yes, and the operations team is the one group that modifies the system during the incident
Communication	Issues periodic updates to the response team and stakeholders	No
Planning	Files bugs, arranges handoffs, and tracks how the system diverged from normal so the divergence can be reverted	No

Name the person in each role at the top of the live incident document, so anyone joining the response can see who leads.

4. Keep one live incident document and hand off command explicitly

The SRE chapter calls keeping a living incident document the incident commander’s most important responsibility, says the document can be messy but must be functional, and says to keep the document for postmortem analysis. When command changes hands, the outgoing commander states the handoff explicitly, stays until the new commander firmly acknowledges it, and tells everyone working on the incident who leads now.

A live incident document that serves both the response and the review holds six things.

The current severity, the status, and the name of the incident commander.
A timeline of what was observed, what was changed, and by whom.
The customer-facing impact and the time of the next update.
Open questions and the person working on each one.
Suspect changes with their times: deploys, configuration edits, and flag edits.
Follow-up items for the review.

5. Stop the impact, restore service, and preserve the evidence

The SRE chapter puts the order in one line of its practices: stop the bleeding, restore service, and preserve the evidence for root-causing. PagerDuty’s severity page lists rolling back as a typical SEV-3 response when the issue is related to a recent deployment. The ops lead starts from the suspect changes in the live document, and the incident commander decides whether to reverse a suspect change before its cause is understood.

Feature flag edits belong on the list of suspect changes, because a flag edit changes what customers receive without a deploy. In Loglune, each flag edit made in the console or through the API is appended to the change history as a change event that records the customer, the actor, the time, the target flag, and the values before and after the edit. Returning a rollout to its earlier percentage in Loglune is a new change event of its own, so the reversal sits in the history next to the edit it reverses.

Preserving evidence for a flag-related incident means keeping the time of the first report and the evaluation record. The Loglune SDK returns each flag value together with an evaluation record containing the version, the attribute key, and the returned value, and an error report or support ticket can carry that record. After service is restored, pick an affected customer and the time T of the report, or import the evaluation record, and Loglune rebuilds the flag state at T from the change history and delivers it in reproduction mode to a development, CI, or staging environment. One delivery counts as one reproduction, so a second rebuild at a time just before the suspect edit, which shows the customer’s flag state before the change, counts as a second reproduction. Loglune reproduces the flag evaluation, not the root cause in application code, and the guide to reproducing a customer-specific bug covers the debugging that follows.

6. Review the incident and practice the process

The review turns the live document into follow-up work: the timeline, the change that started the impact, the change that ended it, what slowed detection or response, and follow-up items with owners. The SRE chapter lists practices that apply after the incident as well: develop and document the procedures in advance with the people who take part in incidents, use the process routinely so it becomes second nature, and rotate roles so the last incident commander takes a different role next time.

The chapter closes with the team’s own result: formulating an incident management strategy in advance, structuring it to scale smoothly, and using it regularly reduced the team’s mean time to recovery. The chapter reports that result for the author’s team at Google without a percentage.

Incident management system: what the tooling needs to record

An incident management system is the tooling that carries the process: paging, the incident record, the live document, status updates, and the timeline. The criteria in the table come from the 6 steps, not from a product ranking.

Step	What the system records	Question to ask when comparing tools
Severity levels	A severity field with defined criteria	Can the level change during the incident, with a timestamp?
Declaration	Who declared the incident and when	Can any on-call engineer declare?
Roles	The incident commander, the ops lead, and the communication lead	Is the current incident commander visible to everyone?
Live document	One document per incident	Does the document keep its edit history for the review?
Restore and evidence	Changes made before and during the incident	Can deploys, configuration edits, and flag edits be lined up by time?
Review	Follow-up items with owners	Are follow-up items tracked until they close?

Loglune is a feature flag platform, not an incident management system. Loglune covers the flag part of the restore-and-evidence row: the change history of flag edits, and the flag state an affected customer had at time T.

The table follows one incident from the first alert to the review. Times, percentages, and names are illustrative.

Time	Event	Who	Record
2:05 p.m.	Invoice export errors rise to 8% of requests	On-call engineer	Alert
2:12 p.m.	Customers report failed exports, and the incident is declared as SEV-2	On-call engineer, now incident commander	Live incident document opened
2:15 p.m.	Ops lead and communication lead assigned	Incident commander	Roles named in the document
2:20 p.m.	Suspect changes listed: no deploy since morning, invoice-export rollout raised from 10% to 50% at 1:58 p.m.	Ops lead	Suspect change in the document
2:24 p.m.	Rollout returned to 10%	Ops lead	New change event in Loglune
2:40 p.m.	Error rate back to baseline, incident resolved	Incident commander	Document kept for the review
Next day	Flag state of one affected customer at 2:05 p.m. rebuilt in staging	Engineer on the follow-up	One reproduction

In the example, service recovered 28 minutes after the declaration, and the reproduction the next day gave the follow-up engineer the flag state behind one failed export without touching production.

Troubleshooting incident management

Nobody declares an incident until the problem is large: apply the three declaration questions from step 2, and let any on-call engineer declare.
Several engineers change production at once: route every change through the ops lead, because the SRE chapter keeps changes during an incident within the operations team.
Stakeholders interrupt responders for updates: assign the communication role and publish the time of the next update.
The incident commander goes off shift mid-incident: hand off explicitly and wait for acknowledgment, as the SRE chapter describes.
Severity debates slow the response: treat the incident as the higher level when unsure, as PagerDuty’s page recommends, and adjust the level later.
A bug appeared right after a flag edit and nobody knows the flag state at the time: rebuild the affected customer’s flag state at the reported time instead of guessing from the current rule.

FAQ

What is the incident management process?

The incident management process is the sequence a team follows from detection to review: classify severity, declare the incident, assign roles, keep a live record, restore service, and review. The 6 steps in this guide are one version of that process, built from the SRE chapter on managing incidents and PagerDuty’s severity levels.

What is incident control?

The SRE chapter does not use the phrase incident control. The chapter keeps an incident under control with one incident commander at a time, explicit handoffs, and a live incident document, and describes the absence of those as the way an incident spirals out of control.

When does an incident become a major incident?

On PagerDuty’s severity page, any incident above SEV-3 is a major incident and gets a more intensive response. The page adds that every SEV-2 is a major incident, while a major incident does not have to be a SEV-2.

How does ITIL incident management differ from the SRE approach?

Both treat restoring service as a core goal. PeopleCert’s description of the ITIL 4 practice centers on restoring normal service operation and reducing downtime, while the SRE chapter adds named roles from the Incident Command System and a live incident document kept by the incident commander.

Deciding how much incident process your team needs

Start with the declaration rule, one incident commander, and the live document, because those three answer the hazards the SRE chapter names: freelancing, poor communication, and a narrow focus on the technical problem. Add severity levels once the team has a user-impact metric per service, and add the planning role when incidents last long enough to need handoffs. Record every flag edit next to deploys and configuration edits, so the ops lead can reverse a suspect change and the follow-up engineer can rebuild an affected customer’s flag state at the time of the report.