Incident Management: Set Up a Process in 6 Steps
- Incident commander
- Ops lead
- Loglune
Incident management is the practice of declaring a service disruption, coordinating the people who respond, restoring normal service, and learning from the event afterward. This guide sets up an incident management process in 6 steps: define severity levels, write the rule for declaring an incident, assign incident roles, keep one live incident document, restore service while preserving evidence, and review the incident.
The steps fit a software team that runs an online service with an on-call rotation and handles outages ad hoc today, with no agreed rule for who leads or who changes production. After reading, the team can write its severity table, its declaration rule, and its role list, and run the next incident from one live document. The steps do not cover security breach handling or regulatory notification.
What incident management means
Google’s Site Reliability Engineering book includes a chapter on managing incidents by Andrew Stribblehill. The SRE chapter follows an incident that spirals out of control under ad hoc practices and names the hazards behind it: a sharp focus on the technical problem, poor communication, and freelancing. The SRE chapter bases Google’s incident management system on the Incident Command System, which the SRE chapter credits with clarity and scalability.
The short form incident mgmt names the same practice. In the SRE chapter, the incident management framework separates the response into roles, so everyone involved knows who leads, who changes the system, and who keeps stakeholders informed.
| Term | Meaning in this guide | Source |
|---|---|---|
| Incident | An event that meets the team’s declaration rule, such as an outage customers can see | SRE book, Managing Incidents |
| Severity level | A numbered level from SEV-1 to SEV-5, where a lower number is more urgent | PagerDuty Incident Response documentation |
| Major incident | An incident above SEV-3, which gets a more intensive response | PagerDuty Incident Response documentation |
| Incident commander | The person who holds the high-level state of the incident | SRE book, Managing Incidents |
ITIL and ITSM incident management
ITIL 4 treats incident management as one of its practices, and ITIL belongs to IT governance and service management. The practice’s role is to restore normal service operation quickly after a disruption, and its processes are a way to minimize service downtime.
An ITSM incident management process and the SRE framework share restoring service as a core goal. The SRE chapter adds the named roles of the Incident Command System and a live incident document kept by the incident commander, and the 6 steps below combine the two: measurable severity levels and a declaration rule first, then command roles and a live record.
Before you set up incident management
Six inputs decide whether the process works on the first real incident.
- The services in scope, and the symptoms customers see when each service fails.
- An on-call rotation with a reachable person for every service in scope.
- One agreed place for incident communication, such as a dedicated chat channel or a bridge call.
- A metric per service that shows user impact, such as the error rate or the share of affected accounts.
- A timestamped record of recent changes: deploys, configuration edits, and feature flag edits.
- Agreement that the incident commander can call in people from other teams.
How to set up an incident management process in 6 steps
1. Define severity levels with measurable criteria
Severity levels tell responders how hard to push and whom to call in. PagerDuty’s public severity levels page, part of its Incident Response documentation, uses five levels from SEV-1 to SEV-5, where a lower number is more urgent. On that page, a SEV-1 is critical enough to need public notification and contact with executive teams, a SEV-2 is a critical system issue that affects many customers’ use of the product, and a SEV-3 is a stability issue or minor customer impact that service owners handle right away.
PagerDuty’s severity levels page says any incident above SEV-3 is a major incident with a more intensive response. The same page adds that every SEV-2 is a major incident, while a major incident does not have to be a SEV-2. The page also recommends severity definitions that are metric driven, and tells responders who cannot choose between two levels to treat the incident as the higher one. The criteria in the table below are illustrative for a web service.
| Level | Example criterion | First response |
|---|---|---|
| SEV-1 | Checkout fails for every customer | Incident commander, status page update, executives informed |
| SEV-2 | Error rate above 5% for many customers | Incident commander and the on-call team |
| SEV-3 | One feature degraded for a minority of customers | Service owner responds right away |
2. Write the rule for declaring an incident
A declaration rule removes the debate over whether a problem is big enough. The SRE chapter lists three questions, and a yes to any one of them makes the event an incident on the author’s team: does the fix need a second team, can customers see the outage, and is the issue still unsolved after an hour of concentrated analysis. The SRE chapter also recommends declaring early, and says declaring an incident and closing it after a simple fix is better than starting the incident management framework hours into a growing problem.
Write the rule where every on-call engineer can find it, and let any on-call engineer declare.
3. Assign incident roles, starting with the incident commander
Roles keep responders from working on the same problem in different directions, which the SRE chapter calls freelancing. The SRE chapter separates responsibilities into four roles, so that everyone involved knows their role and stays out of the others’ work.
| Role | Job in the SRE chapter | Changes production |
|---|---|---|
| Incident command | Holds the high-level state of the incident and keeps the living incident document | No |
| Operational work | Works with the incident commander and applies operational tools to the incident | Yes, and the operations team is the one group that modifies the system during the incident |
| Communication | Issues periodic updates to the response team and stakeholders | No |
| Planning | Files bugs, arranges handoffs, and tracks how the system diverged from normal so the divergence can be reverted | No |
Name the person in each role at the top of the live incident document, so anyone joining the response can see who leads.
4. Keep one live incident document and hand off command explicitly
The SRE chapter calls keeping a living incident document the incident commander’s most important responsibility, says the document can be messy but must be functional, and says to keep the document for postmortem analysis. When command changes hands, the outgoing commander states the handoff explicitly, stays until the new commander firmly acknowledges it, and tells everyone working on the incident who leads now.
A live incident document that serves both the response and the review holds six things.
- The current severity, the status, and the name of the incident commander.
- A timeline of what was observed, what was changed, and by whom.
- The customer-facing impact and the time of the next update.
- Open questions and the person working on each one.
- Suspect changes with their times: deploys, configuration edits, and flag edits.
- Follow-up items for the review.
5. Stop the impact, restore service, and preserve the evidence
The SRE chapter puts the order in one line of its practices: stop the bleeding, restore service, and preserve the evidence for root-causing. PagerDuty’s severity page lists rolling back as a typical SEV-3 response when the issue is related to a recent deployment. The ops lead starts from the suspect changes in the live document, and the incident commander decides whether to reverse a suspect change before its cause is understood.
Feature flag edits belong on the list of suspect changes, because a flag edit changes what customers receive without a deploy. In Loglune, each flag edit made in the console or through the API is appended to the change history as a change event that records the customer, the actor, the time, the target flag, and the values before and after the edit. Returning a rollout to its earlier percentage in Loglune is a new change event of its own, so the reversal sits in the history next to the edit it reverses.
Preserving evidence for a flag-related incident means keeping the time of the first report and the evaluation record. The Loglune SDK returns each flag value together with an evaluation record containing the version, the attribute key, and the returned value, and an error report or support ticket can carry that record. After service is restored, pick an affected customer and the time T of the report, or import the evaluation record, and Loglune rebuilds the flag state at T from the change history and delivers it in reproduction mode to a development, CI, or staging environment. One delivery counts as one reproduction, so a second rebuild at a time just before the suspect edit, which shows the customer’s flag state before the change, counts as a second reproduction. Loglune reproduces the flag evaluation, not the root cause in application code, and the guide to reproducing a customer-specific bug covers the debugging that follows.
Loglune’s example dataset shows the suspect-change list and the preserved evidence for one flag-related change. The dataset is synthetic input labeled Example data in the Loglune console at loglune.com/app, visible without signing in, and it describes no customer and no real incident. The example dataset holds 7 change events, each with an actor, a time, and the values before and after. The console’s diff of version 2 and version 3 lists two changes, both made by jordan@example.com on August 20, 2026, at 11:15 UTC: the default variation of the flag checkout-v2 from off to on, and the config value checkout-timeout-ms from 3000 to 5000. Those two lines are the suspect changes an ops lead copies into the live incident document, with the actor and the time already attached. Rebuilding the flag state at a time T uses the latest version published at or before T, so T = August 10, 2026, returns version 2, and a T before August 1, 2026, at 09:00 UTC returns 404 because no version existed yet. The example bug-report fingerprint ex-fp-2, recorded on August 23, 2026, at 07:48 UTC, carries version 3 and checkout-v2 = on with the reason default, which is the evaluation record a support ticket keeps as evidence.
6. Review the incident and practice the process
The review turns the live document into follow-up work: the timeline, the change that started the impact, the change that ended it, what slowed detection or response, and follow-up items with owners. The SRE chapter lists practices that apply after the incident as well: develop and document the procedures in advance with the people who take part in incidents, use the process routinely so it becomes second nature, and rotate roles so the last incident commander takes a different role next time.
The SRE chapter closes with the team’s own result: formulating an incident management strategy in advance, structuring it to scale smoothly, and using it regularly reduced the team’s mean time to recovery. The SRE chapter reports that result for the author’s team at Google without a percentage.
Incident management system: what the tooling needs to record
An incident management system is the tooling that carries the process: paging, the incident record, the live document, status updates, and the timeline. The criteria in the table come from the 6 steps, not from a product ranking.
| Step | What the system records | Question to ask when comparing tools |
|---|---|---|
| Severity levels | A severity field with defined criteria | Can the level change during the incident, with a timestamp? |
| Declaration | Who declared the incident and when | Can any on-call engineer declare? |
| Roles | The incident commander, the ops lead, and the communication lead | Is the current incident commander visible to everyone? |
| Live document | One document per incident | Does the document keep its edit history for the review? |
| Restore and evidence | Changes made before and during the incident | Can deploys, configuration edits, and flag edits be lined up by time? |
| Review | Follow-up items with owners | Are follow-up items tracked until they close? |
Loglune is a feature flag platform, not an incident management system. Loglune covers the flag part of the restore-and-evidence row: the change history of flag edits, and the flag state an affected customer had at time T.
Editorial example (not a customer case): a flag-related SEV-2 from alert to review
The table follows one incident from the first alert to the review. Times, percentages, and names are illustrative.
| Time | Event | Who | Record |
|---|---|---|---|
| 2:05 p.m. | Invoice export errors rise to 8% of requests | On-call engineer | Alert |
| 2:12 p.m. | Customers report failed exports, and the incident is declared as SEV-2 | On-call engineer, now incident commander | Live incident document opened |
| 2:15 p.m. | Ops lead and communication lead assigned | Incident commander | Roles named in the document |
| 2:20 p.m. | Suspect changes listed: no deploy since morning, invoice-export rollout raised from 10% to 50% at 1:58 p.m. | Ops lead | Suspect change in the document |
| 2:24 p.m. | Rollout returned to 10% | Ops lead | New change event in Loglune |
| 2:40 p.m. | Error rate back to baseline, incident resolved | Incident commander | Document kept for the review |
| Next day | Flag state of one affected customer at 2:05 p.m. rebuilt in staging | Engineer on the follow-up | One reproduction |
In the example, service recovered 28 minutes after the declaration, and the reproduction the next day gave the follow-up engineer the flag state behind one failed export without touching production.
Troubleshooting incident management
- Nobody declares an incident until the problem is large: apply the three declaration questions from step 2, and let any on-call engineer declare.
- Several engineers change production at once: route every change through the ops lead, because the SRE chapter keeps changes during an incident within the operations team.
- Stakeholders interrupt responders for updates: assign the communication role and publish the time of the next update.
- The incident commander goes off shift mid-incident: hand off explicitly and wait for acknowledgment, as the SRE chapter describes.
- Severity debates slow the response: treat the incident as the higher level when unsure, as PagerDuty’s page recommends, and adjust the level later.
- A bug appeared right after a flag edit and nobody knows the flag state at the time: rebuild the affected customer’s flag state at the reported time instead of guessing from the current rule.
FAQ
What is incident control?
Incident control, in the terms of the SRE chapter, means one incident commander at a time, explicit handoffs, and a live incident document, which together keep an incident from spiraling out of control. The SRE chapter does not use the phrase incident control itself, and describes the absence of those three practices as the way an incident spirals out of control.
What is the difference between incident management and problem management?
Incident management restores service after a disruption, and problem management finds and manages the causes of incidents so they do not recur. In ITIL 4, incident management’s role is restoring normal service operations swiftly following disruptions, and problem management’s role is identifying and managing the root causes of incidents to prevent recurrence, including managing workarounds and known errors.
Deciding how much incident process your team needs
Start with the declaration rule, one incident commander, and the live document, because those three answer the hazards the SRE chapter names: freelancing, poor communication, and a narrow focus on the technical problem. Add severity levels once the team has a user-impact metric per service, and add the planning role when incidents last long enough to need handoffs. Record every flag edit next to deploys and configuration edits, so the ops lead can reverse a suspect change and the follow-up engineer can rebuild an affected customer’s flag state at the time of the report.
Rebuild an affected customer's flag state during incident management
Loglune records every flag edit as a change event with the customer, the actor, the time, the target flag, and the values before and after. Pick an affected customer and a time T, and Loglune rebuilds the flag state at T for your development, CI, or staging environment. The Free plan includes 5 reproductions.