Canary Deployment: 6 Steps to Roll Out in Stages

A canary deployment sends a change to a small part of production for a limited time, compares that part with the rest, and uses the comparison to decide whether the rollout continues. This guide runs a canary deployment in 6 steps: choose the canary population, pick the metrics and pass criteria, set the size and duration, deploy and compare against the control, roll forward in stages or roll back, and record which version or flag value each customer received.

The steps apply to a service that runs on servers the team controls, where traffic can be split between builds or a feature flag can switch a code path per user. It is for engineers who ship changes to such a service and want a measured signal before everyone gets the change. Software installed on users’ devices needs extra work, covered in the FAQ.

What canary deployment means

The Google SRE workbook chapter on canarying releases defines canarying as deploying a change to part of a service for a limited time and evaluating it, and the evaluation decides whether the rollout proceeds. In the workbook, the part of the service that receives the change is the canary, and the rest of the service is the control.

Danilo Sato’s note Canary Release on martinfowler.com (June 25, 2014) describes rolling out a change to a small subset of users before rolling it out to everybody. Sato notes that the technique is also called a phased rollout or an incremental rollout, and traces the name to miners who carried canaries in cages into coal mines.

Strategy	What moves	Rollback	Source
Canary deployment	A share of traffic or users to the new build, compared with a control	Route the canary users back to the old build	SRE workbook, Sato
Blue-green deployment	All traffic from one production environment to an identical second one	Switch the router back	Fowler
Flag percentage rollout	A share of users to a new code path inside one deployed build	Lower the percentage or switch the flag off	Hodgson, flagd
Rolling update	The current instances, replaced incrementally by instances of the new build	Revert the Deployment to a previous version	Kubernetes documentation

The Kubernetes tutorial Performing a Rolling Update describes a rolling update as incrementally replacing the current Pods with new ones so that a Deployment update takes place with zero downtime, and says any Deployment update can be reverted to a previous (stable) version.

A rolling update ends with every instance on the new build, while a canary deployment holds the new build on a small part of production and compares it with the control. In the SRE workbook, that evaluation against the control is the step that decides whether the rollout proceeds.

Before you run a canary deployment

The SRE workbook lists three capabilities a canary process needs: a method to deploy the change to a subset of the service, an evaluation that decides whether the change is good or bad, and integration of that evaluation into the release process. Four inputs turn those capabilities into a plan.

• A way to send part of production to the new build: a load balancer weight, a separate canary instance group, or a flag rule.
• Metrics that show user-facing problems, starting from the service level indicators the team already tracks.
• A control group that runs the current build over the same time window.
• A rollback path the on-call engineer can run without a new build.

How to run a canary deployment in 6 steps

1. Choose the canary population and how users are selected

Sato lists several ways to choose who sees the new version: a random sample, internal users and employees first, or users chosen by profile. For geographically distributed users, Sato suggests rolling out to one region first, and for a company with several brands, one brand first. Sato also describes Facebook running several canaries, the first one visible to internal employees with all feature flags turned on, so problems with new features show up early.

When the canary splits by user rather than by request, each user has to stay in the same group for the whole canary. A user who moves between canary and control on alternate requests experiences both builds, and a report from that user cannot be tied to one of them.

2. Pick the metrics and the pass criteria before the deploy

The SRE workbook says a canary metric first has to indicate problems in the service, and recommends service level indicators as the place to start. The workbook adds that the success ratio alone is not enough for a meaningful canary, and that some metrics, such as queue depth, need more time or a larger canary population to give a clear signal.

Write the pass criteria before the deploy starts. For example, the canary’s error rate stays within 0.5 percentage points of the control’s error rate, and the canary’s 95th percentile latency stays within 10% of the control’s. The numbers in the example are illustrative and depend on each service’s normal variation.

3. Set the canary size and duration

The SRE workbook says the canary process risks a small fragment of the error budget, limited by the canary’s duration and population size. In the workbook’s example, a 5% canary that serves 20% errors produces a 1% overall error rate. The workbook ties duration to development velocity: a team that deploys continuously, for example 20 times a day, needs a much shorter canary. Sato’s note says a canary rollout should complete in minutes or hours, while an A/B test can take days.

The workbook also asks for a canary large enough and long enough to be representative of the full deployment. After a few canaries, choose size and duration from the typical failure rates of past canary evaluations instead of hypothetical worst-case scenarios, as the workbook suggests.

4. Deploy to the canary and compare it against the control

Deploy the new build to the canary population and compare the canary’s metrics with the control’s over the same time window. The SRE workbook says that when the canary’s error rate is too far from the control’s, the canary deployment is bad, and the response is to pause and roll back or to bring in a person to troubleshoot.

Run one canary at a time for a service. The workbook says simultaneous canaries add significant mental effort to track system state and raise the risk of signal contamination when the canaries overlap. Loglune does not compare canary metrics with the control, so the comparison runs in the team’s monitoring system.

5. Roll forward in stages or roll back

A passing canary grows in stages, such as 5%, 25%, 50%, and 100%, with the pass criteria checked at each stage. A failing canary rolls back, which in Sato’s note means routing the canary users back to the old version. Sato also points out that a canary gives a capacity test of the new version in production, with a rollback path if problems appear.

Database changes need care in both directions. Sato recommends ParallelChange, which lets the database support both versions of the application while the rollout runs, so a rollback does not meet a schema that the old version cannot read.

6. Record which version or flag value each customer received

A canary spreads one release across hours, so a customer report from the middle of the rollout needs the stage the rollout was in and the group the customer belonged to. For an infrastructure canary, write the build version into every request log line, so a report can be matched to canary or control.

When the canary runs as a flag percentage rollout, the build stays the same and the flag rule decides the code path. Loglune evaluates flags on its server and assigns percentage buckets with the same MurmurHash3 hashing as the flagd fractional operation (checked September 16, 2026). It compares each hash against cumulative weights, so a customer lands in the same bucket whichever Loglune SDK sends the request. Every change to the rollout percentage in Loglune is appended to the change history as a change event with the customer, the actor, the time, the target flag, and the values before and after.

To reproduce a customer-specific bug reported from the middle of the rollout, pick the customer and the time T of the report. Loglune rebuilds the flag state at T from the change history and delivers it in reproduction mode to a development, CI, or staging environment, and one delivery counts as one reproduction. Loglune reproduces the flag evaluation, not the root cause in application code.

Loglune’s example dataset shows two reports on the same flag landing on different versions. The dataset is synthetic input labeled Example data in the Loglune console at loglune.com/app, visible without signing in, and it is not a customer or a real incident. The dataset has no percentage split: its flag checkout-v2 moves from a rule for the segment beta-users (version 2, August 8, 2026) to on for everyone (version 3, August 20, 2026). The evaluation record ex-fp-1 was taken on version 2 at August 18, 2026, 13:22 UTC and returned checkout-v2 = off with reason default, and ex-fp-2 was taken on version 3 at August 23, 2026, 07:48 UTC and returned checkout-v2 = on with reason default. Loglune’s Bisect a regression panel, with checkout-v2 returning on as the bad result, returns verdict none for a free-plan customer at August 18, 2026, 13:22 UTC, because checkout-v2 still returned off then, and names the August 20, 2026, 11:15 UTC edit (version 3) for the same customer on the full history.

Canary deployment vs. blue-green deployment

A blue-green deployment moves all traffic at once, while a canary deployment moves a share of traffic and compares it with a control. Martin Fowler’s note on blue-green deployment (March 1, 2010) keeps two production environments as identical as possible and switches the router from the live environment to the new one, with a switch back as the rollback. Choose blue-green when a fast full switch and a fast full rollback matter more than a measured signal, and choose a canary when the team wants a signal from a small share of production before everyone gets the change.

Canary deployments with feature flags

Canary deployments and flag rollouts work at different layers. A canary deployment routes a share of traffic to a new build, and a flag rollout keeps one build and switches a code path for a share of users. Sato’s Facebook example combines the two layers: the first canary is internal and runs with all feature flags turned on.

A team can run a short infrastructure canary for the build and then raise a flag’s percentage for the feature, with a separate rollback path at each layer. The build rollback reverts code, and the flag rollback lowers the percentage without a deploy.

Editorial example (not a customer case): a checkout change through a canary and a flag rollout

The table follows one checkout change through an infrastructure canary and a flag rollout. Times, percentages, and names are illustrative.

Time	Stage	Who gets the change	Check	Result
9:00 a.m.	Build v42 to the canary instance group	5% of requests	Error rate within 0.5 points of the control	Pass after 30 minutes
9:30 a.m.	Build v42 to all instances	Every request, new checkout path switched off	Health checks	Pass
10:00 a.m.	Flag new-checkout on for internal users	Internal team	Manual checks	Pass
11:00 a.m.	Flag rollout	10% of customers	Checkout error rate and latency	Pass
1:00 p.m.	Flag rollout	50% of customers	Checkout error rate and latency	Customer report at 1:20 p.m.
1:40 p.m.	Flag state of the reporting customer at 1:20 p.m. rebuilt in staging	Production unchanged	Reproduction shows the customer in the new-path bucket	Bug fixed in v43

In the example, the build passed its canary, and the defect surfaced during the flag rollout. The reproduction placed the reporting customer in the new-path bucket, so the team fixed the new path instead of searching the old one.

Troubleshooting canary deployments

• The canary passes but the full rollout fails: check that the canary size and duration were representative, as the SRE workbook asks, and lengthen the canary for slow signals such as queue depth.
• Canary and control metrics look the same during a real problem: confirm that the compared metrics indicate user-facing problems, starting from service level indicators such as the error rate and the 95th percentile latency that the step 2 pass criteria use.
• Two canaries overlap and the signal is unclear: run one canary at a time, because the SRE workbook warns that overlapping canaries risk signal contamination.
• The same user sees old and new behavior on alternate requests: split by a stable user or customer key instead of by request.
• A rollback fails because the old build cannot read the new schema: make the schema change compatible with both versions first, the ParallelChange approach in Sato’s note.
• A customer report arrives hours after the rollout moved on: rebuild that customer’s flag state at the reported time instead of inferring the bucket from the current percentage.

FAQ

Is a canary deployment the same as A/B testing?

No. A canary deployment and an A/B test share technical plumbing but answer different questions. The SRE workbook describes canarying as effectively an A/B testing process, while Sato’s note recommends keeping the two apart: a canary detects problems and regressions, an A/B test checks a hypothesis with variant implementations, and an A/B test can take days to gather enough data while a canary rollout should finish in minutes or hours.

What are the downsides of a canary deployment?

A canary deployment spends part of the error budget, spreads one release across minutes or hours, and needs routing, metrics, and database changes that work for two builds at once. The SRE workbook says the canary process risks a small fragment of the error budget, that some metrics such as queue depth need more time or a larger canary population to give a clear signal, and that simultaneous canaries raise the risk of signal contamination. Sato’s note Canary Release says canaries are harder for software installed on users’ devices and recommends ParallelChange so the database supports both versions while the rollout runs.

Can installed or mobile software use canary deployments?

Yes, with more work: canary deployments are harder for software that runs on users’ devices, because the team has less control over when each installation upgrades, as Sato’s note points out. Sato suggests ParallelChange when the distributed software talks to a backend, so the backend supports both client versions while the team monitors which versions are in use.

Deciding when a canary deployment is worth running

Run a canary deployment when a change carries risk that tests cannot rule out and the service has metrics that show user-facing problems within the canary’s duration. Skip the canary for a change the team can roll back at once and that touches no shared state. When the risky part is a feature rather than the build, run the canary as a flag percentage rollout and keep the change history, so a report from the middle of the rollout can be traced to the flag state one customer had at that moment.