Why this matters
A bad release is a clock problem before it is a code problem. Every minute the broken version takes traffic, more users hit the error, more support tickets pile up, and more bad rows land in the database. The goal in the first ten minutes is not to find the bug. It is to stop the bleeding.
There are two exits. Roll forward with a hotfix, or roll back to the last version that worked. Rolling back is usually faster, because the artifact already exists and already passed tests. Use a simple rule: if you can describe the fix in one sentence and ship it in five minutes, roll forward. Otherwise roll back now and fix calmly afterwards.
A rollback that only swaps code is half a rollback. Migrations, config values, feature flags, cache entries and background jobs all moved forward with the release. This guide, How to roll back a bad release in minutes, walks through the whole set, not just the deploy step.
Before you start
You need immutable artifacts and a short history of them. Every deploy should point at a specific build number, image digest or commit SHA, and the previous two or three versions must still exist in the registry. If your rollback process rebuilds from a branch, you are shipping untested code under pressure.
Write down what each release contains while things are calm. One line per deploy: who shipped it, which commit, which migrations ran, which flags changed. During an incident that note is the difference between rolling back the right thing and guessing.
Name a decider before you need one. Someone on call has the authority to say roll back, and someone, often the same person, runs the command. Everyone else investigates. Group decisions during an outage are slow and produce half measures.
Keep the command and the target next to each other in a runbook. Exact command, dashboard link, the version you would return to, and the name of the person who owns the database. Rehearse it in staging once a quarter so the muscle memory survives staff changes.
How to do it
Start by confirming the release is actually the cause. Put the deploy time next to the first spike in errors or latency. If the error started twenty minutes before the deploy, you have a different problem and a rollback will waste the window.
Then freeze the pipeline. Cancel running deploys, pause the queue, stop merges to main. Two deploys in flight during a rollback is how a small incident becomes a long one.
Roll back the code first, because that is the fastest single change, then work outward to flags, config, scheduled jobs and data. Tell support and update the status page in the same minute you start, not after it works.
- Confirm the trigger. Compare the deploy timestamp with the first error or latency spike in your dashboard. If the timeline does not line up, stop and look elsewhere before you revert anything.
- Freeze the pipeline. Pause the deploy queue and stop merges to the main branch. Say in the incident channel that a rollback is in progress so nobody ships over it.
- Pick the target version. Choose the last build that was healthy, not the last commit. Take the build number or image digest from your deploy record so you restore exactly the artifact that ran before.
- Redeploy the old artifact. Run the rollback command in your pipeline, or redeploy the previous image by digest. Do not rebuild from a branch, because a rebuild produces new and untested code.
- Turn off the new flags. Disable any feature flags the release switched on. Flags flip in seconds and often clear the visible symptom before the deploy even finishes.
- Decide about database changes. If the release ran an additive migration, leave it and roll back the code. If it dropped or renamed a column, restore from backup or run the reverse script before the old code serves traffic.
- Drain and restart workers. Restart background workers, cron jobs and queue consumers so they stop processing with the new code. Flush caches that hold serialized objects in the new shape.
- Tell the people who answer the phones. Post the rollback and the expected recovery time to support, the status page and the account managers. Users forgive a short outage more than silence.
- Record the state. Note the bad version, the good version and what you changed in the incident ticket while it is fresh. You will need it for the postmortem and for the real fix.
How to check it worked
Watch the signal that started the incident, not just the homepage. If checkout errors triggered the rollback, watch the checkout error rate until it returns to the level from before the deploy. Give it a few minutes of steady readings rather than one green poll.
Verify the version on the running system, not in the pipeline. Check the commit or image digest reported by the app, a health endpoint or your dashboard. Pipeline logs show what you asked for. The running process shows what you got.
Look for leftover damage. Rows written in the wrong shape, jobs queued with payloads only the new code understands, sessions holding new tokens, CDN copies of new static assets. Each one can keep the incident alive after the code is old again.
Then close the loop. Update the status page, tell support the fix is in, and keep the incident open until the permanent fix, or the decision to leave the rollback in place, is written down.
Mistakes to avoid
Rolling back code while leaving a destructive migration in place. If the release dropped a column, the old code will fail on its first query. Design migrations so they survive a rollback: add the new column, write to both, read from the old one, and drop the old column in a later release.
Rebuilding from a branch on the way back. It feels faster than finding the old image, and it ships code nobody tested.
Rolling back silently. Support keeps telling users to clear their cache, on call gets paged a second time, and the next engineer repeats the work from scratch.
Deleting the bad artifact. Keep it. You need it to reproduce the bug in a safe environment, and a rollback you cannot investigate turns into a rollback you repeat next week. If the cycle keeps repeating, the gap is usually in the release process rather than the code: pipelines, environments, a tested rollback and a written runbook. Cloudverg offers that work as a fixed price package, Delivery platform setup, at $14,500 over four weeks, which is one option among several. Writing the runbook and rehearsing it in staging this week is the cheaper first step.
Frequently asked questions
How long should a rollback take?
With immutable artifacts and a one command rollback, swapping the code takes a couple of minutes and the full check takes ten to fifteen. If yours takes an hour, the time usually goes to rebuilding, waiting on migrations or hunting for the right version, not to the deploy itself.
Can you roll back a database migration?
Additive changes usually do not need it: leave the new column in place and redeploy the old code. Destructive changes such as dropped columns or rewritten rows need a restore or a reverse script written before the release. That is why the migration and the code should ship as separate steps.
Should I roll back or roll forward?
Roll back when the fix is unclear or needs more than a few minutes of work. Roll forward when it is a one line change you can ship and verify right away. When in doubt during an active incident, roll back first and fix afterwards with a clear head.