What it is
When a system breaks badly enough to need more than one person, someone has to run the response. That is the incident commander. The job is coordination and decisions, not debugging. The incident commander sets severity, assigns roles, keeps the timeline, and decides when to escalate, roll back, or call a vendor.
The role differs from the tech lead. The tech lead owns diagnosis and the fix. The incident commander owns the process around it: who is on the bridge, what customers hear, how often updates go out, and when the team stops investigating and restores service first. Both roles can sit in the same incident, and they should.
An incident commander has named authority for the duration of the incident. They can pull people out of meetings, pause a deploy, order a rollback, or wake a vendor. That authority comes from a written policy, not a job title. Often it is a trained engineer or engineering manager who is not the deepest expert on the failing system, because that expert needs to be free to work.
How it works
An incident starts when an alert fires or a customer reports a problem. The on-call engineer declares it and names an incident commander. The commander opens one channel per incident, writes the severity in the topic, and starts a timeline with times and decisions. They assign a tech lead and a communications contact, plus a scribe on larger incidents.
During the response, the commander runs a simple loop: what we know and what the next checkpoint is. They ask for status instead of giving technical direction. On long incidents they set a cadence, often every 30 minutes, so the room does not go silent. They keep one person speaking at a time and route side questions from sales and executives through the communications contact.
Some moments need a decision rather than more analysis: roll back, fail over, disable a feature, page the vendor, or bring in the database owner. The commander makes the call, takes input from the tech lead, and records the reason in the timeline. They also set the customer update rhythm.
The incident ends when the service is stable and monitoring agrees. The commander declares it resolved, schedules the review within a few days, and hands action items to named owners. Then they close the channel and stand the role down. A short handoff note at the end saves hours in the review.
Why it matters for a business
Without a commander, an outage response drifts. Two engineers debug the same theory. Nobody tells support what to say. The loudest person sets the direction. An incident commander removes that confusion, which shortens the time to restore and makes the response repeatable.
Support gets a clear contact and regular updates. Account managers know when to talk about service credits. Leadership gets information without interrupting the people fixing the system. For audits and customer security reviews, a timeline with decisions and owners shows that incidents follow a process rather than memory.
The role also helps people. It gives engineers a way to lead without being the top expert. It protects on-call staff from doing coordination and repair at the same time. Hard calls, like rollback or feature shutdown, become normal and documented instead of personal.
Common mistakes
Appointing the best debugger as incident commander. The room goes quiet, the commander disappears into logs, and nobody is running the response. Keep the two roles separate, even in a small team.
Giving the role no authority. If every rollback needs a director's approval at 2 a.m., the commander is decoration. Write the authority into the policy and make sure managers respect it during the incident.
Skipping severity definitions, or writing ten levels nobody remembers. Three or four levels with clear examples work better. The same goes for the timeline. Memory rewrites itself after a stressful night, so write down what was tried and why.
Forgetting handoffs. Long incidents cross shifts. A verbal handoff loses detail, so use a written summary: current state, what was tried, what is next, who owns it. Keep the review blameless. If people expect punishment, they stop reporting near misses, which are the cheapest warnings you get.
How to get started
Write a one page incident policy. Cover severity levels with examples, who can declare an incident, who can serve as incident commander, how to page, and what authority the role carries. Name a rota with at least three trained people so the role does not depend on one person.
Set up the basic tools: one channel per incident, a shared timeline document, a status page, and paging that reaches a backup. Test the paging path. Run a game day with a fake outage and let a new commander practice while a trained one watches.
Train by shadowing. A new commander watches one real incident, then runs a low severity one with a buddy. Review each response for process gaps: late updates, unclear ownership, missing decisions. Keep the notes short and change one thing at a time.
The technical side matters as much as the process. Fast rollback, separate environments, and monitoring you trust make a commander's decisions easier to execute. Cloudverg's Delivery platform setup at $14,500/month is one option: four weeks to put pipelines, staging and production environments, secret management, monitoring and alerting, and a tested rollback in place. Other options include hiring a site reliability engineer or building the same pieces in house.
Frequently asked questions
Does the incident commander fix the problem?
No. The incident commander runs the response and makes decisions, while the tech lead and responders diagnose and fix. On very small incidents one person may hold both roles, but say out loud which hat is on.
Who should be an incident commander?
Usually a trained engineer or engineering manager who stays calm, communicates clearly, and can decide with incomplete information. They do not need to be the strongest debugger. Rotate the role across several people so it does not become one person's permanent second job.
When should an incident be declared?
Declare when a customer-visible service is degraded, when the team needs more than one person, or when a decision is needed that one engineer should not make alone. Declaring early costs little. Waiting for certainty usually costs more.