The problem
The summary lands at the worst time. An incident closes at 2am, and the on-call engineer still has to write what happened for the morning handoff, the postmortem, support, and anyone who watched the status page. The details sit in PagerDuty alerts, a busy Slack channel, deploy logs, dashboards, and a runbook that may be stale. Writing takes focus the engineer no longer has.
Raw timelines are not summaries. PagerDuty and Slack exports show events, not meaning. A reader still has to work out what customers felt, which change mattered, what the team tried, and what remains open. An AI agent can assemble that draft from approved systems and cite each claim. It should not guess a root cause or post to customers on its own.
AI agent for incident summaries for on-call: a playbook is about one workflow: turning a closed or active incident into an internal draft with evidence, then routing it through a named human before it goes wider. Keep the scope that narrow. Broad incident command is a different problem.
What a good solution looks like
A good draft has a fixed shape. Title, severity, start time, detection time, resolve time, customer impact, current status, actions taken, open questions, next steps, and owners. Put suspected cause in its own field and label it as a hypothesis. Add an evidence list with links and timestamps for every factual claim.
Tool access should come through APIs, not screen scraping. Read tools: PagerDuty or Opsgenie incident records, Slack channel history for the incident, GitHub or GitLab deploys, observability queries for the incident window, status page update history, and the relevant runbook. Write tools: create a draft document in Confluence or Google Docs, and open follow-up tickets in Jira or Linear. The write tools need approval.
Approval rules need to be explicit. The agent may create an internal draft automatically. A named on-call lead or incident commander must approve any status page update, customer email, severity change, incident closure, or ticket that assigns work to another team. The approval screen should show the exact text and the evidence behind it.
The best drafts say what is unknown. If logs do not show a cause, the draft says the cause is under review. If a customer impact number is missing, it says the number is missing rather than inventing one. A short summary beats a long one padded with guesses. Keep the internal draft to a page, and keep customer-facing text shorter.
How to build it
Start with the output contract. Write the fields you need and the rules for each one. Then pull twenty to forty past incidents from your own systems and build an evaluation set. Redact secrets and customer identifiers before you store it. For each case, write the summary a senior engineer would accept. That set becomes your test, not a demo script.
Build thin tool adapters around the official APIs. Normalize every event into a common record: timestamp, source, actor, event type, link, and raw text. Query by incident ID and a time window, not by keyword search across everything. Fetch only the fields you need. This keeps prompts small and makes tool errors easier to see.
Have the agent write JSON that matches your schema, then render it into a document. Put hard rules in the prompt: cite evidence, mark inference, do not state a cause without a log line or change record, and leave a field blank when data is missing. Include a few good and bad examples from your own incidents. The examples matter more than clever wording.
Add approval gates, token and time budgets, and full logs of inputs, tool calls, and outputs. Log the prompt version too. Build a replay mode so you can rerun the same incident after a prompt change. Plan a tuning window after launch, because the first real incidents will show gaps your evaluation set missed.
Risks and how to manage them
The main risk is a confident wrong cause. An agent can connect a deploy to an outage that started before the deploy. Force evidence links next to every cause claim. Have the reviewer check the timeline before approving. In the evaluation set, include incidents with unknown causes so the agent learns to say so.
The second risk is automation bias. A tired reviewer approves a clean-looking draft without reading it. Make the approval screen show the evidence, not just the text. Require the reviewer to pick a reason when they edit or reject. Review the first several dozen approvals closely, then sample after that.
Data handling needs care. Incident channels often contain customer names, email addresses, access tokens, and raw request bodies. Use allowlisted fields, redact before sending text to the model, and keep the agent out of channels it does not need. Log access so you can trace what was sent.
Tool permissions are another failure point. Give the agent read-only access by default. A write action should run through a separate service account with narrow scopes and an approval step. Add rate limits, a kill switch, and alerts when tool calls fail or when the agent tries an action outside its allowlist. Do not let it close incidents or page people.
How to measure success
Measure the workflow, not the model. Track the time from incident resolution to a posted internal summary. Track reviewer minutes per summary. Count drafts approved with no edits, drafts with small edits, and drafts rejected. Count missing required fields and wrong cause claims caught in review. Count follow-up tickets created with enough detail to act on.
Use a before-and-after sample. Pull the last twenty incidents before launch and the first twenty after. Score both against the same checklist: impact, timeline, cause evidence, actions, open items, owners. Have a reviewer who did not build the agent score them. That keeps the result honest.
Watch cost and reliability separately. Track tokens per summary, tool calls per summary, failed tool calls, and summaries that needed a manual rerun. Set a budget per incident and alert when a run crosses it. A cheap draft that misses the cause is not cheap.
The measured goal is a draft the on-call engineer can approve with light edits in less time than writing from scratch. If the edit time stays high, fix the schema or the retrieval before adding features. If you want a fixed scope for one workflow, Cloudverg's AI agent build package is $19,500 at a fixed price over 5 to 6 weeks and includes an evaluation set from your past cases, approvals, logging, budgets, and a monitoring dashboard. It is one option among others.
Frequently asked questions
What should the AI agent be allowed to do without human approval?
It can read approved systems, assemble a timeline, and create an internal draft. Anything that messages customers, changes severity, closes an incident, or assigns work to another team needs a named human approval. Keep write tools narrow and logged.
How do you stop the agent from inventing a root cause?
Require evidence for any cause statement and label it as a hypothesis. Include incidents with unknown causes in the evaluation set. Make reviewers check the timeline and the linked log lines before they approve. If the evidence is thin, the draft should say the cause is under review.
How do you measure success for incident summaries?
Track time from resolution to posted summary, reviewer edit minutes, approved drafts with no edits, missing fields, and wrong cause claims caught in review. Compare a sample of incidents before and after launch with the same checklist. Cost per summary and tool failure counts belong in the same report.