Incident Response

Auto-Generate Incident Postmortems

When a PagerDuty incident resolves, Opulent drafts a structured postmortem with timeline, root cause, and action items.

APIMCP

AuthorOpulent

CategoryIncident Response

FeaturesAPI, MCP

start it with one message

Set up a resolved-incident-to-postmortem pipeline. When an incident is marked resolved in FireHydrant or Rootly, start an Opulent run that pulls Sentry signals, builds a timeline, identifies the root cause, lists action items, and posts the draft postmortem to #incident-reviews.

Run this in OpulentCopy it, swap the names for your own, and send it.

connected systems

SentryReview errors, analyze root causes, and suggest fixes for rapid issue resolution

SlackRead and write Slack conversations in Manus

FireHydrantManage incidents, alerts, and retrospectives with FireHydrant

RootlyManage incidents, on-call schedules, and alerts with Rootly

step 1

Connect incident and signal sources

Connect FireHydrant or Rootly so Opulent can read incident metadata: title, service, urgency, created and resolved times, and the incident URL. Connect Sentry for error signals that support the timeline.

The postmortem structure lives in a playbook, a reusable, named set of steps Opulent follows every run. It defines the sections, severity classification, and where the draft lands.

Playbook: !postmortem

When an incident resolves:
1. Read the incident record: title, service, severity, start and
   resolution times.
2. Pull Sentry errors for the service during the incident window.
3. Check for deploys, config changes, or upstream events in the
   same window.
4. Build a timeline from first signal to resolution.
5. Identify root cause and contributing factors.
6. Draft action items with owners and due dates.
7. Post the structured postmortem to #incident-reviews.

Sections: Summary, Timeline, Root Cause, Impact, Action Items.

Tip

Add memory entries about your architecture and service ownership so the postmortem connects the incident to the right downstream services and teams without a human explaining the map each time.

step 2

Deploy the webhook bridge

Create a small service that listens for the incident management tool's 'resolved' webhook and calls the Opulent API to start a postmortem run. Deploy it as a serverless function or lightweight container.

The handler verifies the webhook signature, extracts the incident fields, and sends them to Opulent with the !postmortem playbook. Filter by event type so only resolved incidents trigger a run.

Bridge endpoint: POST /incident-resolved

Headers to verify:
- X-Webhook-Secret (stored as WEBHOOK_SECRET)

Body fields to forward:
- event.event_type == "incident.resolved"
- event.data.title
- event.data.service.summary
- event.data.urgency
- event.data.created_at / resolved_at
- event.data.html_url

Opulent API call:
POST /v3/organizations/{org_id}/runs
{
  "prompt": "A [service] incident has resolved. Draft a postmortem.",
  "playbook": "!postmortem",
  "tags": ["postmortem", "service:{service}"]
}

step 3

What Opulent generates

When an incident resolves, Opulent follows the playbook and drafts a structured postmortem. It correlates Sentry signals with the incident window, names the likely root cause, and lists concrete action items.

The draft is posted to a channel where incident reviewers can read it, correct it, and copy the final version into the wiki or incident management tool.

# Postmortem: Database connection pool exhaustion, orders-service
**Date:** 2026-02-10 | **Duration:** 46 min | **Severity:** P1

## Summary
orders-service experienced connection pool exhaustion between
14:32 and 15:18 UTC, causing 502 errors for ~12% of order
placement requests.

## Timeline
- 14:15 UTC, Deploy #387 released (commit e4f29a1)
- 14:28 UTC, Connection pool usage climbed from 60% to 92%
- 14:32 UTC, Pool exhausted; incident triggered
- 14:38 UTC, On-call engineer acknowledged
- 14:45 UTC, Deploy #387 added a new inventory check that opens
  a DB connection per line item without releasing it in finally
- 15:02 UTC, Rollback to Deploy #386 initiated
- 15:18 UTC, Connection pool recovered; incident resolved

## Root Cause
A new function opened a DB connection per line item and only
released it on the success path. Failed inventory checks leaked
connections until the pool exhausted.

## Action Items
- [ ] Fix connection leak with finally block (PR #388)
- [ ] Add pool-usage monitor with alert at 80%
- [ ] Add integration test for multi-item orders with inventory
      failures
- [ ] Review other DB access patterns for similar leaks

step 4

Customize and route by severity

Tailor the playbook to match your team's postmortem process: required sections, severity thresholds, and where to store the output. Route by severity so only P1 and P2 incidents generate full postmortems; lower-severity incidents get a lightweight summary.

Instead of posting to Slack, have Opulent commit the postmortem to a docs/postmortems/ directory or create a wiki page, depending on where your team keeps incident records.

step 5

Sharpen the postmortem loop

After each review, correct the draft and write the corrections into memory (the notes a run recalls next time): 'this service depends on that one; always check the upstream status.' The next postmortem starts closer to your team's actual narrative.

Track action items in Linear or your incident tool so the postmortem does not become a document nobody follows. Link each item to an owner and a due date.

The natural chain: when the postmortem points to a code fix, hand it to Auto-Fix Failing CI Builds or Daily Sentry Error Fixes; when it reveals a missing monitor, route it to Daily Datadog Health Digest.