Auto-Investigate Datadog Alerts
Wire PagerDuty or Datadog alerts to Opulent for automatic incident investigation.
Connect your observability tools and write a triage playbook
Connect the tools an on-call engineer already opens during an incident: Sentry for errors, Grafana for metrics and dashboards, and Coralogix for logs and traces. Each connector gives a run read access to that data without a human pasting it in.
The investigation quality comes from the playbook, a reusable, named set of instructions that tells Opulent how your team investigates, not just "look into the alert". Save it once under Playbooks and reference it by name from every run.
Write the checklist the way an engineer would work the alert: pull the recent errors, group them, correlate with the last deploy, then read the code the trace points at. Specific beats long.
Playbook: !triage-alert An alert just fired for one of our services. Then: 1. Pull error logs and events for the named service from the last 30 minutes (Sentry, Coralogix). 2. Group by error message and name the dominant failure. 3. Check for a deploy to that service in the same window and note the commit. 4. Read the source file and recent commits for the failing path. 5. Summarize: root cause, affected files, the suspect deploy, and a suggested fix. If the cause is clear, open a hotfix PR.
Give the run read-only access to your production database and Grafana so it can confirm a metric or query a table itself, instead of guessing from the alert text alone.
Build the alert-to-Opulent bridge
An alert should start a run with no human in the loop. That takes a small webhook service that receives the alert payload and calls the Opulent API to open a run against your triage playbook. Deploy it as a serverless function or a small container.
The handler reads the fields the alert monitor sends (title, service tag, link), builds the kickoff message, and names the playbook to run. The service authenticates with an API token you generate once for a service user.
from flask import Flask, request, jsonify
import requests, os
app = Flask(__name__)
@app.route("/alert", methods=["POST"])
def handle_alert():
payload = request.json
alert_title = payload.get("title", "Unknown alert")
tags_str = payload.get("tags", "")
service = next(
(t.split(":", 1)[1] for t in tags_str.split(",")
if t.strip().startswith("service:")),
"unknown-service",
)
alert_url = payload.get("link", "")
org_id = os.environ["OPULENT_ORG_ID"]
requests.post(
f"https://platform.opulentia.ai/api/v1/organizations/{org_id}/runs",
headers={"Authorization": f"Bearer {os.environ['OPULENT_API_KEY']}"},
json={
"prompt": (
f"Alert fired: '{alert_title}'\n"
f"Service: {service}\n"
f"Alert link: {alert_url}"
),
"playbook": "!triage-alert",
},
)
return jsonify({"status": "run started"}), 200Create a service user with permission to start runs, copy its API token into OPULENT_API_KEY on the bridge, and set OPULENT_ORG_ID to your organization ID. Never hardcode the token in the handler.
Route your alerts to the webhook
With the bridge live, point your alert sources at it. In Datadog, go to Integrations, then Webhooks, then New Webhook, and set the URL to your bridge endpoint. Add the webhook handle to any monitor's notification message to arm that monitor.
For PagerDuty, open Services, then your service, then Integrations, add a Generic Webhooks (v3) integration, set the URL to the bridge, and filter to the incident.triggered event so only real incidents open a run.
The sharp edge: a noisy monitor becomes a noisy run queue. Arm one warning-level monitor first and read what it produces before routing critical alerts, and keep flapping alerts off the bridge until their thresholds are tuned.
Route by severity from the start. Send P1 alerts to a playbook that opens a hotfix PR, and P3 alerts to a playbook that only posts a root-cause note, so a low-priority blip never opens code changes.
Watch a run investigate a live alert
A monitor trips and the bridge opens a run before anyone reads the page. Take this Datadog alert as the concrete case:
"High error rate on payments-service (5.2%, threshold 1%)" fired at 14:32 UTC. The run follows the triage playbook end to end:
It works through the checklist as named operations, then lands its findings where the team already is, posted to your incident channel and, when the cause is clear, opened as a hotfix PR linked back to the alert.
Run actions on payments-service alert: - Pulled error logs for the last 30 min and grouped by message , 92% are "Unhandled rejection in handlePaymentIntent". - Found deploy #492 (commit abc123f) released at 14:28 UTC, three minutes before the error rate climbed from 0.3% to 5.2%. - Read src/webhooks/stripe.ts: the deploy moved the handler to async/await and dropped the try/catch around handlePaymentIntent(). - Opened PR #493 with an error boundary, structured logging, and 4xx responses for client errors. - Posted the timeline and root cause to #incidents.
What the run leaves behind
When the run finishes, an engineer arriving at the alert finds the investigation already done and verifiable item by item:
A timeline in your incident channel tying the error spike to a specific deploy and commit. A named root cause pointing at the exact file and change. A hotfix PR (when the cause was clear) linked to the alert, with the fix and reasoning in the description.
The run's steps are its own audit trail (which logs it read, which deploy it correlated, which files it opened) so a reviewer can retrace the reasoning before merging the fix rather than trusting a summary.
Sharpen the pipeline over time
When runs keep missing something (a related service, a log source they never check) add that step to the !triage-alert playbook, or write it into memory (the persistent notes a run recalls on later incidents) as a rule like "payments logs live in Coralogix, not Sentry".
Seed memory with your service context: normal thresholds, architecture, and on-call runbooks, so an investigation starts from your team's knowledge instead of from scratch. Ask Opulent to read past incidents and draft those entries for you.
The natural next link: once investigations are trustworthy, chain the resolved incident into Auto-Generate Incident Postmortems so a closed alert produces a timeline and action items without anyone writing the doc.