Antifragile Deployments
Use production signals to deploy agents, verify them, and improve memory, runbooks, evals, and tool rules with every release.
Connect production signals and define the verification playbook
Connect GitHub for deploy events, Vercel for deployment status and performance metrics, and Sentry for error signals. Opulent watches these signals to decide whether a release is healthy or needs rollback.
The verification logic lives in a playbook, a reusable, named set of steps Opulent follows every run. It defines what 'healthy' means for this service, how long to wait, and what to do if signals degrade.
Playbook: !verify-deploy For every production deploy: 1. Capture the deploy SHA, timestamp, and affected service from GitHub/Vercel. 2. Wait for the soak window (e.g., 15 minutes or 1,000 requests). 3. Pull error rate and p99 latency from Sentry and Vercel for the deploy window. 4. Compare to the pre-deploy baseline from the same time window yesterday. 5. If error rate < 0.5% and latency regression < 10%: mark healthy. 6. If any signal breaches: mark degraded, trigger rollback, and open a post-mortem issue. 7. Write the result to memory with the SHA and the decision reason.
Start with a generous error-rate threshold and tighten it as the playbook proves itself. A threshold that fires on every deploy teaches the team to ignore the signal; one that never fires hides real regressions.
Wire the trigger: every deploy or on a schedule
Trigger the playbook automatically when Vercel reports a successful production deployment, or when GitHub Actions emits a deployment event. The run starts only after the deploy is live, so the signals are real.
For services that deploy continuously, schedule a rolling verification every 15 minutes instead of per-deploy, so bursts of small releases do not trigger overlapping verification runs.
The sharp edge: verification that runs too early sees no traffic and looks healthy; verification that runs too late delays rollback. Tune the soak window to your request volume.
Watch a deploy prove itself or fail fast
Take a deploy that updates the checkout flow. Opulent captures the SHA, waits for 1,000 requests, and pulls error and latency data for that window.
Error rate is 0.3% versus a 0.2% baseline, and p99 latency is 180ms versus 175ms baseline, both within thresholds. Opulent marks the deploy healthy and writes the result to memory.
A later deploy that changes the session handler shows error rate spiking to 2.1% and p99 latency to 310ms. Opulent marks it degraded, opens a rollback request, and files a post-mortem issue with the signals attached.
Deploy: checkout-flow@v2.4.1 (SHA a1b2c3d) Soak window: 1,000 requests - Error rate: 0.3% (baseline 0.2%, threshold 0.5%) -> PASS - p99 latency: 180ms (baseline 175ms, threshold +10%) -> PASS - Decision: HEALTHY - Memory update: v2.4.1 verified under load. Deploy: session-handler@v2.5.0 (SHA e5f6g7h) - Error rate: 2.1% -> FAIL - p99 latency: 310ms -> FAIL - Decision: DEGRADED -> trigger rollback + post-mortem issue.
What exists when the run finishes
A verification run produces a decision record: the deploy SHA and timestamp; the signals compared to baseline; the pass/fail verdict with the threshold values; and the action taken, promote, rollback, or extend soak. If it failed, it also includes a post-mortem issue with the signals and the likely next investigation step.
The memory entry is the proof-of-work: every future run recalls which deploys passed, which failed, and why, so the loop learns instead of repeating the same mistakes.
Sharpen the loop with every release
When a deploy fails for a reason the playbook missed, add the signal or check to the runbook and write the lesson into memory (the notes a run recalls next time). If the same failure pattern appears twice, add an eval (an automated test for the run itself) that checks the playbook would catch it.
Use passing runs to expand the verification: add a check for mobile performance, a key user journey, or a business metric once you trust the basic signals.
The natural chain: when a deploy degrades, hand it to Auto-Investigate Datadog Alerts or Auto-Generate Incident Postmortems; when it passes, feed the verified pattern into the eval suite so the next agent version starts smarter.