Daily Datadog Health Digest
Daily session that scans Datadog for errors and posts a health digest to Slack.
Connect the observability sources and define thresholds
Connect Sentry for error signals, Grafana for metrics and dashboards, and Coralogix for logs and traces. Opulent reads from all three to build a unified daily health view.
The digest format lives in a playbook, a reusable, named set of steps Opulent follows every run. It defines the thresholds that turn a metric into a finding and the severity scale for the team.
Playbook: !daily-health-digest For the past 24 hours: 1. Error rate, flag any service above 1%. 2. p99 latency, flag any endpoint above 500ms. 3. CPU and memory, flag anything above 80% utilization. 4. Monitor status, list any monitors currently in Alert or Warn. 5. Newly recurring errors, errors that reappeared after being resolved. For each finding include: - Severity (Critical / Warning / Info) - Affected service and current metric values - One-line recommended next step If everything is healthy, post an all-clear summary.
Add memory entries that distinguish normal from abnormal for each service, like 'payments-service normally runs at 200ms p99; anything above 400ms is a concern.' Without baselines, every spike looks like an outage.
Create the daily schedule
Set the schedule to run daily 30-60 minutes before standup so the team can review findings during the meeting. Route the output to a Slack channel where on-call and engineering leaders already gather.
The sharp edge: vague thresholds produce noisy digests that get ignored. Be specific about what counts as critical versus warning, and tune after the first week of real data.
Schedule: Daily health digest - Frequency: daily at 07:30 (30 min before standup) - Slack channel: #ops-alerts - Playbook: !daily-health-digest - Lookback: past 24 hours
What the team sees each morning
Opulent queries Sentry, Grafana, and Coralogix, compares the results to the thresholds, and posts a formatted digest. Findings are grouped by severity so the critical items are visible first.
The team can reply in the thread to dig deeper; Opulent keeps the context from the scan and can run follow-up queries or open an investigation run.
Daily Health Digest, Feb 10, 2026 CRITICAL - payments-service: error rate at 3.2% (threshold: 1%) Started 4h ago, correlates with deploy #487. -> Check /api/checkout handler and recent changes. WARNING - search-service: p99 latency at 620ms (threshold: 500ms) Gradual increase over 3 days. Likely index degradation. -> Review query plans for product search. - web-app: memory at 82% (threshold: 80%) Trending upward since Tuesday. -> Investigate session handler for possible leak. INFO - All other services within normal parameters. - 2 monitors recovered in past 24h.
Schedule weekly knowledge updates
Daily digests generate corrections: 'this is a known baseline,' 'ignore this monitor during deploys,' 'this spike always correlates with the nightly job.' Schedule a weekly run to turn that feedback into memory entries.
Over time, the digest becomes more accurate (fewer false alarms, better severity ratings, and smarter recommendations) because the weekly run remembers what the team has already decided.
Sharpen the digest loop
When a recurring finding is always noise, add an ignore rule to the playbook or write the context into memory (the notes a run recalls next time). When a finding always leads to the same fix, link the playbook step to that fix workflow.
Add new signal sources as the stack changes (a new APM tool, a cost metric, a business KPI) so the digest stays aligned with what the team actually cares about.
The natural chain: when a digest item needs root-cause analysis, hand it to Auto-Investigate Datadog Alerts or Anomaly Detection Across Connected Data; when the cause is known, route it to Auto-Fix Failing CI Builds or Daily Sentry Error Fixes.