Reason Over Entire Datasets with Async Spawns
Plan the selection, spawn deep work in parallel, review focused intervals, and return evidence with reusable outputs.
Connect the warehouse and define the chunking plan
Connect Databricks, Snowflake, or MotherDuck as the primary warehouse; add Bigdata.com when the dataset needs external market or financial context. Opulent runs the queries directly, so each chunk is verifiable.
The plan lives in a playbook, a reusable, named set of steps Opulent follows every run. It defines how to partition the dataset, what each worker analyzes, and how to combine their outputs, so a full-dataset run is repeatable.
Playbook: !parallel-dataset-reasoning For a dataset or table: 1. Plan: state the question and the grain of the answer (e.g., per-customer, per-day, per-product). 2. Partition: split the dataset into non-overlapping chunks by a natural key, date range, customer segment, region. 3. Spawn: kick off one worker per chunk; each worker runs the same analysis prompt against its slice. 4. Review: inspect each worker's findings for consistency, edge cases, and outliers before merging. 5. Synthesize: combine the focused results into one answer, weighted by chunk size and confidence. 6. Package: return the answer, the partition map, the per-worker evidence, and a reusable skill spec.
Choose a partition key that keeps related records together. Splitting by customer_id is usually safer than splitting by row number, because relationships inside one customer's history stay in one chunk.
Arm the trigger: on-demand or scheduled
Send the initial prompt with the dataset and question filled in. Opulent plans the partition, spawns the workers, and returns the synthesized answer.
For standing full-dataset questions (monthly cohort health, quarterly customer segmentation) schedule the playbook so the answer refreshes automatically and posts to the team.
The sharp edge: partitions that are too small create coordination overhead; partitions that are too large defeat the purpose. Size each chunk so it answers a meaningful slice in one pass, then merge.
Watch a full-dataset question split and merge
Take the question: 'Which customers are at risk of churning in the next 30 days?' The dataset is 4 million activity events across 80,000 customers.
Opulent partitions by customer cohort, spawns one worker per cohort, and has each worker query the last 90 days of events, build a risk score, and return the top risk factors. The workers run in parallel, so the wall-clock time is closer to one cohort than the whole dataset.
Opulent reviews each cohort's findings, catches a false signal in the enterprise segment, and synthesizes the final list with per-customer risk scores, the evidence for each, and the SQL that produced them.
Question: Which customers are at risk of churning in 30 days? Run actions: - Partitioned 80k customers into 8 cohorts by acquisition quarter. - Spawned 8 workers; each analyzed activity, support, and billing signals for its cohort. - Reviewed cohort findings; flagged enterprise cohort for extra validation because of low sample size. - Synthesized final at-risk list: 1,247 customers with scores and top risk factors. - Returned SQL per cohort and a combined view for the team.
What exists when the run finishes
A full-dataset run produces a reusable package: the final answer with confidence; the partition map so a reviewer can see how the dataset was split; per-worker findings with the SQL or queries each worker ran; and a reusable skill spec so the same reasoning can be applied to the next dataset.
The per-worker evidence is the proof-of-work: you can drill into any cohort's logic and re-run it, which makes the combined answer auditable even though it came from parallel work.
Sharpen the parallel reasoning loop
When workers disagree on an edge case, add the resolution rule to the playbook or write it into memory (the notes a run recalls next time), so the next full-dataset run handles the same edge consistently.
Refine the partition strategy as the data grows: move from quarterly cohorts to monthly cohorts, or add a separate partition for high-value accounts, so no chunk becomes too large to reason over in one pass.
The natural chain: once a cohort-level pattern is found, hand it to Data Analysis as an Agent-Run Workflow for a focused, business-readable readout, or to Customer Health Summaries in Slack for account-by-account distribution.