Migrations

Refactor a Multi-Million-Line ETL Monolith

Delegate data-class migrations across a multi-million-line ETL monolith as parallel, verified work packages, the pattern behind order-of-magnitude efficiency gains.

Advanced CapabilitiesPlaybooks

AuthorOpulent

CategoryMigrations

FeaturesAdvanced Capabilities, Playbooks

start it with one message

Refactor our multi-million-line ETL monolith into sub-modules by delegating data-class migrations as parallel, verified work packages. Define the target module structure, split the work by domain, and run each package through the migration playbook with human review at the end.

Run this in OpulentCopy it, swap the names for your own, and send it.

connected systems

GitHubManage repositories, track code changes, and collaborate on team projects

DatabricksQuery lakehouse data and AI tools with Databricks

SnowflakeQuery structured and unstructured data in Snowflake using natural language

LinearTrack issues, manage projects, and organize workflows across your team

step 1

Define the target architecture and migration playbook

Connect GitHub for the code, Databricks or Snowflake for the data catalog, and Linear for tracking. Opulent starts by reading the monolith and proposing a target module structure: which data classes belong to which module, and which dependencies cross module boundaries.

The migration follows a playbook, a reusable, named set of steps for moving a single data class or transformation from the monolith into a sub-module while preserving behavior.

Playbook: !etl-refactor

For each data class or transformation:
1. Identify the target module based on domain and current usage.
2. Trace imports: find all files that reference this class.
3. Move the class implementation to the new module.
4. Update imports in dependent files to point to the new module.
5. Run the test suite for affected modules.
6. Run a representative data pipeline sample against the moved
   class to verify parity.
7. Open a PR with the change, import graph, and test results.
8. If uncertain, flag for human review instead of merging.

Tip

Start with a pilot module of 10-20 classes. The first migration package teaches Opulent the patterns and edge cases of your monolith, which makes the later packages faster and more consistent.

step 2

Split the monolith into independent work packages

Opulent groups data classes by domain and dependency graph. Packages are independent when they do not move the same class or update the same imports. Domain boundaries and shared utilities are the natural split points.

Schedule shared or foundational modules first. Modules that depend on them start only after the foundation merges, so the import updates are stable.

Module split plan:

Module: Billing transformations
- 18 data classes, isolated from other domains.
- Run: parallel package 1.

Module: Customer identity
- 12 data classes, used by billing and risk.
- Run: parallel package 2 (foundational, starts first).

Module: Risk scoring
- 22 data classes, depends on customer identity.
- Run: parallel package 3 (starts after customer identity merges).

Stop rule: if a class is imported by more than one module, keep
it in a shared module until its owner is clear.

step 3

Run packages in parallel with human review

Each package is a separate Opulent run. The run moves the assigned classes, updates imports, runs tests, and opens a PR. A human reviews the PR before merging, especially for the first few packages.

Opulent also builds small scripts for mechanical steps it will repeat across many packages, for example, detecting the country extension of a class from its file path or generating the new import statements. These scripts accumulate and speed up the whole effort.

Package: Customer identity (12 classes)

Actions:
- Mapped 47 files that import these classes.
- Moved class implementations to modules/customer-identity/.
- Updated imports across 38 files.
- Ran module tests and Snowflake sample pipeline.
- Opened PR #145: refactor: extract customer identity module.
- One class flagged for human review because of circular import.

step 4

Verify parity and merge

Before merging, run a representative pipeline sample against the moved classes and compare the output to the monolith baseline. This is the parity check: the refactored module must produce the same results as the original code.

After a package merges, update the monolith's import map and re-run the full suite for the affected modules. Repeat until the monolith shrinks to the shared core and the new modules are independent.

step 5

Sharpen the ETL refactor loop

When a package hits an edge case, add the rule to the playbook or write it into memory (the notes a run recalls next time). After a few packages, the same edge cases stop surprising Opulent.

Add an eval (an automated test for the run) that checks the import graph after each move: no circular imports between modules, and all moved classes are reachable from the new module's entry point.

The natural chain: when the refactor reveals a testing gap, use Add Unit Tests to Your Payments Service; when a data quality issue appears, use Data Analysis as an Agent-Run Workflow; when the work spans many repos, use Fleet-Wide Maintenance Across Thousands of Repositories.