Re-architecting mission-critical
ETL for event-driven scale
How I designed and implemented a new execution model that separated orchestration from compute, removed a scaling bottleneck, standardized workloads with containers, and reduced both processing time and infrastructure cost.
A reporting platform that had
outgrown its execution model.
The platform processed healthcare and financial data across multiple hospital systems, feeding centralized BigQuery tables used by operational, analytical, and revenue-sensitive reporting workflows.
As adoption grew, thousands of ETL jobs increasingly depended on a Cloud Composer environment that was doing two jobs at once: orchestrating workflows and executing compute-heavy processing.
That coupling became the platform’s main scaling constraint. Delayed processing no longer meant only slower pipelines — it affected downstream reporting and business operations.
Composer had become both
scheduler and compute platform.
Apache Airflow was a good fit for workflow orchestration, but the environment had gradually accumulated thousands of tightly coupled processing tasks executing directly inside Composer workers.
As workload volume increased, the orchestration layer itself became a bottleneck.
- Scheduling pressure
- Slower processing throughput
- Dependency conflicts
- Rising infrastructure cost
- Increasing operational complexity
Separate orchestration
from execution.
Rather than continuing to scale Composer vertically, I designed and implemented a new responsibility boundary between orchestration and execution.
Composer would remain responsible for scheduling, dependency management, and workflow state. Compute-heavy ETL stages would move into independently deployable containers running on Cloud Run.
That changed the architecture from a tightly coupled orchestration environment into an event-driven execution model where workloads could scale independently.
The simplest platform that
solved the scaling problem.
I evaluated Kubernetes, but it would have introduced cluster lifecycle and operational overhead that the organization was not yet positioned to absorb.
Cloud Run provided the characteristics the platform actually needed:
- Container-native execution
- Independent horizontal scaling
- Usage-based billing
- Minimal infrastructure administration
- Faster adoption by teams still building container operational maturity
The decision was less about choosing the most sophisticated platform and more about choosing the simplest platform that satisfied the scaling requirement.
Operational simplicity was a feature,
not a compromise.
Containers became the
execution boundary.
Moving execution out of Composer also exposed another problem: workload environments were inconsistent.
Python dependencies, package versions, and runtime assumptions varied between pipelines. Containerization gave every ETL service a reproducible execution boundary.
I standardized workloads around Docker images stored in Artifact Registry, giving teams independently deployable environments with isolated dependencies and consistent delivery patterns.
Modernization could not
trade away recoverability.
The platform supported reporting workflows where missed processing windows had operational and financial consequences, so modernization also had to improve recoverability.
I designed a disaster-recovery approach around reproducible infrastructure, backup Composer environments, DAG synchronization, and Terraform-driven environment reconstruction.
The goal wasn’t an expensive permanently active duplicate platform. It was a recovery model that could recreate critical orchestration infrastructure consistently while keeping the standby footprint economical.
Recovery was treated as
infrastructure state, not tribal knowledge.
The architecture changed
the operating envelope.
85%+
Lower daily infrastructure cost
approx. $2,000/day$300/day
14h 4h
End-to-end processing window
approx. 14 hours4 hours
Externalizing compute reduced pressure on Composer, enabled greater processing parallelism, and shifted the platform toward consumption-based execution.
The architectural change also established containerization and standardized deployment patterns that could be reused by other engineering teams.
Three decisions that
carried forward.
- Lesson 01
Separate control planes from workload execution.
Systems become difficult to scale when orchestration and compute compete for the same operational boundary.
- Lesson 02
Choose operational complexity deliberately.
GKE could have solved the compute problem, but Cloud Run solved it without introducing a second operational problem.
- Lesson 03
Standardization compounds.
Docker was initially an execution mechanism. It eventually became a common deployment boundary that improved reproducibility, onboarding, and service ownership.