From Chaos To Control: Building systems that won’t blink under pressure

Have you ever uploaded a file and just crossed your fingers hoping it would actually work? That feeling was the daily reality of an old data ingestion system at Ema.

It was a chaotic free-for-all, where every task from processing a tiny text file to a massive 100-slide PowerPoint was thrown into the same lane. The result? Unpredictable performance, cascading failures, and a system that would fall over when we needed it most.

This pushed us towards a complete re-architecture of our data orchestration layer at Ema, in line with the constant push to build systems that scale with our ambition.

The goal became to design an intelligent ingestion platform that could handle anything, gracefully. We reimagined how data moves, how workloads are orchestrated, and how systems stay resilient under pressure.

More power was the problem, not the solution

Our first instinct was to throw more hardware at the problem, but we quickly learned a hard lesson: adding power just created bigger traffic jams. The core issue was a fundamental flaw in our thinking.

As our initial design proposal stated, "Scaling infrastructure does not translate into proportional throughput. We often face either resource underutilization or overload, leading to failures." Any fixed concurrency limit was doomed to fail; it would either be set too low, underutilizing our hardware, or too high, guaranteeing overload from a few heavy jobs.

The system treated a heavy task (like converting a 100-slide PPTX) and a light task (a 2-page text file) as equals. Heavy jobs would grab all the available resources, overwhelming the worker processes and causing them to crash. This not only blocked lighter tasks but often brought the entire pipeline to a halt. The old architecture tied our two main services, the Go-based Data Ingestion Service (DIS) and the Python-based Transform Service together with direct gRPC calls. This tight coupling created a single point of failure, amplifying crashes during high-load scenarios.

Separating the 'ask' from the 'act'

The breakthrough was architectural, not algorithmic. We needed to fundamentally decouple the request for work from the execution of that work. To do this, we moved from direct, uncoordinated calls to an orchestrated, queued workflow using Temporal. If the old system was a chaotic mosh pit, the new one is run by the master conductor of an orchestra.

The change itself sounds deceptively simple, but its impact was profound:

"We've completely separated the request for a job from the actual execution of that job. It sounds simple, but that one change is what gives us all the power to manage our resources without anything blowing up."

This separation was implemented using a "Sub-Activities Pattern." Instead of one monolithic, high-stakes operation, every complex task is broken down into a simple, 3-step workflow: Prepare → Remote Activity → Complete.

Each step is an independent, observable, and retryable unit of work. This means any hiccup during the 'Remote Activity' step can be retried independently, without having to re-run the entire 'Prepare' step, saving significant compute and time. This pattern guarantees that temporary resources, like files created during processing, are always cleaned up, preventing resource leaks.

Beyond retries, this approach significantly improved cost efficiency and observability. By isolating expensive operations into discrete steps, we eliminated wasted compute when failures occurred mid-pipeline. Additionally, each sub-activity now emits independent metrics and logs, allowing us to pinpoint performance bottlenecks and failure causes at a far more granular level.

Every worker needed its own "smart bouncer"

Decoupling orchestration wasn’t enough; our workers were still at risk of overload. The solution was a custom resource-based slot supplier, a “smart bouncer” for every worker that continuously checks CPU and memory before letting a new task in. If there’s capacity, the task comes in. If not, it waits. This mechanism gave us predictable stability and let us run workers right at their safe maximum without tipping into crashes.

But the real lesson wasn’t just resource guarding; it was understanding how concurrency actually works across different runtimes:

Go vs Python: Same Problem, Different Playbooks

Go: Concurrency comes from goroutines, which are extremely lightweight (a few KB stack). Goroutines are multiplexed onto a smaller pool of OS threads by the Go scheduler (the M:P:G model: Machines = OS threads, Processors = logical schedulers, Goroutines = tasks). If a goroutine blocks on I/O, the scheduler parks it and immediately schedules another goroutine on the same thread. Parallelism is controlled by GOMAXPROCS, which caps how many threads can run goroutines simultaneously across CPU cores.

Python: Threads are real OS threads, but the Global Interpreter Lock (GIL) allows only one thread to execute Python bytecode at a time. This means threads are great for I/O-bound tasks (since the GIL is released during syscalls) but not useful for CPU-bound parallelism. For true parallelism, Python relies on multiple processes (for example, multiple worker processes or Kubernetes pods). Each process has its own GIL, so you get one core per process. Async frameworks (for example, asyncio) take a different approach: a single-threaded event loop drives thousands of concurrent tasks cooperatively; ideal for network-heavy workloads.

Kernel Threads and Scheduling

Both Go and Python ultimately rely on the kernel’s thread scheduler, but they use it differently. Go’s runtime does additional user-space scheduling, while Python delegates most concurrency to the kernel. This is why Go can run millions of goroutines by multiplexing them over a few threads, while Python cannot, though it can handle large I/O loads efficiently with async loops.

Why It Mattered in Our Case Temporal workers straddle both worlds: our Go ingestion service (DIS) can run thousands of goroutines with minimal overhead, naturally saturating all available cores when needed. Our Python transform workers, on the other hand, rely on multiple processes (Pods) to achieve true parallelism, while using threads or async for I/O concurrency inside each process.

Crucially, the smart bouncer is not a generic gatekeeper; it’s runtime-aware. In Go, it monitors goroutine pressure and ensures the runtime never oversubscribes beyond GOMAXPROCS. In Python, it accounts for the GIL and memory pressure, throttling tasks so that async execution doesn’t starve processes or cause thread contention.

The net effect? Workers run hot but never burn out. And when we need more throughput, we scale horizontally with Kubernetes HPAs. HPA reacts to CPU metrics; we use queue depth telemetry to guide scaling parameters and capacity planning, bringing new worker pods online as load increases. Each pod then uses the slot supplier to decide how much work it can safely accept. Together, these two layers form a feedback loop. The orchestration layer distributes tasks intelligently, and the runtime layer ensures every pod runs at peak efficiency without overload. This combination ensures near-linear scalability. As backlog grows, HPA adds capacity, and the slot supplier guarantees that each new pod operates at peak utilization without destabilizing the system.

One of the most impactful but invisible parts of this work was observability. We integrated OpenTelemetry metrics directly into the Temporal SDK, exposing key workflow and activity-level metrics. A background component (called StartQueueStatsPoller) exports queue depth, backlog size, and processing rates, which are scraped by Prometheus and visualized in SigNoz. This visibility transformed operations from reactive debugging to proactive capacity planning.

A "rosetta stone" for our services

One of the biggest hidden challenges was a technical language barrier. Our orchestration service (DIS) is written in Go for its speed and concurrency, while our processing service (Transform) is written in Python for its rich data science and AI libraries. Getting them to communicate flawlessly through Temporal was harder than it looked.

This "Protobuf Challenge" arose because the two languages handle data serialization differently. Go prefers camelCase for field names in JSON (fileName), while Python expects snake_case (file_name). On top of that, nested protobuf objects, arrays, and complex data types behave differently during serialization.

Without a translation layer, these mismatches weren’t cosmetic; they could corrupt payloads, cause silent workflow failures if not handled carefully or introduce subtle, hard-to-debug errors.

To solve this, our engineers built a custom Temporal data converter. This component acts as a "Rosetta Stone," translating data between the two services in a type-safe way, handling field naming, nested structures, and serialization nuances seamlessly. It's a prime example of the deep, foundational engineering required to build a truly reliable system.

We're now looking on to what we can build next that scales with our ambition.