Data platform lead
Owns the Spark configuration and the throughput target.
CONCURRENCY · SMALL JOBS
High-concurrency small jobs turn into an orchestration project
Running many small concurrent jobs on Spark becomes a scheduling problem solved with thread pools. A stateless engine absorbs the bursts without the workaround.
THE PROBLEM
Running many small concurrent ETL or agent jobs on Spark is treated as an orchestration problem to solve with thread pools and schedulers, because the engine is built for large jobs rather than high concurrency of small ones.
THE PROBLEM ANATOMY
Who it hits
Data engineer
Owns the notebooks and jobs, and writes thread-pool code to fan them out.
Platform SRE
Owns the cluster the concurrent jobs contend on.
The situation
A team wants to run many small jobs at once, each doing little work. On Spark the advice is that this is an orchestration problem: wrap the work in a function and fan it out with a thread pool, or lean on the scheduler, rather than expecting the engine to handle the concurrency itself.
That turns a runtime concern into application code. Every team rebuilds the same fan-out plumbing, and the cluster still contends when the small jobs pile up.
The specific challenges
Built for big jobs
the engine favors large tasks over many small ones
Orchestration workaround
concurrency is handled in thread-pool code, not the engine
Repeated plumbing
each team rebuilds the same fan-out logic
Cluster contention
small jobs still compete for the same shared resources
WHAT IT COSTS
PRACTICAL
Custom fan-out
engineers write and maintain thread-pool code per workload
Tuning time
cluster sizing gets revisited every time concurrency grows
BUSINESS
Slower delivery
work goes into plumbing instead of pipelines
Capped throughput
small-job concurrency hits a wall the engine will not clear
EMOTIONAL
Reinvented wheels
the same orchestration hack shows up in every project
Quiet frustration
a runtime problem lands on application developers
THE WAY OUT
A lakehouse compute engine with stateless per-vCPU scaling
Swap in a pilot of e6data's query engine alongside the existing Spark stack to:
Absorb the bursts
stateless services add and remove single vCPUs as concurrent jobs arrive, so many small jobs run at once without a thread-pool wrapper. The jobs and open tables stay in place. What changes is that concurrency is handled by the engine, not the application.
Drop the plumbing
with the engine scaling on its own, the fan-out code and scheduler tricks stop being the way to reach throughput. The orchestration that still fits the work stays. What is removed is the per-workload concurrency hack.
Keep the stack
the tables, catalog, and existing Spark jobs stay as they are, and adoption is an endpoint change rather than a migration, needing only fast validation of the required integration mapping. What is added is headroom for small-job concurrency.
1,000+QPS without a thread pool
1-vCPUstateless scaling
0fan-out code to maintain
The result is small-job concurrency the engine handles on its own, instead of plumbing every team rebuilds. Setup and connection details are in the query engine documentation.
TALK TO US
Book a demo on your own workloads
Reach out to book a demo, share challenges you’re facing, and tell us how this fits into what you’re currently working on or thinking about.
Problems we’re solvingThanks.
An actual person replies, usually within one working day.