Data platform lead
Owns the warehouse and the SLA for customer-facing analytics.
CONCURRENCY · CLUSTER CAP
Ten concurrent queries per cluster, and a thousand users waiting
Reaching hundreds to a thousand users means stacking clusters and paying for each one. An engine that scales by the vCPU holds the concurrency without the stacking.
THE PROBLEM
A SQL warehouse caps concurrent queries at roughly ten per cluster, so reaching hundreds or a thousand concurrent users means adding clusters and paying for each one.
THE PROBLEM ANATOMY
Who it hits
FinOps partner
Owns the bill that grows with every cluster added for headroom.
Product owner
Owns the dashboard promised to thousands of users at once.
The situation
A team needs a dashboard to serve many users at the same time. The warehouse runs about ten concurrent queries per cluster, so the only way to reach hundreds or a thousand is to run more clusters in parallel.
Each cluster is billed while it is up, so concurrency scales with cost rather than with the work. Contention shows up before the cap is even reached, and response times stretch during peaks.
The specific challenges
A low cap
about ten concurrent queries run per cluster
Clusters as the fix
more users means more clusters, each billed
Early contention
response times stretch before the cap is reached
Peak scaling
spikes force more clusters exactly when the bill is highest
WHAT IT COSTS
PRACTICAL
Cluster sprawl
capacity is added in whole clusters to cover peaks
Peak tuning
engineers size for the busiest minute and pay for it all day
BUSINESS
Linear cost
concurrency and spend rise together
Capped product
a plan for a thousand users stalls on the concurrency ceiling
EMOTIONAL
Budget dread
every new customer cohort reopens the sizing debate
SLA anxiety
peaks threaten the latency promised to users
THE WAY OUT
A lakehouse compute engine with per-vCPU scaling
Swap in a pilot of e6data's query engine alongside the existing warehouse to:
Hold the concurrency
the engine keeps 1,000 or more queries per second at p95 under two seconds without stacking clusters. The dashboards and open tables stay in place. What changes is that concurrency stops being a per-cluster ceiling.
Scale by the vCPU
capacity adds and removes single vCPUs as traffic rises and falls, so spend follows the actual load. The existing peak patterns stay. What gets returned is a cost curve that tracks work, not headroom.
Keep the stack
open tables stay under the current catalog, access controls carry over, and adoption is an endpoint change rather than a migration, needing only fast validation of the required integration mapping. What is added is room for the next cohort of users.
1,000+QPS at p95 < 2s
1-vCPUscaling, not per-cluster caps
0cluster stacking for peaks
The result is concurrency that grows with the user count rather than with the cluster count, on the tables already in place. Setup and connection details are in the query engine documentation.
TALK TO US
Book a demo on your own workloads
Reach out to book a demo, share challenges you’re facing, and tell us how this fits into what you’re currently working on or thinking about.
Problems we’re solvingThanks.
An actual person replies, usually within one working day.