Scalability
Scalability is the ability to handle more load by adding resources, partitioning state, reducing work, or changing the workload contract. It is not the same as speed: a service can be fast at low traffic and fail to scale because a single queue, database partition, model endpoint, or object prefix saturates. In ML systems, batch and online inference often scale through different mechanisms.
Little’s Law and scaling limits
A useful first approximation is Little’s Law:
where is average concurrency, is arrival rate, and is average time in system. Horizontal scaling then needs a per-replica capacity model:
required replicas = ceil((arrival_rate * service_time) / safe_concurrency_per_replica)That formula is only the start. Managed compute can add replicas, but distributed data processing still needs partitioning and distributed model training still needs communication bandwidth. Reliability also constrains scaling because retries and failover traffic can become the largest load source during incidents.
Worked capacity check
Estimate pods from request rate, p95 service time, per-pod concurrency, and target utilization:
| Workload | In-flight requests | Capacity denominator | Required pods |
|---|---|---|---|
| 240 rps, 0.12 s p95, cap 20, util 0.70 | 28.8 | 14.0 | 3 |
| 1000 rps, 0.08 s p95, cap 50, util 0.65 | 80.0 | 32.5 | 3 |
The second workload has more traffic but shorter service time and larger safe concurrency, so it still needs three pods. Real autoscalers add stabilization windows, metric lag, startup time, and min/max bounds; the calculation is the baseline to compare against observed HPA or Cloud Run behavior.
Caveats
Scaling the API layer cannot fix a single-writer database, hot key, slow decoder, or saturated GPU KV cache. Average latency hides p99 queueing. Caches improve read scalability but add invalidation and cold-start behavior. Over-scaling can increase cost management risk and make downstream dependencies fail sooner.
References
Nav