Autoscaling at Modal

Modal runs your code in containers, on machines you never have to think about. The autoscaler determines how many of those containers exist for your function.

I spent the summer working on this. Besides searching for the best formula or scaling algorithm, the hardest part was testing these potential changes.

You can't A/B test an autoscaler

The naive way is to ship a change and watch. It's slow, expensive, and not fair: traffic does not repeat exactly in production, so two versions never face the same test.

So I built a simulator. It works as a discrete event model with one controller loop and one coroutine per simulated container. It can replay real production traffic, or generate synthetic patterns like spikes, waves, or redeploys. This means hours of traffic play in a couple seconds.

With this, I could run different approaches against identical traffic across more than a hundred scenarios, and compare them on latency, cost, and wasted capacity. There was no clear winner but useful tradeoff curves instead.