Scalability: definition, horizontal vs vertical scaling and how autoscaling works

2026.11.17 Clever Cloud Bannière Blog Scalabilité EN
When traffic grows, you are usually offered two fixes. A bigger machine. Or more machines. Both assume that your application can make use of them, and that is not a given.

That is precisely what scalability measures: the ability of an IT system to handle a growing load (more users, requests or data) when it is given extra resources, without its performance degrading in proportion. Scalability plays out on two levels: the infrastructure supplies the resources, and the application has to be able to use them. A platform that scales automatically does not, on its own, make an application scalable.

What is scalability in computing?

The word comes from the verb to scale: to change size or proportion. In practice, a system is scalable when you can multiply its capacity by adding resources, and the gain stays close to the effort you put in.

Take an API that tops out at 500 requests per second on two servers. With four servers, it should get close to 1,000. If it stalls at 600, a bottleneck is hiding somewhere else: a database, a lock, a shared file. Scalability is therefore measured under load, and always against a specific resource.

The word is also part of business vocabulary, where a “scalable” model grows its revenue faster than its costs. The two meanings often meet, because an online service can only take on new customers if its infrastructure keeps up. And making the infrastructure keep up brings us back to the two fixes from the start.

Horizontal vs vertical scaling: two ways to add capacity

These two fixes have names: vertical scaling and horizontal scaling.

Vertical scaling (scaling up) gives more resources to the same machine: more vCPUs, more memory. It asks nothing of the application, which keeps running as a single copy. Its limit is therefore physical: the largest instance size your provider offers sets the ceiling.

Horizontal scaling (scaling out) instead adds identical instances of the application, spread behind a load balancer. The ceiling moves much further away, but the price is paid in the code: every instance must be able to handle any request, which means storing state (sessions, files, caches) outside the instance.

Criterion Vertical scaling (scale up) Horizontal scaling (scale out)
Mechanism A larger instance replaces the previous one Identical instances are added in parallel
Ceiling The largest instance size available The maximum number of instances, and above all the capacity of the dependencies
Requirement on the application None No local state: the application must be able to run as several copies
Effect on the database Same number of connections, but more queries to handle, unless a cache absorbs them One more connection pool per instance
Availability A single instance remains a single point of failure If one instance fails, the others keep responding
Typical use case Software that handles redundancy poorly, monolithic application APIs, stateless web applications, workers

How to choose between horizontal and vertical scaling

The nature of the application decides, and two questions help you settle it.

First, can your application run as several copies? Some software handles redundancy poorly, because it synchronises internal state or writes to a local disk. Such software scales vertically, by design. If your application works with multiple instances, horizontal scaling is usually the simplest lever: you add workers that run in parallel.

Second, will your dependencies keep up? Each extra instance opens its own connections to the database. Ten instances with a pool of twenty connections each add up to two hundred simultaneous connections for the database. If it is not sized for that, adding instances moves the bottleneck without removing it, and you are back to the API stuck at 600 requests per second.

Bigger or more numerous machines add capacity. That capacity should still not be confused with two related notions.

Scalability, elasticity and high availability: three different notions

These three terms come up together in architecture discussions and are often confused, because they all deal with resources.

Scalability tells you whether the system can handle more load when it is given more resources.

Elasticity adds movement: resources follow demand, up and down, without human intervention. Autoscaling is what produces that movement. An application can therefore be scalable without being elastic, if every change in resources goes through a person.

High availability answers a different question: whether the service holds up when a component fails. It relies on redundancy. Two instances deployed for availability do not necessarily add useful capacity, and a single instance, however large, remains a single point of failure.

Autoscaling is how a platform delivers elasticity. It belongs to the first level, the infrastructure.

Autoscaling: how a platform decides to scale

Autoscaling adjusts an application’s resources without human intervention, based on observed metrics. To do this, an autoscaler needs to know what to watch and at which threshold to react. The ranges within which it acts are yours to set.

Why CPU and memory are not treated alike

An autoscaler can watch two kinds of signals. Some measure resources: available memory or disk, or CPU, whose load is, strictly speaking, a cost drawn from a resource. Others describe the application’s performance or health, such as the number of requests or connections, or response time. Autoscalers differ in the signals they track and in how they react to them.

At Clever Cloud, our autoscaler watches CPU and memory, and does not treat them the same way. When CPU activity stays high for a sustained period, over a window of around one to two minutes, it scales up. When free memory drops below a threshold, it reacts immediately, without waiting for the situation to be confirmed. The difference comes from how serious each situation is. A saturated CPU slows the application down or makes it refuse new connections: your users feel it, but the application recovers on its own once the load falls. Running out of memory, on the other hand, can kill the process, or even corrupt data, and is far more costly to recover from.

To scale down, the autoscaler follows the opposite logic: it waits for CPU load to stay low over a longer window before removing resources. Kubernetes applies the same principle to its Horizontal Pod Autoscaler: by default, it scales up without delay and waits five minutes before reducing the number of replicas. Scaling up fast and down slowly stops an application from oscillating between two sizes with every passing spike.

Ranges: two settings and a spending cap

You do not set these thresholds. You set the ranges within which the platform acts. Each instance, which we call a Scaler, is an independent virtual machine defined by its CPU and memory. For horizontal scaling, you choose a minimum and maximum number of Scalers, up to 40. For vertical scaling, a minimum and maximum size, from pico to 3XL (16 vCPUs and 32 GiB of memory). You configure this in the Console or from the command line:

clever scale --min-instances 2 --max-instances 6
clever scale --min-flavor S --max-flavor XL

When both ranges are set, the platform favours vertical scaling: it first enlarges the running Scalers, then adds more once the maximum size is reached. With sizes from S to L and 2 to 4 Scalers, scaling up therefore follows the order 2-S => 2-M => 2-L => 3-L => 4-L, in line with the rule described in our documentation.

These ranges also act as a spending cap. Billing is based on instance size and running time, per second: the maximum therefore sets the highest possible spend for the application. Finally, be careful with the smallest sizes: pico and nano run with a lower CPU priority on the host. If your application is sensitive to latency, choose a larger minimum size.

What happens during scaling: no build, a boot, a switchover

Once the decision is made, adding a Scaler does not rerun the build. We reuse the build cache and start a virtual machine of the required size, which takes a few seconds. On top of that comes your application’s own start-up time: a JVM loading a large context takes longer than a Go binary.

Vertical scaling, for its part, replaces one instance with another and follows the same path as a redeployment. The new instance starts, and we check that it responds: on port 8080, then on the paths declared in CC_HEALTH_CHECK_PATH if you have configured any. Only then does Sōzu, our load balancer written in Rust, send it new traffic. Connections still open on the old instance get a grace period to finish, then the old instance shuts down. During this overlap, two instances run, but only one receives new requests.

This overlap is the default behaviour, controlled by the “Zero downtime deployment” option in the Console, zero-downtime on the command line. If you disable it (clever config update --disable-zero-downtime), the old instance stops before the new one starts, and scaling then causes an interruption. Cases that justify this setting are rare, apart from an application that genuinely cannot run as two copies, even briefly.

For a rise in CPU load, expect a few minutes between the start of the spike and the new capacity taking traffic. All of this belongs to the first level, which the platform handles on its own. The second level, the application, is down to you.

What stops an application from scaling

Adding ten instances achieves nothing if the application keeps its state locally: at best, the new instances sit underused; at worst, they return inconsistent results. Before enabling autoscaling, four points are worth checking.

Sessions stored in local memory

A user logs in on instance A, their next request lands on instance B, which does not know them, and they are logged out. The fix is to store sessions in a shared service, typically a key-value database. If your application keeps its sessions in memory and you cannot change it right away, enable sticky sessions with clever config update --enable-sticky-sessions. The load balancer will then always send the same user to the same instance. This workaround lets an application that was not designed for it run as several copies, at the cost of a less even distribution of requests. If your application does not need them, leave them disabled.

Files written to the local disk

Files raise the same problem as sessions. A file uploaded to one instance does not exist on the others, and disappears when the instance is replaced. Persistent files therefore belong in object storage or a shared file system.

Database connections

Instances do share the database, however, and each one adds its own connection pool: that is the two-hundred-connection calculation from earlier. Size the database accordingly, or put a connection manager in front of it, such as Pgpool-II for PostgreSQL or ProxySQL for MySQL.

The database itself

That leaves the database itself. Relational databases such as PostgreSQL or MySQL do not suit the kind of autoscaling that applications get: replacing a live database means keeping its data in exact sync with its replacement, and some engines keep memory allocated even when they no longer use it, which skews the reading of their load. On Clever Cloud, autoscaling therefore applies to applications, not to managed databases. Distributed stores follow a different logic: Materia KV, built on FoundationDB and replicated across three data centres, runs on a cluster sized for a whole region and designed to scale horizontally.

These four points stem from the same principle, formalised by the Twelve-Factor methodology: stateless processes that share nothing with each other, and capacity that grows by adding processes. An application that follows it is well suited to horizontal scaling. Even so, it cannot rely on autoscaling for everything, since autoscaling reacts to load rather than anticipating it.

What autoscaling absorbs, and what it does not

Autoscaling absorbs the variations that are part of your application’s life cycle: the morning ramp-up on a business tool, the lunchtime peak on a food delivery service, the month-end close on an accounting application. These variations repeat and can be observed, and you can adjust the ranges as you test.

It does not absorb shocks, however. An application that goes from a few hundred to hundreds of thousands of requests in a few seconds, because a TV news programme has just mentioned it, saturates before the detection window is over. Such events can be anticipated: if you know the date, raise the minimum number of instances before the event rather than waiting for the autoscaler to react, and put a cache in front of the application (Varnish for HTTP, Redis at application level) to take load off the application and its database. Clément Nivolle, co-founder of Clever Cloud, describes how we got BioDemain’s website ready in 24 hours for an appearance on the French TV channel M6, with Redis for sessions, ProxySQL for MySQL connections and Varnish as an option.

Conversely, a maximum set too high can hide a problem. A memory leak will trigger successive scale-ups until the maximum is reached, and the bill will follow. On Clever Cloud, the Console shows how the number of instances changes, and each application has a Grafana dashboard that collects CPU, RAM and requests per second with no configuration, with threshold alerts. An autoscaler that regularly hits its maximum points either to a range set too low or to an application that consumes more than it should.

Either way, the safest approach is to know your application’s limits before a spike reveals them.

Testing how your application scales before you need it

That is the purpose of load testing: it sends a controlled, increasing load to the application, then observes how it responds in terms of response times, error rate and scaling triggers. You find out at what load the application degrades, whether the autoscaler reacts at the right time and which component gives way first. That component is your next piece of work, and it is not always the application, as with the API stuck at 600 requests per second.

Run the load test on an environment that mirrors production, scaling ranges included, and play out two scenarios. A gradual ramp-up checks how the autoscaler behaves. A sudden surge measures what the application can absorb before the new capacity arrives.

These tests apply whatever scales your application, a managed platform or Kubernetes. The difference between the two lies elsewhere.

Autoscaling on a managed platform or on Kubernetes

Kubernetes comes with its own tooling. The Horizontal Pod Autoscaler adjusts the number of replicas in a Deployment. The Vertical Pod Autoscaler adjusts the resources requested by pods. The Cluster Autoscaler adds nodes when pods no longer fit, and removes them when part of the cluster stays idle. You choose the metrics, write the policies, tune the stabilisation windows and make sure the three mechanisms stay consistent with each other: the Vertical Pod Autoscaler documentation, for example, advises against combining it with the Horizontal Pod Autoscaler on the same CPU or memory metric.

On a managed platform, by contrast, the platform makes those choices: you set ranges, it decides when to act and manages the capacity of the underlying machines. Scaling then sits within the same automated deployment pipeline as your releases to production.

The difference therefore lies in the operating model: the managed platform takes on the operation of scaling, Kubernetes gives you control over it. Both run demanding production workloads, and the choice depends on what your team wants to operate itself. If you are unsure, these criteria for choosing a managed Kubernetes service can help you decide.

Is your application scalable?

Back to the two fixes from the start: a bigger machine or more machines only help an application that can make use of them. An application is scalable when three conditions hold together: it runs as several copies without local state, its dependencies (the database first among them) keep up with rising load, and a load test has confirmed it. Autoscaling then automates capacity that already exists. It does not create it. If you do not know which of these three conditions gives way first in your case, start there, with a simple question: what load brings your application down today?

Q&A – Scalability

What does scalable mean?

A system is scalable when it can change scale: serve ten times more users when it is given more resources, without being rewritten and without its performance collapsing.

What is the difference between scale up and scale out?

Scaling up, or vertical scaling, enlarges an existing instance. Scaling out, or horizontal scaling, adds instances in parallel. The first asks nothing of the code but runs into the maximum size of a machine. The second pushes that ceiling back, provided the application keeps no local state.

What are the disadvantages of horizontal scaling?

Horizontal scaling asks more of the application and of its dependencies. Every instance must be able to handle any request, so sessions, uploaded files and caches have to live outside the instance, in a shared service. Each extra instance also opens its own database connections: ten instances with a pool of twenty connections each mean two hundred simultaneous connections, which the database has to be sized for. If it is not, adding instances moves the bottleneck instead of removing it.

What triggers autoscaling?

An autoscaler reacts to observed metrics. On Clever Cloud, it watches CPU and memory: CPU activity that stays high for a sustained period, over a window of around one to two minutes, triggers a scale-up, and free memory dropping below a threshold triggers an immediate reaction. Scaling down waits for CPU load to stay low over a longer window. You do not set these thresholds. You set the ranges, in number and size of Scalers, within which the platform acts. Other autoscalers, such as the Kubernetes Horizontal Pod Autoscaler, can also react to application metrics, for example the number of requests.

Does autoscaling reduce costs?

It saves you from paying for the peak all the time: resources follow the load, up and down. The saving depends on the gap between your average load and your peak load. An application with a constant load gains nothing, while an application with marked variations gains more. The maximum caps the spend.

Do databases scale automatically?

Not traditional relational databases such as PostgreSQL or MySQL. Replacing a live database means handing its exact state over to its replacement, and some engines keep memory allocated even when they no longer use it, which skews the reading of their load. They are sized manually and placed behind a connection manager. Distributed stores, designed as clusters, scale horizontally by design.

What is a scalable product or business model?

In business terms, a scalable model grows its revenue faster than its costs, like subscription software where each new customer costs little to serve. Technical scalability is one of its conditions: without infrastructure that can keep up, commercial growth runs into technical limits.

Blog

À lire également

Scalability: definition, horizontal vs vertical scaling and how autoscaling works

When traffic grows, you are usually offered two fixes. A bigger machine. Or more machines. Both assume that your application can make use of them, and that is not a given.
Engineering

From the Console to Terraform: import your existing Clever Cloud infrastructure as code (IaC)

Nobody starts with Terraform. You create your first application in the Console, add a PostgreSQL database with Clever Tools, then a Redis instance, a Cellar bucket, and a “temporary” staging environment that is about to turn two. One morning, the organisation has forty resources, and nobody can say why the production application runs on two M instances instead of one, or who changed that variable on a Friday evening.
Engineering

Configure SSO with Keycloak

To configure single sign-on (SSO) with Keycloak, deploy Keycloak as an identity provider, then create a dedicated realm separate from master and a client for each application.
Features