Autoscaling is one of the most difficult (and thus useful) features that are found in scalable software. This is difficult because it’s often not just one piece of software that needs to be scaled. In a simple three-tier web deployment model, it’s three pieces of software: the load balancer, the application servers, and the database. Each of these need to be handled differently and have their own quirks. For AppScale 1.12.0, we’ve revamped how we autoscale application servers to better serve users. Let’s talk about how!
Let’s start off with an example to show you what the software stack looks like in AppScale. Suppose you deploy a single Google App Engine app to AppScale. That results in us deploying your app with the following technologies:
- Load balancer: nginx for static file serving and SSL, and haproxy for health checks
- App Server: A modified version of the open source App Engine SDK (CherryPy for Python/Go/PHP, and Jetty for Java)
- Database: Cassandra and ZooKeeper
Scaling Before AppScale 1.12.0
What’s changed in AppScale is where we place each of these services. Before AppScale 1.12.0, a typical multinode deployment would look like the following:
Here, we place nginx on the head node by itself, and it proxies traffic to haproxy on the slave nodes, which also run AppServers (and usually also the database). Every 20 seconds, each slave node queries haproxy to see how much traffic is coming in on each app. If more than 5 requests are queued up, waiting to be served, we try to add an AppServer on that node. We add up to 10 AppServers per node, and once a node is “full”, it tells the node that another node should be added. If two nodes tell the head node that another node should be added, we actually go ahead and add the new node in. This prevents the head node from accidentally scaling up if one node gets overloaded but everyone else is ok.
The same process occurs for downscaling. Here, if a node notices that there are zero requests enqueued with haproxy, then it removes an AppServer. It never removes the last AppServer for any app though. This is nice because it means a user that shows up at a random time will always have at least one instance ready for them. If it gets down to that last AppServer, it tells the head node to remove a node, and just like in the upscaling case, if the head node sees two nodes asking, it removes a node.
This approach is nice, and has some cool perks. Primarily, the head node doesn’t need to worry about the slave nodes: they autoscale themselves up and down, without it needing to be involved. However, this approach has a big downside: if every node is maxed out at 10 AppServers, and a new node is added, it initially starts with 1 AppServer. But since haproxy runs on all the nodes, nginx just sees haproxys and considers them all equal. This causes too much traffic to be sent to the new node and not enough traffic to be sent to the old nodes. This imbalance causes requests to be lost, which sucks!
Scaling In AppScale 1.12.0
AppScale 1.12.0 remedies this problem by moving haproxy up the chain:
Now, haproxy runs on the same node as nginx, and AppServers run by themselves on the slave nodes. This solves the traffic imbalance, because now, haproxy considers all AppServers to be equal, and sends them traffic equally (instead of forcing nginx to do this). But this requires a tweak in how scaling is done.
Now, every 20 seconds, the head node queries haproxy to see how much traffic comes in on each app. If more than 5 requests are queued up, waiting to be served, it tries to add an AppServer on the first slave node. It will add AppServers there as long as enough traffic comes in, up to 10 AppServers. At that point, it moves on to the next slave node. If all of the slave nodes are “full”, it adds a new node. This time around, we don’t need to wait for two nodes, since we know that all nodes actually are busy.
Downscaling happens in a similar fashion. Here, if the head node notices that there are zero requests enqueued with haproxy, it removes an AppServer off the first slave node. Again, it never removes the last AppServer, but at that point, it moves onto the next node and starts removing AppServers there. If all nodes are down to their last AppServer, then it removes a node. Like in the upscaling case, we no longer have to get a confirmation from a second node, as the one haproxy has a global view on the traffic coming to all nodes.
The new autoscaling support is pretty cool! But there are still some more tweaks that we can do here:
- Let users indicate what the scaling up and scaling down thresholds are, via the AppDashboard web UI and the CLI
- Amazon EC2 charges for instances on a per-hour basis, so instead of killing instances, leave them around until the end of the hour. This would help if we use a box for 20 minutes, don’t need it anymore, but then do need it before the hour time limit is reached.
- Let users indicate the maximum and minimum number of AppServers to run on a single machine.
With that, check out our new AppServer autoscaling in AppScale 1.12.0 and let us know what you think!