Simulating Spot Instance Termination to Test AWS Node Termination Handler
Spot instances are becoming a popular way to achieve large, 70%+ cost savings on the cloud, but it can be difficult to ensure your Kubernetes workloads continue working correctly when spot instances are reclaimed. For example, what happens when a spot instance is reclaimed but there is no more capacity available in the cluster?
Fortunately, AWS gives a spot instance a 2-minute warning before terminating a reclaimed instance. This warning allows a Kubernetes cluster to combine the power of the AWS Node Termination Handler with Kubernetes Autoscaler to cleanly handle spot reclamations without having to worry about service interruptions or insufficient capacity in your cluster.
The AWS Termination Handler makes sure that your workloads are cleanly drained by stopping new requests from being made to the pods in the reclaimed instance through the Kubernetes cordon operation and starts rescheduling those pods in other nodes through the Kubernetes drain operation. But what happens if there isn’t enough capacity in the remaining nodes? This is where the Kubernetes Autoscaler comes in.
The Kubernetes Autoscaler can detect that there is not enough capacity left in the remaining instances and asks AWS to spin up new instances. For this to work correctly it is important to make sure that your ASGs, the Termination Handler, and the Autoscaler are configured correctly; however, that explanation is outside the scope of this article, what I want to focus on is how do we test this mechanism?
AWS gives a Spot Instance a 2-minute warning by changing the response of the metadata API. The metadata API endpoint /spot/instance-action usually returns a 404 response, but when an instance is about to be reclaimed the metadata API Endpoint http://169.254.169.254/latest/meta-data/spot/instance-action returns with a 200 response along with a JSON body that looks like:
The AWS Termination Handler polls the metadata API for this event as can be seen here: https://github.com/aws/aws-node-termination-handler/blob/v1.3.1/pkg/ec2metadata/ec2metadata.go#L135-L153
With this information, we know that we can simulate this event by mocking the response of the metadata API and configuring the instance to hit our mock response.
Being the l̶a̶z̶y̶ efficient programmer that I am, I found a service that already does this. I can then deploy this service to my Kubernetes cluster by applying the following manifest
apiVersion: apps/v1 kind: Deployment metadata: labels: app: ec2-spot-termination-simulator name: spot-term-simulator spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: ec2-spot-termination-simulator template: metadata: labels: app: ec2-spot-termination-simulator spec: containers: - image: shoganator/ec2-spot-termination-simulator:1.0.1 imagePullPolicy: Always name: spot-term-simulator env: - name: PORT value: "80" dnsPolicy: ClusterFirst restartPolicy: Always terminationGracePeriodSeconds: 1 --- apiVersion: v1 kind: Service metadata: labels: app: spot-term-simulator name: ec2-spot-termination-simulator spec: ports: - name: http port: 8082 nodePort: 30000 protocol: TCP targetPort: 80 selector: app: ec2-spot-termination-simulator sessionAffinity: None type: NodePort
This deploys the mock service but I still need to override the metadata API, before we do this we need to figure out which node the service is running on and shell into that node.
In my cluster I don’t have SSH access to the Kubernetes instances by design, but I can still gain access to the instance configuration through the aws-node pods. I open up a shell in one of these pods through the command:
kubectl exec -it aws-node-<id> bash
We then need to install 2 yum packages so that we can reroute the metadata API, net-tools and socat:
yum install -y net-tools yum install -y socat
We can finally reroute the API traffic and test that our Node Termination Handler and Cluster Autoscaler work correctly by running the commands:
ifconfig lo:0 169.254.169.254 up socat TCP4-LISTEN:80,fork TCP4:127.0.0.1:30000
As soon as we run these commands, we can see that the cluster has started draining the pods, however, I don’t have enough capacity in my cluster to immediately reschedule the pods which causes some pods to be pending
Our cluster autoscaler, however, notices this and immediately asks for new nodes; 90 seconds later our node is ready to go and schedules the pending pods:
Now I’m happy with my configuration and can sleep in peace knowing that spot reclamations are not going to ruin my night.