Use spot capacity on AWS without breaking things

Why this matters

Spot capacity is spare EC2 capacity that AWS sells at a discount to On-Demand. The trade-off is simple: AWS can take it back with a short warning, so the workload has to expect interruptions. That makes Spot a poor fit for a primary database, a single-node service, or any process that keeps its only copy of state on local disk. It fits batch jobs, CI runners, container workers, queue consumers, rendering, and data processing where work can be retried or split.

Teams get into trouble when they treat Spot as a cheaper version of On-Demand. They pick one instance type, put a stateful service behind it, and then wonder why a reclaim event turns into an outage or a corrupted job. The fix is architectural, not a setting: diversify capacity, keep a small On-Demand baseline for critical parts, and make every unit of work safe to stop and restart.

Before you start

Pick one workload to move first. A stateless worker pool behind a queue is easier than a customer-facing API. Write down what the workload needs: CPU, memory, GPU, local disk, network, and any licensing tied to instance families. Check whether the app writes checkpoints, reads from a queue, or stores state in a database or object store. If it only works on one instance type, fix that before you touch Spot.

Confirm your AWS account has room. Spot has a separate vCPU quota, so check Service Quotas for your Region. Decide which parts stay on On-Demand or covered by a Savings Plan: the control plane, a minimum worker count, and anything that must not stop. Look at the Spot placement score in the EC2 console for candidate instance types and Regions. A low score means AWS is less likely to have spare capacity there.

Set up a pilot environment and an owner. You need someone who can watch job failures, queue depth, and interruption events during the first week. Give the team permission to use EC2, Auto Scaling, EventBridge, CloudWatch, and IAM roles for the workload. Do not start with production traffic.

How to do it

Use an EC2 Auto Scaling group with a mixed instances policy. Add several instance types with similar CPU and memory, across at least three Availability Zones. Let AWS pick from a list instead of pinning one type. Keep a base of On-Demand instances if the service needs a floor, and let Spot fill the rest.

Use the capacity-optimized or price-capacity-optimized allocation strategy. Both look at available spare capacity; price-capacity-optimized also weighs price. Avoid setting a maximum Spot price unless you have a specific reason. The default is to pay the current Spot price, and a low cap can leave you without capacity.

Treat the two-minute interruption notice as a normal event. The Spot instance action field in the instance metadata service returns the action and time. EventBridge also emits EC2 Spot Instance Interruption Warning events. Wire both into a handler that stops taking new work, finishes or checkpoints in-flight work, drains connections, and exits. In Kubernetes, use a node termination handler and PodDisruptionBudgets. In ECS, drain the container instance and let the service reschedule tasks.

Test before you trust it. Run the same job on Spot in a pilot, then kill instances on purpose and confirm the queue recovers. Check that retries are idempotent: the same message processed twice must not double-charge a customer or write duplicate rows. Timeouts and retries should be shorter than the interruption window, or the work should resume from a checkpoint.

1. Pick a safe first workload. Choose a stateless or checkpointed job with a queue or object store behind it. Leave databases and single-node services on On-Demand for now. Move one pool, not the whole platform.
2. Create a mixed instances Auto Scaling group. Build an Auto Scaling group with a mixed instances policy instead of a single launch template type. Add instance types that have the same CPU, memory, and network profile your app expects. Set a base of On-Demand capacity if the service needs a floor.
3. Spread capacity across types and zones. List several instance families and at least three Availability Zones in the policy. Use the Spot placement score to drop Regions or types that look tight. A broader pool gives AWS more places to place your request.
4. Choose an allocation strategy. Select capacity-optimized or price-capacity-optimized for the Spot portion. These strategies look at spare capacity instead of just the lowest price. Do not pin a low maximum Spot price unless you have tested the result.
5. Turn on capacity rebalancing. Enable capacity rebalancing on the Auto Scaling group so AWS can replace instances that show a higher interruption risk. This gives your handler time to drain before a hard reclaim. Watch the group during the first few replacements.
6. Handle the interruption notice. Subscribe an EventBridge rule to EC2 Spot Instance Interruption Warning events and call a small handler. The handler should mark the instance as draining and stop new work. Also check the instance metadata service for the action and time.
7. Drain and checkpoint. Stop accepting new jobs, finish or checkpoint in-flight work, and close connections before the deadline. Make retries idempotent so a repeated message does not create duplicate records. Keep the drain time shorter than the interruption window.
8. Run an interruption drill. Terminate a few Spot instances on purpose while the queue has work. Confirm the Auto Scaling group replaces them and the jobs resume. Repeat the drill after changes to the AMI, container image, or instance types.

How to check it worked

Watch the Auto Scaling group activity history. You should see instances launch and terminate as expected, not a group that sits below desired capacity for long periods. Check the group's CloudWatch metrics for InService instances and group size. If Spot capacity is tight, the group may need more instance types or a different allocation strategy.

Track workload health, not just EC2 health. Watch queue age, job failure rate, retry count, and the number of jobs that resume from a checkpoint. A pool that replaces instances quickly but drops work is not working. Compare the Spot portion of the bill with the On-Demand portion in Cost Explorer or the billing details.

Read the interruption events. You want the handler to log when it received a notice, what it drained, and whether it met the deadline. Run a second drill after any change to the app, the AMI, or the instance types. A Spot setup that passed last quarter can fail after a container image change.

Mistakes to avoid

Pinning one instance type or one Availability Zone is the most common mistake. Spot capacity moves between pools, so a narrow request fails when a broad one would keep running. Another mistake is using Spot for a stateful primary database or for a single-node service with no failover. Keep those on On-Demand or on a capacity plan.

Skipping the drain is the next one. If your handler just kills the process, in-flight requests fail and jobs restart from zero. Store state outside the instance, make writes idempotent, and give the handler a deadline shorter than the interruption notice. Do not set a low maximum Spot price and then blame AWS when capacity does not appear.

Also watch quotas, stale AMIs, and clusters that mix Spot and On-Demand without labels. A node that cannot join the cluster is just wasted capacity. If you want a second opinion on which workloads can move and what baseline to keep, the Cloud cost audit from Cloudverg is a fixed price $4,500 review, but it is one option among several.

Frequently asked questions