Use spot capacity on Google Cloud without breaking things

Why this matters

Spot VMs on Google Cloud use spare compute capacity. They cost less than on-demand VMs, and Compute Engine can stop them when it needs the capacity back. You get a short warning, then the VM goes away. That trade is fine for work that can restart. It is painful for work that holds state in memory and cannot resume.

The useful question is not how to avoid preemption. Assume it will happen, including during a deploy or a long write. Design each Spot workload so a stop is boring: it loses no committed data, it leaves no lock behind, and another worker can pick up the next task. If a workload cannot meet that bar, keep it on on-demand capacity.

Good candidates include batch jobs, CI runners, rendering, log processing, image conversion, test environments, and stateless web workers behind a load balancer. GKE workloads that tolerate termination also fit. Poor candidates include a single primary database, a file server with local state, and any job that cannot be split into repeatable units.

Before you start

Start with a list. For each workload, write what happens if the VM stops with no warning beyond the notice window. Note where state lives: local disk, memory, a database transaction, a file lock, an in-flight upload. If the answer is manual repair, fix that before you create Spot capacity.

Move durable state to a managed service. Use Cloud Storage for objects, Cloud SQL or Spanner for relational state, Pub/Sub for events, and a queue for task leases. Treat the local disk as a cache. If you need a disk, use a persistent disk and plan for reattach, but do not assume the same zone is free when the VM restarts.

Check quotas for Spot VMs in each region you plan to use. Spot capacity can fail to start when the zone has no spare capacity, so plan a fallback: a second zone, a smaller machine type, or an on-demand pool for critical work. Add labels such as workload=spot and team=payments so cost reports separate Spot spend. Pick one tolerant workload for the first run.

How to do it

Pick a pattern before you pick a command. For Compute Engine, build a regional managed instance group from an instance template that uses the Spot provisioning model. Keep the group regional, and test that it brings capacity back after a preemption. For GKE, create a separate Spot node pool with a taint, then add a toleration and node selector to the workloads that should land there. For batch work, use Google Cloud Batch with Spot provisioning and retries.

Add a shutdown path. On Compute Engine, use a shutdown script that flushes work and writes a checkpoint. On GKE, handle SIGTERM in the container, stop accepting new requests, finish or release the current task, and exit. Set terminationGracePeriodSeconds below the notice window, with margin, so the process is not killed mid-flush.

Make retries safe. Give each unit of work an idempotency key. Use a queue lease so a task returns to the queue if the worker dies. Checkpoint after each chunk, not at the end of a long run. Keep chunks small enough that a restart does not replay hours of work.

1. Choose one tolerant workload. Start with a batch job or a stateless worker that can fail without customer impact. Write the failure story for that workload before you change capacity.
2. Move state off the instance. Put job state in Cloud Storage, Cloud SQL, Spanner, or Pub/Sub. Keep only cache and temporary files on the local disk.
3. Add checkpoints and idempotency. Store progress after each small unit of work and key each write so a retry does not duplicate it. On startup, resume from the last checkpoint.
4. Handle the stop signal. Catch SIGTERM or use a shutdown script, stop taking new work, flush, and exit. Set the grace period under 30 seconds and test it.
5. Create Spot capacity behind a fence. Use a separate GKE Spot node pool with a taint, or a regional MIG from a Spot instance template. Keep critical control paths on on-demand capacity.
6. Spread and retry. Run across zones, use retries with backoff, and let the queue redeliver work. Add topology spread constraints so replicas do not share one zone.
7. Roll out in small steps. Send a small share of work to Spot first. Watch retries and completion time before you move more.

How to check it worked

Test the stop path on purpose. On GKE, delete a pod and watch the logs for your shutdown message, checkpoint write, and clean exit. On Compute Engine, stop an instance and check the shutdown script output. Then start it again and confirm the job resumes from the checkpoint, not from zero.

Use Cloud Monitoring to watch the signals that matter: Spot instance count by zone, pod restarts, job retry count, checkpoint age, queue depth, and failed task count. Set alerts for a checkpoint that is older than expected and for tasks that sit in the queue too long. A rising retry count can mean the work is not idempotent or the chunks are too large.

Check capacity errors. If Spot instances fail to start, look for zone capacity errors and confirm your fallback works. In GKE, check for pending pods that cannot tolerate the Spot taint or cannot fit the node shape. Look at cost reports to confirm the Spot labels show up.

Run one full failure drill during a normal work day, not only in staging. The goal is a boring log line, a clean checkpoint, and a retry that finishes. If the drill creates manual cleanup, fix the design before you widen the rollout.

Mistakes to avoid

Putting a stateful singleton on Spot is the common mistake. A database primary, a file server with local uploads, or a job that holds a lock in memory will break in a way that needs a person. Keep those on on-demand capacity or move the state to a managed service first.

Another mistake is using a shared node pool for Spot and on-demand work with no taint. Critical pods land on Spot nodes and get stopped. Use a taint and toleration, plus a node selector, so only tolerant workloads go there. Do not treat a PodDisruptionBudget as protection from Spot preemption; it helps with voluntary drains, not capacity reclaim.

Skipping the shutdown test is costly. A grace period set too close to the notice window leaves no time to flush. Retries without backoff can hammer a service when a large Spot pool goes away. Long transactions and large checkpoints create replay work. Test the restore path, not only the start path. If the bill is the larger issue, the Cloud cost audit from Cloudverg is a fixed price of $4,500 for a two-week read-only review that ranks savings, and it is one option among others.

Frequently asked questions