Use spot capacity on Oracle Cloud without breaking things

Why this matters

Oracle Cloud Infrastructure calls spot capacity preemptible capacity. You choose it as a capacity type when you launch a compute instance, and OCI can take the instance back when it needs the room. A preemptible instance is a bad home for a database, a single control plane node, or any process that writes to local disk and cannot replay that work.

The useful pattern is a pool of identical workers that pull short jobs from a queue. If one worker disappears, another picks up the message. That fits batch transforms, CI runners, media jobs, log crunching, and stateless API workers behind a load balancer. It does not fit a workload that needs a stable host name or a license tied to a machine.

Most failures on preemptible capacity come from work that is too large to retry, state left on the boot volume, or a team that never tested the loss of a worker. Treat every preemptible instance as disposable, and put durability somewhere else.

Before you start

Write down which workloads can lose an instance without a customer noticing. Tag them with workload, owner, environment, and capacity=preemptible. Keep databases, brokers that store data on local disk, and anything with a manual failover step on on-demand or reserved capacity.

Build an image that starts with no manual steps. Put the app in a custom image or a cloud-init script, pull secrets from OCI Vault, pull config from Object Storage, and send logs to OCI Logging. Do not bake secrets into the image. If the instance needs a block volume, the data must be read only or rebuildable.

Pick the queue before the shape. OCI Queue gives you visibility timeouts and delete on success. A database table with a lease can work, but you must handle lock expiry and duplicate delivery. Keep each job short enough to finish inside the timeout and short enough that a reclaim is cheap. Do not design a single run that depends on staying up for days.

Decide the fallback now. Keep a small on-demand floor, and know what automation does when a preemptible launch fails with Out of host capacity. Trying another availability domain can help, but bound the retries. Tag the fallback capacity too so the bill is readable.

How to do it

Launch the first worker by hand before you automate. In the OCI Console, go to Compute, Instances, Create instance. Open Show advanced options, set Capacity type to Preemptible, and choose Preemption action. Stop keeps the boot volume for a later start; Terminate is simpler for a stateless worker. Check the image, VCN, and fault domain, then create the instance.

The drain handler decides whether the design holds. Poll the instance metadata service for the preemption notice. Then stop taking new work, finish or return the current item, flush logs, and exit. Keep the handler short. If the notice gives little time, the queue lease must expire so another worker can retry.

When you use instance pools or autoscaling, set the capacity type in the instance configuration and cap the maximum size. A pool that keeps replacing workers during a shortage will churn. Scale on queue depth and job age, not CPU alone. Keep the on-demand floor outside the preemptible pool.

1. Tag the workload as interruptible. Add tags for workload, owner, environment, and capacity=preemptible. Put the same tags on instances, volumes, and load balancer backends so Cost Analysis and cleanup can find them.
2. Make the instance start clean. Bake the app into a custom image or cloud-init, and pull secrets from OCI Vault at boot. Send logs to OCI Logging and write artifacts to Object Storage, not the boot volume.
3. Put work on a queue with leases. Use OCI Queue or a database table with a lease and a completion record. Make each job idempotent, and set the visibility timeout longer than a normal run but shorter than your patience for a stuck message.
4. Launch with preemptible capacity. Set Capacity type to Preemptible in the instance or instance configuration. Choose Stop or Terminate for Preemption action, and write down which one you chose, because cleanup and restart behavior differ.
5. Add the drain handler. Poll the instance metadata service for the preemption notice. On notice, stop claiming jobs, return the current message if it is not done, flush logs, and exit with a clear log line.
6. Keep an on-demand floor. Run a minimum number of on-demand workers for the same queue. Let preemptible workers add burst above that floor, and alarm when only the floor is left.
7. Test reclaim and capacity failure. Stop or terminate a worker while the queue is busy, and confirm the job returns and completes elsewhere. Then force a launch failure with a shape or availability domain that has no capacity, and check the fallback path.

How to check it worked

After a test reclaim, look at the queue first. In flight messages should return after the visibility timeout, retries should stay bounded, and the oldest job age should come back down. If messages sit in flight until you delete them by hand, the drain handler or lease handling is wrong.

Search OCI Logging for the preemption notice and your drain log line, with the instance OCID. You want a record that the worker saw the notice, stopped claiming work, and exited. If the instance disappears with no log line, the handler is not running or logs are not leaving the instance.

Set OCI Monitoring alarms on worker count, queue depth, job age, and failed jobs. Add an alarm for preemptible instances stopped for more than a few hours, and for unattached boot volumes. A stopped instance still leaves storage on the bill.

Use Cost Analysis with your tags to compare preemptible and on-demand spend for the same workload. Check that fallback appears when capacity is short, not every night. If the preemptible line is small but storage and network grew, the design is not saving what you expected.

Mistakes to avoid

The first mistake is putting a stateful service on preemptible capacity because a test ran fine for a week. A database, a single broker, or a control plane can look healthy until the reclaim. Keep those on on-demand or reserved capacity.

The second is a job that takes too long to checkpoint. If a worker can lose an hour of work, you will feel every reclaim. Split the job, write checkpoints to Object Storage, and make retry the normal path.

The third is assuming preemptible capacity is always there. Launch attempts can fail with Out of host capacity, and a retry loop can leave orphan instances and volumes. Use bounded retries, try another availability domain, and fall back to on-demand for the floor.

The fourth is skipping cleanup and cost review. Stopped preemptible instances, old boot volumes, and idle load balancers can eat the discount. If the bill is the reason you are trying preemptible capacity, the Cloud cost audit from Cloudverg is a fixed price $4,500 two-week read-only review of up to three cloud accounts. It ranks rightsizing, idle resource, storage tier, and commitment findings. It is one option among others; you can also start with tags and Cost Analysis yourself.

Frequently asked questions