Why this matters
A secret in a config file or a container image is a copy nobody tracks. It gets cloned, pushed to a registry, pasted into a ticket, left in shell history. Google Cloud Secret Manager gives you one place to hold the value, versioned, with IAM in front of it and an audit trail behind it.
The data model is small. A secret is a named container. A version is the payload, and versions are immutable. Your app reads a version; nobody edits a value in place. That is what makes rotation safe: add version 2, roll the workload, disable version 1, keep a rollback path.
It does not do the whole job. It will not patch your app or rotate a vendor key for you, and it will not stop a process that already read the value from writing it into a log. Treat it as the store plus the access control, and keep the lifecycle steps in your own runbook.
How the app reads the secret matters too. An environment variable is read once at start and can surface in crash dumps or debug endpoints. A mounted file lets the app re-read after a rotation, which is the better default for anything long-lived.
Before you start
Enable the Secret Manager API in every project that runs a workload, not just the one holding the secret. If the API is off at runtime, client calls fail with an error that reads like a permissions problem and is not one.
Split environments by project. Secrets are project-scoped, so a grant in dev cannot read a production value unless you build that bridge yourself. If you run everything in one project, at least use one service account per workload and put the environment in the secret name.
Create a dedicated service account for each workload and stop using the default compute service account. The runtime account gets roles/secretmanager.secretAccessor on the individual secret. Humans and CI get roles/secretmanager.admin only where they actually create and rotate, and short-lived federated credentials beat downloaded keys.
Decide replication before you create anything: automatic lets Google pick regions, user-managed pins the payload to regions you list. Pick user-managed if you have residency commitments. If you add customer-managed keys through Cloud KMS, the Secret Manager service agent needs encrypt and decrypt on the key. And if you use Terraform, create the secret and the IAM binding there, but add versions from a CI step: a payload in state is a plaintext copy in a bucket.
How to do it
The flow is the same in the console, in gcloud, or in the client libraries: create the secret, add a version, grant on that secret, point the workload at it. Do the grant last so you never widen a role while you wait.
Watch the version alias. latest is convenient, but a new version changes what the next cold start reads with no review step. Pinning a numeric version and rolling forward on purpose is easier to explain during an incident. Record which version is live in your deploy notes.
- Enable the API where the workload runs. Run gcloud services enable secretmanager.googleapis.com in each project that will read a secret. Confirm with gcloud services list --enabled and look for the Secret Manager API entry.
- Name the secret for the app and the setting. Use a pattern like orders-api-db-password, lowercase with hyphens. Put the environment in the project, not the name, unless you share one project across environments. Anyone who can list secrets sees the names, so keep tenant and customer identifiers out.
- Create the secret with a replication policy. Run gcloud secrets create orders-api-db-password --replication-policy=automatic, or pass --locations=us-central1,europe-west1 for user-managed replication. Add --labels=app=orders,env=prod so you can filter and report later.
- Add the first version from standard input. Pipe the value in: printf '%s' "$DB_PASSWORD" | gcloud secrets versions add orders-api-db-password --data-file=-. Passing it as an argument puts it in your shell history and in the process list of the machine you ran it from.
- Grant access on the secret, not the project. Run gcloud secrets add-iam-policy-binding orders-api-db-password --member=serviceAccount:[email protected] --role=roles/secretmanager.secretAccessor. Read it back with gcloud secrets get-iam-policy and check that no group or domain is listed.
- Point Cloud Run at the secret. Add --set-secrets=DB_PASSWORD=orders-api-db-password:3 for an environment variable, or --set-secrets=/secrets/db-password=orders-api-db-password:3 to mount a file. The file form lets the app re-read the value without a rebuild.
- Point GKE at the secret. Install the Secret Manager add-on for GKE and write a SecretProviderClass that lists the secret and the Kubernetes secret to sync it into. The add-on uses Workload Identity, so bind the Kubernetes service account to a Google service account that holds secretAccessor.
- Read from the API in your own code. Call the access version method with Application Default Credentials at startup and cache the value in memory. Do not log the response object and do not write it to disk.
- Rotate in three moves. Add the new version, deploy or restart, check the logs for a clean start, then run gcloud secrets versions disable with the old version number. Destroy the old version only after your rollback window closes.
How to check it worked
Test as the workload identity, not your own: gcloud secrets versions access 3 --secret=orders-api-db-password --impersonate-service-account=orders-api@PROJECT.iam.gserviceaccount.com. If that returns the value, your app has the same path. If your own account works and the impersonated one fails, the binding is wrong or the API is off in that project.
Test the negative case with a secret the workload should not read. A permission denied is the result you want. Most teams check that access works and skip checking that it stays narrow.
Turn on data access audit logs for Secret Manager, then filter Logs Explorer on protoPayload.serviceName="secretmanager.googleapis.com". You should see an AccessSecretVersion entry for each read, with the principal and the version number. That is the record you want during an incident.
Last, check the artifacts. Grep the image, the Helm values file, the build log and the repo history for the value. Look at the mounted path inside the container and confirm which user can read it.
Mistakes to avoid
Granting secretAccessor at the project level. It reads tidy in a policy, and it means every service account in the project can read every secret in it. Bind at the secret level, and give people secretmanager.admin only where they create or rotate.
Using latest and forgetting it. A new version silently changes what the next cold start reads, so treat adding a version as a deploy with a rollback plan. The opposite failure is quieter: you pin a version, rotate at the vendor, and the app keeps running on a value you believe you retired.
Destroying a version mid-rotation. Destroy is permanent and takes the payload with it. Disable first, hold it for your rollback window, destroy later. Then restart the workload. A long-lived VM that read the value at boot keeps using it until something restarts the process.
Baking secrets in at build time. A value read during the build becomes part of the image and stays in the registry after the secret changes. If your build needs a credential, get a short-lived token through Workload Identity Federation instead. When you want an outside read on how your grants look across projects, the Cloud security review from Cloudverg is $6,500/month and covers a read-only pass over up to five cloud accounts, including over-broad and unused permissions. It is one option among others.
Frequently asked questions
Is Google Cloud Secret Manager the same as environment variables?
No. Secret Manager is a versioned store with IAM and audit logging. An environment variable is one way to hand a value to a process. You can load a secret into an environment variable at start, but that copy is unmanaged and lives as long as the process.
How do I rotate a secret without downtime?
Add a new version and deploy or restart so the workload reads it. If the vendor accepts two valid credentials at once, create the new one first, switch the app, then revoke the old one. Disable the old Secret Manager version after your rollback window and destroy it later.
What is the difference between disabling and destroying a version?
Disabling blocks access but keeps the payload, so you can re-enable it. Destroying deletes the payload permanently and cannot be undone. Disable during a rotation, then destroy once you are sure nothing needs the old value.