Why this matters
Secrets are the credentials that let code reach data and services: database passwords, API keys, signing keys, webhook tokens, TLS private keys and cloud access keys. They change between development, staging and production. A single reused secret across those environments turns a small leak in a test system into a production incident.
This guide, How to manage secrets across environments, treats the problem as a design issue, not a tool issue. The application should receive a secret at runtime from a trusted store, using an identity that proves which environment and workload it belongs to. Engineers should not copy production passwords into chat, tickets or local files for routine work.
The common failures are easy to spot. A .env file is committed. A CI variable is visible to every pipeline. A Kubernetes Secret is pasted as base64. A Docker image carries a key. A log line prints a connection string. The goal is one source of truth per environment, separate credentials, least privilege, an audit trail and a tested rotation path.
Before you start
Start with an inventory. List every secret your systems use: databases, caches, message queues, third-party APIs, webhooks, SMTP, JWT signing, certificates and cloud keys. For each one record the owner, the systems that read it, the environments that need it and how it is rotated. Mark which values are true secrets and which are ordinary configuration. That list becomes your migration checklist.
Decide where the source of truth will live. Options include a cloud secret manager such as AWS Secrets Manager, Azure Key Vault or Google Secret Manager, a platform such as HashiCorp Vault, or the secret store built into your deployment platform. Pick one primary store for each environment. Separate the production store from nonproduction stores by account, project or subscription when your provider allows it.
Define naming and access before you move a single value. Use a name like billing-prod-db-password or web-staging-stripe-key, with the application, environment and purpose in the name. Tag secrets with owner and rotation date. Decide how each workload will authenticate: an IAM role, a Kubernetes service account, a workload identity or an AppRole. Avoid shared user accounts and long-lived cloud keys.
Set the local development rule early. Developers need a way to run code without holding production secrets. Give each developer a sandbox identity, use mock services for third-party APIs, or provide a development-only secret namespace with fake values. Never give a laptop a production database password just to make local setup easy.
How to do it
Build the migration in small slices. Move one application and one environment at a time. Start with staging, prove the access path and the rollback, then repeat in production during a normal change window. Keep the old path working until the new one is tested, then remove the old credential so it cannot be used again.
The shape you want is simple. A workload proves its identity to the secret store, reads only its own secret path, and gets a short-lived credential or a value at startup. CI may read deploy-time secrets, but it should not hold runtime database passwords. Human access is the exception, not the default. Production reads should be logged and tied to a named person or role.
In Kubernetes, native Secrets are only a transport layer unless you enable encryption at rest and strict RBAC. External Secrets Operator, the Secrets Store CSI driver or Sealed Secrets can sync from a central store. On virtual machines, use the instance role and a small agent. In serverless, use the platform's secret integration. In all cases, keep secrets out of build artifacts, container images and Terraform state. If a tool must store a secret in state, lock down the state bucket and treat the state as sensitive.
- Inventory every secret and assign an owner. Create a table with the secret name, owner, systems, environments and rotation method. Include database passwords, API keys, signing keys and certificates. Mark which values are true secrets and which are plain configuration.
- Separate stores and names by environment. Use different secret stores or accounts for production and nonproduction. Name each secret with the application, environment and purpose, such as billing-prod-db-password. Tag it with an owner and rotation date.
- Remove secrets from code and build outputs. Move values out of .env files, Dockerfiles, Helm values and CI config. Scan the current tree and git history for old keys. Rebuild any image or artifact that may contain a secret.
- Give each workload its own identity. Create one role or service account per application and environment. Grant read access only to that application's secret paths. Avoid shared admin roles and long-lived cloud keys.
- Inject secrets at runtime, not build time. Have the workload fetch its secret from the store at startup, or mount it as a file from the platform. Use environment variables only as a transport from the store, not as the source of truth. Fail fast when a secret is missing.
- Rotate on a schedule and revoke old values. Set a rotation interval for each secret and automate it where the provider supports it. Use an overlap window so the old and new values both work during the change. Revoke the old value after the window.
- Turn on audit logs and a break-glass path. Log every read of a production secret and alert on human reads. Define a break-glass role for emergencies and review each use. Keep the logs long enough to support an incident review.
How to check it worked
Run a secret scanner over the current branch and the full git history. Gitleaks and TruffleHog are common choices. Scan CI config, Dockerfiles, Helm charts and infrastructure code too. Check your artifact registry for images built before the migration; a clean repository does not clean an old image. The result should be zero live credentials in code, history or build outputs.
Test access from the workload, not from your admin account. From a staging pod or instance, try to read the production secret path. It should fail with an authorization error. From the CI deploy role, read only the paths that deploy step needs. From a normal engineer account, confirm that production application secrets are not readable by default. Check that the audit log records the workload identity, the secret path and the time.
Run a rotation drill on one noncritical secret in staging and one in production. Rotate the value, confirm the old credential stops working after the overlap window, and confirm the application picks up the new value without a manual edit. Then test rollback. If the new value breaks the app, you should be able to restore the old one from the secret store and redeploy. A secret process that has never been rotated is only a theory.
Onboard a new engineer without sending a secret through chat. They should get access through a group or role, pull only the development values they need, and see the same naming and paths as everyone else. If onboarding requires a person to paste a production password, the design is not finished.
Mistakes to avoid
The most common mistake is using one secret in every environment. A staging API key that also works in production is a production key. It will end up in a test script, a screenshot or a support ticket. Create separate credentials even when the provider makes it easy to reuse one.
The second mistake is storing secrets in places that spread. CI variables are convenient, but a variable available to every pipeline is available to every branch. Kubernetes Secrets are base64, not encryption. Docker images keep old layers. Frontend code is public. Logs capture connection strings. Terraform state holds values. Treat each of these as a possible leak path and remove the secret from the path.
The third mistake is access without ownership. A shared admin role, a permanent cloud key or a password in a password manager folder with no owner will not be rotated. Assign an owner, a rotation date and a revocation step. Give people a break-glass path for real emergencies, then review every use of it.
If you need the surrounding platform as well as the secret store, Cloudverg's Delivery platform setup at $14,500 fixed price is one option among others. It covers pipelines, separate environments, secret management, least privilege deploy roles and monitoring. You can also build the same pieces in-house or combine a managed secret store with your existing deployment tools.
Frequently asked questions
Should I use environment variables or a secret manager?
Environment variables are simple, but they can leak through logs, process lists and crash dumps. A secret manager gives you access control, audit logs and rotation. Use environment variables only as a way to pass a value from the secret store into the process.
How many separate secrets do I need per environment?
Use separate secrets for every environment that can run code or hold data. At minimum that means development, staging and production. Give each developer sandbox its own values and never share a production secret with a lower environment.
Are Kubernetes Secrets encrypted?
By default Kubernetes Secrets are base64 encoded, not encrypted. They are stored in etcd unless you enable encryption at rest. Use strict RBAC, and consider External Secrets Operator or the Secrets Store CSI driver to pull from a central secret manager.