Designing a CI/CD pipeline developers will not route around
A pipeline succeeds when it is fast enough to trust and strict enough to matter. Getting that balance wrong is why teams disable checks.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- A committed secret is compromised, even after deletion — rotate it.
- Fetch secrets at runtime through workload identity where possible.
- Separate credentials per environment limit blast radius.
- Scan both new commits and existing history automatically.
Why this keeps happening
Secrets end up in repositories because it is the path of least resistance during development and nobody removes them afterwards. Git history means deleting the file later does not remove the secret.
Assume any credential that has ever been committed is compromised and rotate it, rather than assessing whether it was likely to have been found.
Use a purpose-built store
A secrets manager — the cloud provider's own, or a dedicated product — gives encrypted storage, access control, audit logging and rotation. Applications fetch secrets at runtime using workload identity rather than holding a static key.
Environment variables injected at deployment are acceptable for many cases and are considerably better than files in a repository, provided the deployment system itself stores them securely.
Rotation and least privilege
- Rotate on a schedule and immediately when someone with access leaves.
- Scope each credential to the narrowest permission set that works.
- Use separate credentials per environment so a development leak cannot touch production.
- Prefer short-lived tokens over long-lived keys wherever the platform supports it.
Detect automatically
Add secret scanning to the pipeline and to pre-commit hooks so a credential is caught before it is pushed. Most platforms provide this natively at no cost.
Scan the existing history as well. Repositories that have been running for years almost always contain something, and finding it deliberately is better than finding out another way.
Responding to an exposed credential
Exposure is common enough that the response should be rehearsed rather than improvised. Speed matters more than investigation, because the credential is already out.
- Rotate immediately — do not wait to assess whether it was used.
- Revoke the old value so it fails rather than lingering.
- Review logs for use of that credential, in parallel with rotation.
- Remove it from the repository history if it was committed, and treat any fork as compromised.
- Record what happened and what allowed it.
The last step is what prevents recurrence. Most exposures trace to a missing scanning step or an unclear process rather than to carelessness.
Frequently asked questions
They are a reasonable mechanism provided the values come from a secure store and are not printed in logs or exposed in error pages. The weakness is usually elsewhere in the chain.
Frequently enough that a leaked credential has limited useful life. Short-lived automatically issued tokens are better than any manual rotation schedule.
Never bake them in. Images get pushed to registries and shared. Inject at runtime instead.
