What is Secret Rotation?

Definition

Secret rotation is the practice of regularly changing credentials (passwords, API keys, OAuth secrets, service account keys, certificates, encryption keys) to limit the blast radius of any leak or compromise.

Even with perfect secret management, secrets can leak — via vendor breach, insider, code commit, log, or supply-chain attack. Rotation ensures that leaked secrets become useless quickly.

In simple terms:

Rotation is the assumption that secrets will leak — and the discipline of making leaks harmless quickly.

---

Why It Matters

• Most secrets are long-lived; one leak = years of access.
• Vendor breaches expose your secrets without your fault.
• Insider risk: departing employees may take secrets.
• Supply-chain compromises (npm, GitHub Actions, etc.) leak secrets indirectly.
• Compliance frameworks require rotation cadences.

---

Rotation Patterns

1. Scheduled Rotation

Rotate every X days regardless of incident.

• Cadence — by sensitivity (e.g., 30 days for prod DB, 90 for vendor API, 365 for TLS).
• Automation — scheduler triggers rotation flow.

2. On-Demand Rotation

Manual or event-triggered.

• Triggers — incident suspicion, vendor breach disclosure, leaver event.

3. Dual-Secret (Active + Standby)

Two valid secrets at any time; rotate one while other serves traffic; swap.

• Zero downtime.
• Common for OAuth client secrets, API keys.

4. Leaver-Triggered

Owner leaves; NHIs they touched rotate.

5. Compromise-Triggered

Incident detected → immediate rotation.

6. Continuous Rotation

Secret refreshes on each use (e.g., short-lived OAuth tokens, federation).

---

What to Rotate (and Cadences)

Secret Type	Recommended Cadence
Production database password	30–90 days
Service account password	30–90 days
OAuth client secret	90–180 days
Vendor API key	90 days
Personal Access Token (PAT)	30–90 days
Webhook signing secret	180 days
TLS certificate	≤ 1 year (90-day modern best practice)
SSH key	≤ 1 year (use cert authority instead)
JWT signing key	90–180 days
Encryption KEK	1–2 years (data re-encryption considered)
Code-signing key	1–2 years (HSM-backed)
Krbtgt	Twice annually (AD)

After incident: immediately for affected secrets.

---

How Rotation Works

Step 1: Generate

• New secret created (high-entropy, by vault).

Step 2: Distribute

• New secret pushed to consumers (via vault references, deploys, runtime fetch).

Step 3: Validate

• Confirm new secret works.

Step 4: Deactivate Old

• Old secret marked inactive (or removed).

Step 5: Monitor

• Watch for failures from consumers using old secret.

Step 6: Audit

• Log rotation event.

For Dual-Secret

• Both secrets active during transition window.
• Old secret removed after window.

For Cert Rotation

• New cert issued (CA-signed).
• Distributed to workloads.
• Old cert remains until expiry.

---

Common Rotation Failures

1. Manual, Forgotten

Calendar-based; missed; secrets persist for years.

2. Hardcoded Secrets

Rotation requires code change; high friction; skipped.

3. Multiple Consumers

One consumer not updated; rotation breaks production.

4. No Validation Step

Rotation deploys broken secret; outage.

5. Insufficient Window for Dual-Secret

Old secret removed too quickly; in-flight requests fail.

6. Audit Gaps

No record of when rotated.

7. Rotation Without Inventory

Don't know what to rotate.

---

Real-World Examples

1. Vendor Breach Triggers Mass Rotation

Vendor disclosed breach affecting customer API keys. Organization rotated all keys for that vendor within hours via automation; no compromise observed downstream. Mitigation: rotation playbook + automation.

2. PAT Sprawl

Found 8K active PATs; many > 1 year old. Implemented 90-day expiry policy + secret manager; reduced active PATs to ~2K with rotation.

3. Leaker SQL Server Password

Production SQL Server password unchanged for 7 years; in many scripts. Coordinated rotation broke 4 forgotten dependencies; restored via dual-secret pattern next iteration.

4. Cert Auto-Renewal

Cert-manager + Let's Encrypt auto-renew certs every 60 days. Eliminated cert-related outages.

5. AWS Access Key Rotation

Periodic IAM access key rotation revealed orphan IAM users (rotated keys never used) → decommissioned.

---

Best Practices

1. Automate rotation wherever possible.
2. Inventory all secrets first; can't rotate what you can't see.
3. Centralize in vault — vault manages rotation.
4. Reference (not embed) secrets in config / code.
5. Dual-secret pattern for zero-downtime rotation.
6. Cadence by sensitivity — risk-tiered.
7. Validate before deactivating old.
8. Audit every rotation.
9. Incident playbook for emergency rotation.
10. Eliminate rotation by federating (workload identity → no static secret).
11. Cert lifecycle automation (cert-manager, ACME).
12. HSM for highest-value signing keys.

---

Checklist

• Inventory of all secrets?
• Centralized vault?
• Automated rotation?
• Cadence by sensitivity?
• Dual-secret for zero downtime?
• Validation step?
• Audit per rotation?
• Incident rotation playbook?
• Federation replacing static secrets?
• Cert auto-renewal?
• HSM for high-value keys?
• Stale-secret KPI tracked?

---

How Forestall Helps

Forestall identifies stale secrets across cloud / code / SaaS, tracks rotation cadence, and alerts on missed rotations.

---

Frequently Asked Questions

Is rotation necessary if secrets are in a vault?

Yes — leakage can still occur (vendor breach, insider, supply chain). Rotation is defense in depth.

Can rotation cause outages?

If done poorly, yes. Use dual-secret pattern + validation.

How often is too often?

Depends on operational maturity. Daily rotation works for some (Netflix). Quarterly works for many. Risk-tier the cadence.

What about workload identity federation?

Best — no static secret to rotate. Use it where possible.

What's the ROI on rotation automation?

High — eliminates manual toil, reduces incident impact, improves compliance posture.

---

Conclusion

Secret rotation is identity security's defense in depth — assume secrets will leak, and ensure leaks are short-lived. Inventory, centralize, automate, dual-secret for zero-downtime, audit, and federate where possible. With disciplined rotation, leaked secrets become a contained event rather than a catastrophic breach.