Secrets & encryption
Lesson 8.3 made the best argument about secrets: the safest secret is the one that doesn't exist (keyless auth). But you can't make every secret vanish — a database password, a third-party API token, a TLS private key have to live somewhere. This lesson is about doing that somewhere right: storing secrets in a purpose-built vault rather than in code, making them short-lived wherever possible, rotating them, encrypting everything at rest with managed keys, and — the rule that catches everyone eventually — never committing a secret to Git.
On-ramp in one breath: keep secrets in a dedicated vault, hand them to apps at runtime (not baked into code or images), prefer ones that expire and rotate, encrypt everything, and keep them out of Git forever.
The vocabulary, defined once
- Secret — any sensitive value an app needs but that must not be public: passwords, API tokens, private keys, certificates.
- Secret manager — a dedicated service that stores secrets encrypted, controls who can read each one (via IAM/RBAC), audits every access, and hands secrets to apps at runtime. Examples: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault.
- KMS — Key Management Service: a managed service that creates and guards encryption keys and performs encrypt/decrypt on your behalf, so the raw key material never sits in your code. Examples: AWS KMS, Cloud KMS, Azure Key Vault keys.
- Encryption at rest — data stored on disk is kept encrypted, so a stolen disk or snapshot is useless without the key.
- Rotation — replacing a secret with a fresh value on a schedule, so a leaked-but-unnoticed secret stops working.
Why a secret manager, not a config file
Chapter 4 warned that a Kubernetes Secret is only base64-encoded, not encrypted — encoding is not encryption. A plain config file or environment variable is worse. A secret manager fixes the whole category:
- Encrypted storage — the value is encrypted at rest (often by KMS), not sitting in plaintext.
- Access control — least-privilege IAM/RBAC decides which identity can read which secret (8.2). The bucket-reader role can't read the payment key.
- Audit trail — every read is logged: you can answer "who accessed this secret, when?"
- Rotation hooks — it can rotate a secret and update the consumers, often without downtime.
- Runtime delivery — apps fetch the secret when they run, so nothing sensitive is baked into the image or repo.
Dynamic (short-lived) secrets: the secret that expires
The most powerful idea, straight from 8.3's playbook applied to secrets: dynamic secrets. Instead of a static database password that lives forever, the secret manager generates a brand-new, short-lived database credential on demand — valid for, say, one hour — and revokes it when the lease ends. Trace it:
- The app asks Vault: "I need database access."
- Vault, which holds admin rights on the database, creates a fresh user/password valid for 1 hour, scoped to least privilege.
- The app uses it; after an hour, Vault revokes it automatically.
The payoff mirrors short-lived credentials exactly: there is no permanent database password to leak, and a stolen one self-destructs within the hour. Static secret → dynamic secret is the same upgrade as long-lived key → short-lived credential, applied to the secrets you must store.
:::tip The durable hierarchy of secret-handling
- Best: no secret at all — keyless auth / workload identity (8.3).
- Next best: a dynamic, short-lived secret minted on demand and auto-revoked.
- Acceptable: a static secret in a real secret manager, with least-privilege access, auditing, and scheduled rotation.
- Wrong: a secret in a config file, environment variable baked into an image, or — worst of all — committed to Git. Climb as high up this list as each secret allows. The principle is durable; the specific managers are dated. :::
Encryption at rest and KMS
Encryption at rest means data on disk is stored encrypted; without the key, a stolen disk or leaked snapshot is gibberish. The hard part isn't the encryption math — it's managing the keys. That's what KMS does: it creates keys, never lets the raw key material leave the service, performs encrypt/decrypt operations for authorized callers, controls who may use each key via IAM, and logs every use. Two durable habits:
- Turn on encryption at rest for everything — buckets, databases, disks, snapshots. On modern clouds it's often on by default; confirm it.
- Control the key policy. Who can use and who can administer a key is itself an IAM decision (8.2). A key anyone can use protects nothing.
A subtle but important point: encryption at rest defends against stolen storage. It does not protect a misconfigured-public bucket — the service decrypts transparently for any authorized request, and "public" made the request authorized. Encryption and access control are two different controls; you need both (this is exactly the 8.1 lesson — most leaks are access misconfiguration, not missing encryption).
Keeping secrets out of Git — and the GitOps wrinkle
The rule everyone learns the hard way: never commit a secret to Git. Git history is forever and widely cloned; a key pushed once is compromised even after you delete it, because it lives in the history and in every clone. Two defenses:
- Secret scanning in CI — tools like Gitleaks and TruffleHog scan commits/PRs for things that look like keys and block the merge (this is the shift-left idea of 8.6).
- For GitOps (Ch. 5): GitOps wants everything in Git — but you can't put plaintext secrets there. Two standard answers:
- SOPS — encrypts the secret values inside a YAML/JSON file (keys stay readable, values are ciphertext) so the file is safe to commit; it's decrypted at deploy time using a KMS key.
- External Secrets Operator (ESO) — you commit only a reference ("fetch
db-passwordfrom the secret manager"); ESO pulls the real value from Vault/Secrets Manager into the cluster at runtime. The repo never holds the secret, only a pointer.
Why it matters
Some secrets must exist — so store them right. A secret manager (Vault, cloud secret services) gives encrypted storage, least-privilege access, an audit trail, rotation, and runtime delivery — everything a config file lacks. Climb the hierarchy: prefer no secret (keyless), then dynamic short-lived secrets that auto-revoke, then static secrets rotated in a manager — and never a plaintext secret in code, an image, or Git. Encrypt everything at rest with KMS-managed keys, and remember encryption defends stolen storage, not a public-access misconfiguration. For GitOps, commit references (External Secrets Operator) or encrypted values (SOPS), and catch slips with secret scanning. Secrets are durable risk; this lifecycle is how you keep it contained.
Common pitfalls
- Base64 ≠ encryption. A raw Kubernetes Secret is just encoded. Enable encryption at rest and least-privilege access, or front it with a real manager (Ch. 4).
- Committing a secret "temporarily." Git history is forever and cloned everywhere — that secret is now compromised. Rotate it immediately and add secret scanning so it can't happen again.
- Static secrets that never rotate. A password set in 2019 and never changed is a standing liability. Rotate on a schedule, or go dynamic.
- Encrypting at rest but leaving the bucket public. Encryption doesn't stop an authorized request, and "public" authorized everyone. You need access control and encryption.
- A KMS key everyone can use. If any identity can decrypt with the key, the encryption protects nothing. Scope key usage with IAM.
Where this connects
- Back to Chapter 4 · Config & secrets — this completes the "use a real secret manager" promise made there.
- Back to 8.3 Workload identity & keyless auth — the best secret is no secret; dynamic secrets are the same short-lived idea for the ones you must keep.
- Forward to 8.6 Supply-chain & image security — secret scanning is part of shifting security left into the pipeline.
- Across to the Modern Security Engineer Guide — for cryptography fundamentals (how the encryption itself works).