The previous post in this little series — Backup Verification in 2026: Exit Code 0 Is Not a Backup — grew out of a comment thread under a dev.to article by dafili, "My Backup Exited 0 for Three Weeks and Nothing Paged Me". The conversation did not stop at publication. Dafili corrected one claim in that post (the correction is in, and credited), and then pushed the whole idea one level deeper — to a place the four-level ladder does not reach on its own.

His point, in his words: "You cannot automate the declaration, only the comparison."

This post is about what that sentence means, because once you hear it, a hole opens up in the middle of most restore drills — including the tidy ones from the ladder post. The ladder tells you how to check a backup. It never tells you what "correct" means. And if the definition of correct comes from the same system you are checking, your drill can pass beautifully while verifying nothing at all.

TL;DR: A restore drill is only as honest as the expectations it compares against. If those expectations are derived from the backup or the system being verified — row counts read from the dump itself, file lists generated by the backup tool — the check is circular: it catches corruption, never wrongness. The fix is a fingerprint: a short, hand-written declaration of what a correct restore must contain (an expected file list, one file's sha256, a row count, one config value). A human re-declares it quarterly; automation compares against it nightly and shouts on drift. And the drill itself gets monitored with a days_since_last_successful_manual_restore gauge that pages at about 120 days — because an unmonitored drill quietly stops happening.

Originally published on DevOps Enginer.

The ladder checks the climb, not the destination

The four levels from the previous post — the artifact exists, it is intact, it can be read, it has actually been restored — still stand. Level 4 is where trust lives: restore into a scratch environment and verify the result like a user would.

But look closely at how a typical level-4 drill "verifies." It restores the dump, then runs queries against the restored database: count the rows in users, check that orders has recent entries, confirm the application boots. Where do the expected values for those checks come from? Usually from the restored data itself, or from the same monitoring that watched the original system. The drill asks the backup: do you agree with yourself? The backup always says yes.

That is how a drill passes on data that is perfectly intact and completely wrong. Dafili made this distinction in the same thread: an integrity check catches media or repository corruption, but only a restore catches data that is wrong — and even then, only if the restore is compared against something that did not come from the restore. An empty mount backed up faithfully, checksummed faithfully, restored faithfully. A dump of the wrong database, in perfect condition. Every level green. Nothing recovered that you actually wanted.

He explored the same trap from another angle in a follow-up article, "Three checks that look like they passed but validated nothing": checks that execute correctly, return success, and prove nothing, because they validate a different property than the one you care about. A restore drill with self-derived expectations is exactly that shape of check — bigger, more expensive, and just as capable of validating nothing.

The fingerprint: four things a human writes down

Dafili's answer is a fingerprint — a small file a human writes by hand, declaring what a correct restore must contain. Not generated. Not exported from the database. Declared, the way you would declare what matters before a trip: passport, keys, the one document you cannot replace. His fingerprint has four components:

• An expected file list. The files (or tables, or object prefixes) that must be present in the restore. Not all of them — the ones whose absence would mean the backup missed its point.
• One file's sha256. Pick one file whose content you can predict — a config file, a seed data file, a known document — and record its hash. The restore must reproduce it bit for bit.
• A row count. One table that matters, with a count (or a sane range) a human states from knowing the business, not from querying the backup. Orders should be in the hundreds of thousands by now; if the restore says 41, something is deeply wrong even if the restore "worked."
• One config value. A single setting whose correctness you can declare: the application's environment name, a feature flag that must be on, the schema version the code expects. Config drift is how "successful" restores fail at boot time.

Four declarations. Small enough to write in fifteen minutes, small enough to read at a glance, and — this is the whole design — none of them can be produced by the system they verify. The fingerprint stands outside the backup pipeline and judges it.

An example, as a plain file kept with the runbook:

# restore-fingerprint.yaml — declared by hand, re-declared quarterly
system: billing-db
expected_files:
  - /srv/app/config/app.env
  - /srv/app/data/seed-countries.csv
sha256:
  /srv/app/config/app.env: "9f2c…e41a"   # recorded at declaration time
row_counts:
  orders: ">= 400000"                      # stated from the business, not the dump
config_values:
  schema_version: "2026.09"                # what the current code expects
declared: 2026-11-02
next_review: 2027-02-02

The exact format does not matter. What matters is the direction of the arrow: expectations flow from a human's knowledge into the check, never from the checked system outward.

You cannot automate the declaration

Why insist a human writes it? Because the declaration is not data entry — it is a review disguised as data entry. To write the fingerprint, someone has to sit down and answer questions that no monitoring dashboard asks: What in this system would we actually miss? Which table is the business? Which config value, if wrong, turns a good restore into a bad morning? Those answers live in people's heads and in the incident history, not in the database.

In dafili's practice, the human re-declares the fingerprint quarterly — deliberately on the same cadence as the full restore drill. Systems drift legitimately over a quarter: new tables appear, row counts grow, config values change on purpose. The quarterly re-declaration is the moment a human looks at that drift on purpose, in calm conditions, and says: yes, this is what correct looks like now. Skip it, and the fingerprint slowly becomes a declaration about a system that no longer exists — which is its own quiet failure mode, and exactly why the declaration cannot be a one-time setup task.

This is also why automating the declaration defeats the purpose. A generator that rebuilds the fingerprint from production every night produces a fingerprint that is always exactly right and therefore says nothing: it re-declares whatever the system currently is, including its mistakes. The value of the fingerprint is precisely that it can disagree with the system — and that a human chose the terms of the disagreement in advance.

The comparison is the automation's job

Everything after the declaration should be boring, mechanical, and nightly. Restore the latest backup into a scratch environment, extract the actuals, and diff them against the fingerprint:

#!/usr/bin/env bash
# compare-restore.sh — runs after the nightly restore-to-scratch
set -euo pipefail

FAIL=0

# 1. Expected files present?
while read -r f; do
  [ -f "/tmp/restore-test$f" ] || { echo "MISSING: $f"; FAIL=1; }
done < <(yq -r '.expected_files[]' restore-fingerprint.yaml)

# 2. The declared hash matches?
sha256sum -c <<<"$(yq -r '.sha256."/srv/app/config/app.env"' restore-fingerprint.yaml)  /tmp/restore-test/srv/app/config/app.env" \
  || { echo "HASH DRIFT: app.env"; FAIL=1; }

# 3. The declared row count is in range?
ORDERS=$(psql restore_test -tAc "SELECT count(*) FROM orders;")
[ "$ORDERS" -ge 400000 ] || { echo "ROW COUNT DRIFT: orders=$ORDERS"; FAIL=1; }

# 4. The declared config value survived?
psql restore_test -tAc "SELECT value FROM settings WHERE key='schema_version';" \
  | grep -qx "2026.09" || { echo "CONFIG DRIFT: schema_version"; FAIL=1; }

exit $FAIL

The details will differ with your stack — the pattern is the point. Automation compares; it does not get an opinion. When the comparison fails, it shouts: a failed fingerprint comparison is a page, not a log line, because it means one of two things, both important — the backup is wrong, or the world changed and nobody re-declared it.

Drift is a conversation, not just an alarm

That second case deserves respect rather than dread. Most fingerprint failures, in a healthy team, will be legitimate change: a migration added a table, the row count outgrew the declared range, a config value rolled forward. The correct response is not to silence the check but to re-declare deliberately — a human reviews the drift, confirms it is intended, and updates the fingerprint as its own small, visible act.

Contrast that with the alternative everyone actually lives with: expectations embedded in the check script, edited quietly at 2 a.m. to make the red go away, until the check asserts nothing. The fingerprint does not prevent someone from weakening a check. It makes the weakening a declared event, in a file with a date on it, that a second person can review. That is the difference between a system that learns and a system that merely quiets down.

Monitor the drill itself

There is one more layer, and it might be the most practical idea in the whole thread. Automated nightly comparisons are excellent — right up until the human parts of the practice quietly die. The quarterly manual restore is the first thing skipped in a busy quarter. Then it is skipped twice. Nothing alarms, because the automation is still green; the drill's judgment — a person restoring by hand, noticing the weird thing no fingerprint anticipated — has been absent for a year and nobody knows.

Dafili's countermeasure is to monitor the drill itself with a gauge: days_since_last_successful_manual_restore, and to page when it crosses about 120 days. Not a ticket. A page — the same severity as a failed backup, because that is what it is: past roughly one quarter, you no longer have evidence that a human can restore this system, only evidence that a script once could.

In Prometheus terms it is a timestamp gauge the drill updates when — and only when — a manual restore passes its fingerprint comparison:

# Written by the drill checklist's final step, after a human-run
# restore passes the fingerprint comparison:
echo "last_successful_manual_restore_timestamp $(date +%s)" \
  > /var/lib/node_exporter/textfile/restore_drill.prom

# Page when no successful manual restore in ~120 days
(time() - last_successful_manual_restore_timestamp) / 86400 > 120

Readers of the ladder post will recognise the shape: it is the dead-man's-switch from that article, pointed at the humans. There, the monitored absence was verification success — page when the checks stop succeeding. Here, the monitored absence is the drill itself — page when the people stop practising. Automation watches the backups; the gauge watches the watchers' habits. A restore you have not performed in four months is a hypothesis, and the gauge exists to say so out loud while it is still cheap to fix.

Where does the fingerprint live?

One honest open question, because it came up in the thread and I do not have dafili's answer yet: where should the fingerprint be stored? I asked him directly in the conversation; while his reply is pending, the trade-offs are worth stating plainly, because the obvious answer has a thorn in it.

• With the backup configuration is convenient — the drill finds it without ceremony. But it risks the fingerprint drifting with the thing it verifies: the same change that breaks the backup can quietly rewrite the expectations, the way a scale that recalibrates itself to whatever it weighs can never report a wrong weight.
• In a separate repository, owned like a runbook, keeps the declaration independent and reviewable — changes arrive as commits a second person can see. The cost is one more place to keep current, and a drill that must fetch its expectations from somewhere it trusts.
• On paper, or in the incident binder, sounds like a joke until you remember what the fingerprint is for: it is the declaration that survives the system. Four values fit on an index card.

Wherever it lives, the property to protect is independence: the fingerprint must not be writable by the pipeline it judges. When dafili answers, this section gets an update — his reasoning has improved this series twice already.

Restore-drill fingerprint checklist

• A fingerprint file exists for each system that matters, written by hand — not generated from the system it verifies
• It declares all four: an expected file list, one file's sha256, one row count (or range), one config value
• A human re-declares it quarterly, on the same cadence as the full restore drill
• The nightly drill restores to scratch and compares actuals against the fingerprint automatically
• A failed comparison pages — and legitimate drift is resolved by deliberate re-declaration, never by editing the check to be quiet
• A days_since_last_successful_manual_restore gauge (or equivalent) exists, with a page at about 120 days
• The fingerprint is stored somewhere the backup pipeline itself cannot rewrite
• Expectations in older checks that were derived from the checked system have been replaced with declared ones

FAQ

Isn't this just the level-4 verification from the previous post? It is level 4 done honestly. The ladder says restore and verify; the fingerprint answers the question that post left open — verify against what? If the answer is "against values taken from the restore itself," the verification is circular. Declared expectations are what break the circle.

Four checks seem too few. Shouldn't the fingerprint cover everything? Coverage is the automation's job — checksums, listings, and sampled reads already sweep the whole artifact nightly. The fingerprint has a different job: to be an independent witness. A witness who testifies about everything testifies about nothing; four declarations a human can hold in their head, and re-declare with full attention each quarter, beat four hundred generated ones nobody reads.

What if the business can't state a row count in advance? Declare a range or a floor, stated from business knowledge ("orders grow by roughly ten thousand a month; a restore below last quarter's count is wrong"). Precision is not the point — independence is. An expectation that is roughly right and genuinely external catches the catastrophic wrongness the drill exists to catch.

Does the gauge replace scheduling the quarterly drill? No — it catches the schedule failing. Calendars get ignored under load; that is precisely the failure the gauge exists to surface. Schedule the drill and monitor its successful completion, the same way you schedule backups and monitor verification success.

Summary

The ladder from the previous post — exists, intact, readable, restored — tells you how to check a backup. Dafili's fingerprint tells you what makes the checking mean something: expectations a human declared, standing outside the system, that the system cannot rewrite to flatter itself. You cannot automate the declaration, only the comparison. So declare by hand, re-declare quarterly, compare nightly, shout on drift — and keep a gauge on the drill itself, paging at about 120 days, so the practice cannot die in silence.

Credit where it is due: the fingerprint doctrine, the declaration/comparison line, and the drill gauge all come from dafili, in the comment thread under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" — the same thread that produced the ladder post and corrected it. His follow-up, "Three checks that look like they passed but validated nothing", is the natural next read. This series keeps getting better because a reader keeps arguing with it in public. Long may that continue.

A story made the rounds in a DevOps community recently, and it should make anyone who owns a cron job uncomfortable. An engineer checked on a backup script that had been running perfectly by every visible measure — green in the scheduler, exit code 0, night after night — and found it had been "succeeding" at backing up nothing at all for three weeks, ever since something upstream had quietly changed. No error. No page. Just three weeks of confidence with nothing underneath it.

TL;DR: A backup job's exit code tells you the script ran, not that you have a backup. A backup is only trustworthy at four levels: (1) the artifact exists, (2) it is structurally intact, (3) it can be opened and read, and (4) it has actually been restored and the data verified. Most teams verify level 1 and stop. This guide walks a practical verification ladder — checksums, restic check, pg_restore --list, scheduled restore drills — plus the monitoring change that matters most: page on verification failure, not job failure.

Originally published on DevOps Enginer.

This is deliberately not a Kubernetes backup guide — cluster tooling like Velero and etcd snapshots is covered in the Kubernetes Backup and Disaster Recovery post. This is about the layer underneath every stack: database dumps, file archives, and object-storage backups, and how to prove they work before the day you need them.

Why exit code 0 lies

Exit code 0 means one narrow thing: the last command in your script did not report an error. It says nothing about what the script actually captured. Backup scripts fail silently in a handful of classic ways:

• Wrong or empty source paths. A directory gets moved, a mount fails to come up, or a path is renamed upstream. The archiver dutifully compresses an empty directory and exits 0. Your backup is a perfectly valid archive of nothing.
• Expired or rotated credentials. The dump tool fails to authenticate and writes a 0-byte file — or the wrapper script swallows the error, logs a warning nobody reads, and still exits successfully because the upload of that empty file worked fine.
• Silent partial captures. A database dump taken without the right flags can complete while missing schemas, extensions, or large objects. The file is large, recent, and wrong.

The pattern underneath all of these: the job was monitored, but the backup was never verified. Those are different jobs, and they need different checks.

The four levels of backup confidence

A useful way to think about backup verification is as a ladder. Each level proves something the level below cannot:

Level 1 — The artifact exists. A file or snapshot with the expected name appeared in the expected place, recently, with a plausible size. This is what most monitoring checks. It proves a process ran.

Level 2 — The artifact is structurally intact. The archive is not truncated or bit-rotted. Checksums match; the backup tool's own integrity check confirms its internal structures — indexes, metadata, pack files — are consistent. This proves the bits survived storage.

Level 3 — The artifact can be opened and read. The restore tool can actually parse the backup: list the table of contents of a dump, enumerate the files in an archive, read back a sample of the stored data. This proves the format is valid and the content is what you think it is — not an empty tree, not the wrong source.

Level 4 — The artifact has been restored and the data verified. You restored it into a real (throwaway) environment and checked the result: the database answers queries, row counts and key tables look right, application files open. This is the only level that proves you can recover. Everything below it is inference.

Notice what the ladder implies: levels 1–3 are cheap and automatable on every backup. Level 4 is more expensive, so you schedule it — sampled nightly or weekly, full restores on a rotation. What you cannot do is pretend level 1 is level 4. That gap is exactly where the three-weeks-of-nothing story lives.

The verification ladder in practice

Here is how to test yours, level by level. The examples use common tools — restic, PostgreSQL dumps, plain tarballs in object storage — but the pattern transfers to whatever you run.

Level 1: existence, freshness, and size sanity

Check that the newest artifact exists, is younger than your backup interval (plus slack), and is within a sane size band of its predecessors. A backup that suddenly shrinks by 90% is a failed backup wearing a success costume.

# Fail loudly if no backup newer than 26 hours exists in the bucket
aws s3 ls s3://my-backups/postgres/ --recursive \
  | sort | tail -n 1

# Size sanity: compare today's artifact against the 7-day median
# and alert if it deviates by more than ~50%

Tools like restic make this easier because the repository keeps snapshot metadata:

restic snapshots --last 1 --json   # newest snapshot time, paths, tags
restic stats                       # repository size trend

Encode the rule: a missing, stale, or suddenly-tiny backup is a verification failure, even if the job that produced it exited 0.

Level 2: structural integrity with checksums and tool-native checks

If you produce plain archives, generate a checksum at backup time and verify it at check time:

sha256sum -c postgres-2026-11-01.dump.sha256
gzip -t nightly-files.tar.gz   # quick structural test for gzip streams

restic has integrity checking built in:

restic check

restic check verifies the repository structure — all packs, indexes, and metadata present and consistent. It is fast and belongs in the nightly job. What it does not do by default is re-read your backup data — that is level 3.

Level 3: prove it can be read back

For archives, listing the contents is the minimum viable read test — it forces the tool to walk the whole structure:

tar -tzf nightly-files.tar.gz > /dev/null   # fails on truncation/corruption

For PostgreSQL custom-format dumps, pg_restore --list parses the dump's table of contents without restoring anything:

pg_restore --list postgres-2026-11-01.dump | head -n 50

Scan that listing for what should be there: your critical tables, extensions, and the expected object count. An empty dump has a valid header and almost nothing under it — this is the check that catches the "backed up nothing" failure mode.

For restic, the read-back check is:

restic check --read-data-subset=5%

Understand the caveat before you rely on it: --read-data-subset=5% reads and verifies 5% of the data packs, not 5% of each file. It is sampling. A corrupt pack outside the sample will not be found tonight. Two practices make sampling honest:

• Rotate with the n/t form, not the percentage. --read-data-subset=5% picks a random subset on every run, so runs can overlap and some packs may never get sampled at all. The n/t form is the deterministic one: restic splits the packs into t groups and checks group n. Pin it to the calendar — --read-data-subset=$(date +%u)/7 checks a different seventh each night and guarantees every pack is read within a week, for the same total I/O as one full check spread out.
• Run a full read periodically. restic check --read-data (100%) on a monthly cadence closes any remaining gaps.

Also add a content spot-check at this level: restore a handful of known files to a temp directory and diff them, or query the snapshot for a file you know changed yesterday. Existence of expected content is a much stronger signal than existence of an artifact.

Level 4: restore drills — the only level that counts

Restore into a throwaway environment and verify the result like a user would. For PostgreSQL:

# Restore last night's dump into a scratch database
createdb -T template0 restore_test
pg_restore --no-owner --dbname=restore_test postgres-2026-11-01.dump

# Then verify like the application would
psql restore_test -c "SELECT count(*) FROM users;"
psql restore_test -c "SELECT max(created_at) FROM orders;"

Row counts in a plausible range and a recent max(created_at) on your busiest table prove the restore contains live, current data — not a skeleton schema and not last month's state.

For file backups, restore the latest snapshot to a scratch path and verify file counts and a sample of checksums against expectations:

restic restore latest --target /tmp/restore-test
find /tmp/restore-test -type f | wc -l

The drill is not done when the restore command exits 0. The drill is done when a check of the restored data passes. And time the drill: the duration you measure is your real recovery time, not the one in the runbook.

Make verification a separate job that can fail

The single most important structural change: verification must be its own job, with its own failure state, that runs after the backup job. Folded into the backup script, a verification failure disappears into the same green checkmark. Give it its own exit code and its own alert path.

#!/usr/bin/env bash
# verify-backups.sh — runs after the backup job, fails independently
set -euo pipefail

LATEST=$(restic snapshots --last 1 --json | jq -r '.[0].id')

restic check                                   # level 2
restic check --read-data-subset=5%             # level 3 (sampled)
restic restore "$LATEST" --target /tmp/restore-test --include /srv/app/config
test -s /tmp/restore-test/srv/app/config/app.env   # expected content present

echo "backup_verification_success $(date +%s)" \
  > /var/lib/node_exporter/textfile/backup_verification.prom

The script writes a success timestamp only if every level passed — and that timestamp is what you monitor.

Monitor the verification, not the job

Alerting on "backup job failed" catches crashes. It does not catch the failure mode this article is about, because in that failure mode the job succeeds. Change what you page on:

• Page on verification failure. Any level failing — stale artifact, size anomaly, checksum mismatch, unreadable dump, failed drill — is a page-worthy event, because it means you currently do not have a proven backup.
• Page on missing success. Track "time since last successful verification" and alert when it exceeds your threshold (for a nightly cycle, ~30 hours). This dead-man's-switch pattern catches the silent cases: the verification job itself dying, the scheduler stopping, credentials expiring before the check even runs. Absence of failure is not presence of success — so monitor the presence of success.
• Track your verified restore point. The metric that matters in an incident is "how old is the newest backup that has passed level 4?" If that number quietly grows to three weeks, you want to know before the incident, not during it.

If you use Prometheus, the textfile pattern above plus one alert — time() - backup_verification_success > 30 * 3600 — covers most of this. The tooling is not the hard part; deciding that verification failure pages like a production outage is.

A restore-drill cadence you can actually keep

Verification fails as a practice when it depends on someone remembering. Put it on a calendar:

The rotation matters: every system should get a full restore at least once a quarter, rather than an annual hero-drill on the one system everybody already understands. Record each drill's duration — that number is your measured RTO, and it is almost always worse than the assumed one. Finding that out in a drill is the entire point.

Cloud snapshots deserve a specific warning here: a snapshot of a disk or database is level 1 by default. Snapshots faithfully preserve corruption, they can be silently incomplete if taken mid-write without coordination, and "we have snapshots" has ended more than one incident review badly. Snapshots need the same ladder — restore one into an isolated instance, open it, query it — before they count as backups.

Backup verification checklist

• Every backup has a separate verification job with its own pass/fail, running after the backup completes
• Level 1: freshness and size-band checks on every artifact, every cycle
• Level 2: checksums verified for plain archives; restic check (or your tool's equivalent) nightly
• Level 3: read-back tests — pg_restore --list, tar -t, restic check --read-data-subset rotated with the n/t form, plus expected-content spot checks
• Level 4: automated partial restores weekly; full restores monthly on rotation; quarterly timed drill
• Monitoring pages on verification failure and on missing verification success (~30h for nightly cycles)
• "Age of newest level-4-verified backup" is a visible metric, not a surprise
• Restore durations are recorded, so your RTO is measured rather than assumed
• Snapshots are restore-tested like any other backup before being trusted

FAQ

Is restic check on its own enough? It covers level 2 well and, with --read-data-subset, samples level 3. It never restores anything, so it cannot prove level 4. Treat it as the nightly baseline, not the whole strategy.

Doesn't all this verification double my costs? Levels 1–3 are metadata, checksums, listings, and sampled reads — cheap. Level 4 restores cost scratch compute for a few hours a month. Compare that with the cost of discovering, mid-incident, that the backups were empty.

My backups are managed by my cloud provider. Am I covered? Managed backup services still produce artifacts that can be empty, misconfigured, or pointed at the wrong source. The provider runs the job; verification that the artifact restores your data is still yours.

What is the one change to make first? Add the dead-man's-switch alert on "time since last successful verification." It is a small change, and it converts the worst failure mode — silent, indefinite, total — into a page within a day.

Summary

An exit code tells you a script ran. A trustworthy backup has been checked at four levels: it exists, it is intact, it can be read, and it has been restored with the data verified. Automate the first three nightly; schedule the fourth — weekly partials, monthly full restores on rotation, quarterly timed drills. And page when verification fails or goes missing, not merely when a job crashes. The engineer in the opening story had monitoring. What they did not have was a single check that asked the only question that matters: if I needed this backup tonight, could I restore it? Build the checks that answer that question while the answer is still cheap.

TL;DR: Prometheus fails teams in two predictable ways: cardinality explosions from unbounded labels, and alert noise that trains everyone to ignore pages. The fixes are disciplines, not features. Model metrics as counters, gauges, and histograms with small, bounded label sets; measure services with RED (rate, errors, duration) and resources with USE (utilisation, saturation, errors); compute ratios and quantiles with recording rules so dashboards and alerts stay fast; and alert on user-visible symptoms with an action attached — via Alertmanager grouping and inhibition — never on every cause you can measure. Keep Prometheus itself for what it is (superb short-to-medium-term metrics), and remote-write to long-term storage when history matters.

Originally published on DevOps Enginer.

The model in one picture

Prometheus pulls: on an interval, it scrapes an HTTP endpoint (conventionally /metrics) on each target, parses the exposition format, and stores samples in a time-series database. A time series is a metric name plus a set of labels — http_requests_total{job="api", status="200"} — and each unique combination is a separate series with its own storage and query cost. That one sentence explains almost every Prometheus behaviour and failure mode in this guide.

Three metric types do the work:

• Counters only increase (requests served, errors seen). You almost never graph a counter raw; you graph its rate() — which also handles counter resets when a process restarts.
• Gauges go up and down (queue depth, memory in use, in-flight requests). Current value, min/max/avg over a window — no rate needed.
• Histograms count observations into buckets (le labels), which is what makes server-side quantiles possible: histogram_quantile() aggregates across instances, which summary-type metrics cannot do correctly. Choose bucket boundaries around the latencies you actually care about — your SLO threshold must live inside your buckets to be measurable.

What to measure: RED and USE

Instrumentation fails by excess more often than by absence. Two checklists keep it purposeful:

• RED for every service and endpoint that matters: request Rate, Errors (ratio, not just count), Duration (histogram). This is the user-visible truth of a service, and the raw material of every good alert below.
• USE for every resource: Utilisation (busy fraction), Saturation (queued work — the earliest warning), Errors. CPU steal aside, saturation metrics are the ones that predict incidents instead of narrating them.

Everything else — business metrics, per-feature counters, debug detail — is optional gravy. High-cardinality detail belongs in logs (the logging guide's correlation IDs are exactly for this), not in labels.

Cardinality: the budget everyone blows

Series count is the product of label-value combinations. A metric with 10 endpoints × 5 status codes × 4 instances is 200 series — fine. Add a user_id label and it becomes millions — the database slows, memory climbs, queries time out, and the postmortem writes itself. The rules:

• Labels identify dimensions with few, bounded values: endpoint (normalised — /users/123 becomes /users/:id before it is a label), status class or code, method, instance, zone, version.
• Never label with unbounded values: user IDs, request IDs, session tokens, full URLs, error message text. Each is a series factory.
• Scrape-time relabeling is the safety net: Prometheus can drop or rewrite label values during scraping — the place to strip an offending label from a third-party exporter you cannot change.
• Watch the budget: the targets and metrics with the highest series counts should be known numbers in your stack, reviewed like cloud spend (the Kubernetes cost guide makes the same argument for compute).

PromQL that earns its keep

Four patterns cover most production questions:

Error ratio (the alert-worthy form of errors):

sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

Latency quantile from a histogram:

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[10m])))

Saturation — in-flight work versus capacity is usually a gauge ratio; the shape matters more than the exact metric names.

Absence detection — absent() catches the series that stopped, which is how dead exporters and vanished jobs announce themselves; pair it with a dead man's switch on the alerting pipeline itself, so "no alerts" can be distinguished from "alerting is broken."

Two habits separate useful PromQL from fragile PromQL: always rate() counters (never subtract samples by hand), and aggregate with sum by (...) naming exactly the labels you intend to keep — a query that silently sums across a label you cared about produces confident, wrong dashboards.

Recording rules: pay for queries once

Dashboards and alerts re-run the same expressions relentlessly. Recording rules precompute them into new series on an interval:

groups:
  - name: api-slo
    interval: 30s
    rules:
      - record: job:http_error_ratio:5m
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

Now the alert is trivial, fast, and testable against the recorded series:

      - alert: ApiHighErrorRate
        expr: job:http_error_ratio:5m > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API 5xx ratio above 5% for 10 minutes"
          runbook_url: "https://wiki.example.com/runbooks/api-errors"

Note the anatomy of a good alert: a symptom (error ratio, sustained), a for duration that ignores blips, a severity, and a runbook link. An alert without a next action is a notification, not an alert.

Alerting philosophy: symptoms page, causes ticket

Alert fatigue is an availability risk in its own right — a team paged twenty times a night stops reading page one. The discipline that fixes it:

• Page on symptoms users feel: error ratio, latency burn, saturation of the user path. These have a natural threshold — the SLO — and a natural owner.
• Ticket (or dashboard) the causes: disk filling, a replica down, a certificate ageing. Important, rarely 3 a.m.-shaped; a Deployment with one of three replicas down is healthy by design and should page nobody.
• Burn-rate thinking: rather than a single static threshold, alert on how fast the error budget is being consumed (fast burn pages, slow burn tickets). Recording rules make multi-window burn alerts practical; the concept matters more than any specific window pair.
• Alertmanager does the traffic control: grouping collapses a hundred instance alerts from one outage into one notification; inhibition suppresses downstream symptom alerts while a known root alert fires; silences mute deliberately during maintenance; routing sends each alert to the team that owns the action. Configure these on purpose — default Alertmanager behaviour is a starting point, not a design.

Architecture and storage honesty

• Local retention is short by design. A Prometheus server's local TSDB commonly retains around 15 days by default; that is a feature of its role — fast, recent, operational metrics — not a limitation to apologise for.
• Long-term history goes elsewhere: remote-write to Thanos, Grafana Mimir, or VictoriaMetrics (the common open options) when you need months of data for capacity and trend work. Keep the operational Prometheus queryable even if the long-term tier is down.
• Federation is rarely the answer it first appears to be; hierarchical scraping duplicates data and queries painfully. Prefer remote-write and per-team/per-cluster servers with a global query layer.
• In Kubernetes, the Prometheus Operator's ServiceMonitor/PodMonitor resources turn scrape config into the same GitOps-reviewed YAML as everything else in this series (the observability guide covers the wider stack); kube-state-metrics supplies object state (pending pods, failed jobs, replica mismatches) and node exporter supplies host USE metrics. Watch the operator's own cardinality — Kubernetes state is prolific.
• Short-lived jobs can't be scraped on an interval; the Pushgateway exists for them, with real caveats: it doesn't expire series for you, and pushed metrics lose the scrape's health signal. Prefer recording outcomes from the platform side (the Jobs guide's failure conditions) or pushing sparingly and deliberately.

Common mistakes

• Unbounded labels, the cardinal sin above — most often a well-meant user_id, path, or error label that turns a metric into a million series over a weekend.
• Alerting on causes. CPU at 80% pages; the error budget untouched. Pages spent on non-symptoms are how real symptoms get muted.
• Counters graphed raw — ever-climbing lines that say nothing; or worse, alerts on them. rate() exists; use it.
• Averaging quantiles. Averaging per-instance p95s produces a number that is no instance's p95 and nobody's truth. Histograms plus histogram_quantile over the aggregate is the correct shape.
• No recording rules on heavy dashboards — the same 30-day, high-cardinality query from every open browser tab, grinding the server that is supposed to diagnose the incident.
• Treating Prometheus as the archive. Local retention quietly rotates; the quarter's capacity review finds 15 days of data. Decide the retention tiers like log retention (logging guide): hot and short in Prometheus, long in the remote tier.
• Silent alerting pipeline. Alertmanager down, webhook expired, routing typo — discovered during the incident it existed to announce. A dead man's switch (an always-firing alert whose absence pages) is the standard guard.

Where this fits

Prometheus is the metrics pillar of the observability guide's triad: metrics say what and how much, logs say what exactly, traces say where in the path. The discipline rhymes across all three — bounded cardinality here, volume control in logging, sampling in tracing — because all three are the same budget wearing different clothes: keep the signal, shed the noise, and make sure the alerting end of the pipeline deserves the trust the team puts in it.

FAQ

Prometheus or Grafana — do I need both? They are different layers: Prometheus collects and stores metrics and evaluates alerts; Grafana visualises (Prometheus has a basic built-in UI for queries, not dashboards). Most stacks run Prometheus (+ Alertmanager) for data and alerting, Grafana for dashboards — Grafana also fronts logs and traces, which is where the triad becomes one investigation.

Pull or push — which is right? Pull, by default: it gives Prometheus target health for free (up series), centralises interval control, and survives target restarts gracefully. Push (Pushgateway) is the exception for genuinely short-lived batch work, used with its caveats in mind.

How long should Prometheus retain data? Operationally, two to four weeks local is common and usually plenty for incident work. Anything longer belongs in a remote-write long-term tier, where storage is cheaper and the queries are trend-shaped rather than incident-shaped.

Prometheus or OpenTelemetry? Complementary, not rivals. OpenTelemetry is instrumentation and pipelines (APIs, SDKs, the Collector); Prometheus is a metrics backend with its own exposition format, which OTel can produce and feed (directly or via remote-write-compatible paths). Many stacks instrument with OTel and still query with PromQL.

When have I outgrown a single Prometheus? When scrape volume, cardinality, or query load makes the single server the bottleneck — symptoms are scrape duration creeping toward the interval, slow range queries, and memory pressure dominated by series churn. The step is sharding by team/cluster plus a remote-write tier, not a bigger instance forever.

TL;DR: Deployments run forever; Jobs run to completion — and that one difference changes how you configure everything. A production Job needs an explicit contract: restartPolicy: Never (or OnFailure, knowingly), a backoffLimit you chose, an activeDeadlineSeconds so nothing runs forever, and ttlSecondsAfterFinished so finished work cleans itself up. CronJobs add three decisions on top: a timeZone, a concurrencyPolicy that matches whether runs may overlap, and a startingDeadlineSeconds so a controller outage doesn't fire a pile of missed runs at once. Batch failures are normal — design for retries that are idempotent, and alert on failure rather than discovering it in the data.

Originally published on DevOps Enginer.

The workload Deployments can't express

Everything earlier in this series assumes services: replicas that should always exist, replaced when they die. Batch work inverts the assumption — a migration, a report, a backup, an import has a done, and success means the pods stop existing. Running batch work in a Deployment produces the classic pathologies: a completed container restarted forever, a failing task CrashLooping with no concept of "give up," no record of whether the run succeeded.

The Job controller exists to own that lifecycle: create pods, watch them finish, count successes against completions, count failures against backoffLimit, and stop. CronJob is a Job factory on a schedule — nothing more mystical than that, which is the right way to reason about both.

The Job contract, field by field

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-export
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 3600
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: export
          image: registry.example.com/export:1.4.2
          args: ["--date", "yesterday"]

The fields that decide behaviour:

• restartPolicy: Never (or OnFailure) — Jobs require one of these two; Always is rejected. The choice matters: with Never, a failed container means the pod fails and the controller creates a replacement pod (visible, countable, each with its own logs). With OnFailure, the kubelet restarts the container inside the same pod — fewer pod objects, murkier failure history. Default to Never unless you have a reason; it makes failures legible.
• backoffLimit (default 6) — how many pod failures before the Job itself is marked Failed. Each retry waits with exponential backoff (10s, 20s, 40s… capped at 6 minutes). Six silent retries of a broken image is how a bad deploy burns an hour unnoticed; for most batch work 2–4 is a saner, louder choice. Finer control exists via pod failure policies — rules that fail the Job immediately on specific exit codes (a config error will never fix itself) while ignoring disruptions like node drains.
• activeDeadlineSeconds — the Job's total time budget; exceed it and the Job fails and its pods are terminated. This is the field that prevents the "export job from March is still running" class of surprise.
• ttlSecondsAfterFinished — after the Job completes or fails, wait this long, then delete the Job and its pods. Without it, finished Jobs and their pod corpses accumulate forever (and with them, cluttered kubectl get pods output and etcd weight). A day is a reasonable default window for debugging; the logging pipeline from the logging guide is what preserves evidence beyond it.
• completions / parallelism — for work queues: completions is the total successes required, parallelism the pod ceiling. Leave both unset for the simple one-shot case (one success, one pod). Indexed Jobs (completionMode: Indexed) give each pod a stable index (JOB_COMPLETION_INDEX) for sharded work — partition the input by index instead of running a separate queue service.

Idempotency: the requirement retries impose

A retried Job runs your code again against a world the first attempt may have half-changed. Batch code must therefore be idempotent or transactional: write to a staging key and publish atomically, upsert rather than insert, name outputs deterministically (by date/partition), and check-before-create where the sink allows. The failure this prevents is specific and common: attempt one writes 90% of the export and dies; attempt two appends a second 100%; downstream reads 190% and nobody knows which records are real. Retries are the controller doing its job — making them safe is yours.

CronJobs: a schedule plus three decisions

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-export
spec:
  schedule: "15 2 * * *"
  timeZone: "Asia/Tokyo"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 900
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 3600
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: export
              image: registry.example.com/export:1.4.2
              args: ["--date", "yesterday"]

• timeZone — set it, always. Without it the schedule runs in the controller-manager's timezone (effectively UTC on most clusters), and "2:15 a.m." silently means something else after a cluster migration. The field takes an IANA name and handles daylight-saving transitions the way the name implies.
• concurrencyPolicy — the overlap decision, made explicitly:
  • Allow (the default): runs may stack up. Right for independent, cheap, idempotent work; dangerous when a slow run plus a punctual schedule compounds load.
  • Forbid: skip a scheduled run while the previous is still going. The right default for most sequential batch work — exports, backups, syncs.
  • Replace: kill the running Job and start fresh. For "latest state wins" work where a stale run is worse than none.
• startingDeadlineSeconds — if the controller misses a start (downtime, upgrades), how late a run may still fire. Without it, a controller recovering after an outage can launch a burst of stale runs; with it, missed runs beyond the window are skipped. Roughly 100 missed starts is the controller's own safety cap, but your deadline should be far tighter and deliberate.
• History limits — successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) bound how many finished Job objects linger for inspection. Failed history is debugging gold and set lowest by default — raise it if you investigate on Mondays what failed on Fridays.
• suspend: true — the pause button. CronJobs honour it immediately; it is the clean way to stop a schedule during an incident without deleting the object.

Failure is a normal state — observe it like one

A service failing pages someone; a Job failing at 2 a.m. writes a status field nobody reads. Close that gap deliberately:

• Alert on Job failure and on CronJob staleness. The signals exist in the metrics stack from the observability guide: job-failed conditions, and time since a CronJob's last successful run versus its schedule. "The nightly export hasn't succeeded in 26 hours" is an alert; discovering it from a stale dashboard is an incident review.
• Logs outlive pods only if you shipped them. Failed pods are deleted by TTL and history limits by design — the pipeline from the logging guide is the memory. At minimum, make the failure visible where the team already looks (the alert carries the job name; the runbook says where its logs live).
• Distinguish "failed" from "never ran." Scheduler outages, suspended CronJobs, image-pull failures before any pod starts — the Job object tells different stories in each case (conditions, events, pod statuses), and the fixes differ. The troubleshooting guide's habit applies: read the object's status before touching anything.

Resource and disruption behaviour for batch

• Requests still matter, limits protect neighbours. Batch jobs are the classic noisy neighbour — bursty CPU during everyone else's peak. Set requests honestly (requests-and-limits guide) and consider scheduling heavy batch into off-peak windows or lower-priority classes so services win contention; a report can wait 20 minutes, a checkout cannot.
• Disruption budgets apply differently. A Job's pods are not protected by PDBs the way replicas are (the PDB guide covers voluntary disruptions) — node drains during a long Job mean pod termination and a controller retry. Long jobs should checkpoint progress or be restartable from partial state; the idempotency work above is what makes that survivable rather than catastrophic.
• One-off manual runs: kubectl create job --from=cronjob/nightly-export export-manual-1 clones the template for an immediate run — the clean pattern for backfills and tests, far better than hand-writing a twin Job manifest that drifts from the scheduled one.

When a Job is the wrong tool

• Long-running or never-finishing work is a service — use a Deployment.
• Per-node work (agents, collectors) is a DaemonSet's shape, not a Job's.
• Complex DAG workflows — fan-out/fan-in pipelines, artefact passing, per-step retries with dependencies — outgrow the Job controller quickly. That is the territory of workflow engines (Argo Workflows and kin); the Job primitives remain the right answer for the large simple majority: one task, a schedule or a trigger, a definition of done.

Common mistakes

• restartPolicy: Always copied from a Deployment. The API rejects it for Jobs — usually discovered at apply time in a hurry. The template is not interchangeable.
• No deadline. A hung Job holds resources and blocks Forbid-policy successors indefinitely; the schedule looks fine and nothing runs. activeDeadlineSeconds is the difference between a failure and a mystery.
• Retries without idempotency. The double-write pattern above; the controller retries by design, so the hazard is guaranteed to fire eventually.
• Default backoffLimit and no alerting. Six quiet retries, a Failed status, and silence. Batch observability is an alerting decision, not a dashboard.
• CronJobs in the wrong timezone. The 2 a.m. job running at 11 a.m. after a migration is a rite of passage; the timeZone field is how you skip it.
• Allow concurrency on heavy work. One slow run plus an hourly schedule equals a self-inflicted load test by morning.
• Secrets via environment for one-shot jobs. Jobs end; their specs (and any env values in them) linger in history objects and etcd. The ConfigMaps and Secrets guide's file-based patterns apply with extra force to objects designed to be kept around as records.

Where this fits in the series

Batch is the third workload shape in this series, beside services (Deployments) and per-node agents (DaemonSets): it leans on the requests-and-limits guide for fair scheduling, the storage guide when jobs move real data, the backup guide — whose restores and snapshots are themselves usually Jobs — and the autoscaling guide's cluster-autoscaler behaviour, which treats pending batch pods as the elastic demand they are. Model the done-state honestly and the controller does the rest.

FAQ

Job or CronJob for a task I trigger from CI? A Job, created by the pipeline when the event happens (a migration after deploy, an export on demand). CronJob is for wall-clock schedules; forcing event-driven work into a polling CronJob adds latency and pointless runs.

Why did my CronJob not run at the expected time? In order of likelihood: timezone (no timeZone set, cluster in UTC), the previous run still going under Forbid, suspend: true left on, or a missed start beyond startingDeadlineSeconds after controller downtime. The CronJob's status and events record which.

How many times will a failing Job retry? backoffLimit pod failures (default 6), with exponential backoff between attempts — then the Job is marked Failed and stops. With pod failure policies, specific exit codes can fail it sooner, on the first occurrence.

Can a Job run on a schedule and be triggered manually? Yes — define the CronJob, and create one-off Jobs from its template with kubectl create job --from=cronjob/<name>. The manual run uses the same spec; it just isn't part of the schedule's history.

Should batch jobs share the cluster with services? Usually, yes — with honest requests, sane priorities, and schedules that respect peak hours. Dedicated node pools earn their cost when batch is heavy, spiky, or needs special hardware; reach for isolation when contention is measured, not imagined.

TL;DR: Kubernetes RBAC fails in practice not because the model is weak but because grants accrete: a wildcard here, a cluster-admin binding there, a "temporary" service account that becomes permanent. The model itself is simple — subjects get verbs on resources in a scope, grants are additive only, and nothing is denied, only not granted. Least privilege that holds means: one service account per workload with token automount off by default, namespace-scoped Roles as the default unit, groups (not individuals) as human subjects, secrets treated as the crown jewels they are, and a regular audit of who can do what — verified with kubectl auth can-i, not assumed.

Originally published on DevOps Enginer.

Why RBAC is where cluster security is won or lost

The security-basics guide in this series covered the checklist; this guide goes deep on the one control that decides what a compromised credential can actually do. Most cluster incidents follow the same shape: an attacker (or a fat-fingered script) obtains a token — from a pod, a CI job, a leaked kubeconfig — and the blast radius is whatever that token may do. Network policies limit where it can talk; RBAC limits what it can be. A read-only token in one namespace is an embarrassment. A cluster-admin token is a breach report.

And RBAC sprawl is the default outcome, because every grant is a small act of unblocking someone: the deploy pipeline "needs" broader rights, the dashboard "needs" cluster-wide secrets, the new tool "needs" admin "for setup." Each is reasonable on the day. Together they are the incident, pre-positioned.

The model, precisely

Four ideas carry the whole system:

• Subjects — who: a User (external identity, typically from a certificate or OIDC token), a Group, or a ServiceAccount (in-cluster identity).
• Rules — what: verbs (get, list, watch, create, update, patch, delete, and a few specials) on resources (pods, deployments, secrets…) within API groups ("" for core, apps, batch, rbac.authorization.k8s.io…).
• Scope — where: a Role grants within one namespace; a ClusterRole grants cluster-wide or for cluster-scoped resources (nodes, persistent volumes, namespaces themselves).
• Bindings — the connection: a RoleBinding attaches a Role or ClusterRole to subjects inside one namespace (a ClusterRole bound by a RoleBinding is effectively trimmed to that namespace); a ClusterRoleBinding attaches a ClusterRole everywhere.

Two properties shape everything else:

1. Additive only. There are no deny rules. A subject's rights are the union of all its bindings. "Remove the bad grant" always means deleting or narrowing a binding — and "who granted what" questions are answered by enumerating bindings, never by finding an exception.
2. Default deny. With RBAC as the authorisation mode, a subject with no grants can do nothing. Every capability is an explicit decision — which is why the accretion pattern above is a process failure, not a model failure.

A worked example: read-only log access for support

The support team needs to read pods and logs in one production namespace — nothing else, nowhere else:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: shop-prod
  name: log-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: shop-prod
  name: log-reader-support
subjects:
  - kind: Group
    name: support-engineers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: log-reader
  apiGroup: rbac.authorization.k8s.io

Details doing quiet work here: the subject is a Group (people join and leave groups; bindings should not name individuals), pods/log is a subresource that must be granted explicitly, and there is no secrets, no create, no second namespace. Verify it the way an auditor would:

kubectl auth can-i get pods --namespace shop-prod --as jane --as-group support-engineers   # yes
kubectl auth can-i get secrets --namespace shop-prod --as jane --as-group support-engineers # no
kubectl auth can-i list pods --all-namespaces --as jane --as-group support-engineers         # no

kubectl auth can-i with --as/--as-group impersonation is the ground-truth test for any identity — build the habit of checking grants with it instead of reading YAML and reasoning hopefully.

Service accounts: the identities that actually get stolen

Humans are rarely the compromised identity; workloads are. Service-account discipline is therefore the highest-value RBAC work:

• One service account per workload. The default service account in each namespace should have no grants. A dedicated account per deployment makes grants reviewable and revocation surgical.
• Turn off token automount unless needed. Most applications never call the Kubernetes API. For them the token is pure attack surface:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: shop-api
  namespace: shop-prod
automountServiceAccountToken: false

(The same field exists per-pod in the pod spec, which wins when both are set.) Where a token is needed, modern clusters issue short-lived projected (bound) tokens rather than long-lived secrets — prefer them, and treat any long-lived service-account token secret as a credential to inventory.

• Deployer accounts are prime targets. A CI service account that can create deployments in production can, by design, run arbitrary images with that namespace's service accounts — near-admin inside its scope. Narrow its verbs to exactly the resources the pipeline applies, keep it namespace-scoped, and protect its credential like a production key (the OIDC guide's keyless pattern applies to how the pipeline authenticates to the cloud; the in-cluster account deserves the same suspicion).

ClusterRoles: handle with both hands

Cluster-wide grants are sometimes right — an observability agent reading node and pod metadata everywhere, an operator managing its resource type across namespaces. The disciplines:

• Prefer a trimmed custom role over the built-ins' extremes. The default cluster-admin is total; admin, edit, and view are namespace-oriented defaults with specific shapes (notably, view excludes secrets — a deliberate, instructive choice). For cluster-wide read, write the ClusterRole that lists exactly the resources the tool reads; "cluster-reader" patterns from the community are a starting point, not a destination.
• Aggregation composes, so audit the aggregate. A ClusterRole with an aggregationRule collects rules from labelled ClusterRoles (this is how the built-in admin/edit/view roles grow with CRDs). Powerful for platforms, dangerous when a stray label quietly widens a widely bound role — include aggregated roles in audits.
• ClusterRoleBindings deserve a list, kept short. Every entry is a cluster-wide statement. Review the full list on a schedule; in a healthy cluster it fits on one screen and every line has an owner who can explain it.

The dangerous verbs and resources

Audits should hunt specific shapes, because damage concentrates:

• Wildcards — verbs: ["*"] or resources: ["*"] in anything but a deliberately tiny set of admin roles. A wildcard silently includes tomorrow's new subresources.
• get/list on secrets — especially cluster-wide. Reading secrets is effectively holding every credential in scope (the ConfigMaps and Secrets guide covers why). Treat any new secrets grant as a security review, not a config change.
• create on pods — pod creation with a chosen service account and mounts is a well-trodden path to privilege escalation; so are create on rolebindings/clusterrolebindings and the special bind and escalate verbs, which exist precisely to gate grant-making itself. impersonate is equally sensitive: it is the other identity, for whoever holds it.
• deletecollection and broad delete — the availability side of the same coin.

Human access: groups, short-lived credentials, no shared kubeconfigs

• Map your identity provider's groups to Group subjects once, and manage membership in the IdP — not by editing bindings per person. Joiner/leaver processes then work through the system people already trust for access changes.
• Issue short-lived credentials: exec-plugin/OIDC flows from your cloud or IdP, so a copied kubeconfig expires instead of opening a permanent door. A long-lived client certificate cannot be revoked (Kubernetes has no certificate revocation for this) — another reason to prefer token-based, expiring auth.
• Break-glass, deliberately. Keep an emergency admin path, alarm on its use, and test it — an incident is the wrong time to discover nobody can grant the rights to fix it. Emergency access is a designed feature; routine access through it is the failure.

Auditing: make "who can do what" a query, not a debate

• kubectl auth can-i --list --as <identity> prints an identity's effective rights in a namespace — the fastest spot check.
• Enumerate bindings as code review: RoleBindings and ClusterRoleBindings are YAML in your GitOps repo (the Argo CD guide's pattern) — changes to them should be pull requests with security-aware reviewers, not kubectl improvisation.
• Tooling in the kubectl-who-can / RBAC-lookup family answers the reverse question ("who can get secrets in prod?") across the cluster; run it in CI or on a schedule and diff the output — new entries are events worth a human glance.
• API-server audit logs record who did what (the logging guide covers collecting them); RBAC denials in those logs are a signal of both attacks and of teams fighting their own permissions — both worth knowing.

Common mistakes

• Binding cluster-admin "temporarily." Temporary grants are permanent until the audit that never happens. Time-box with a ticket, an owner, and a removal date — or better, grant the specific role the task needs.
• One service account for everything in the namespace. Every pod inherits the union of rights; one compromised container spends them all.
• Secrets access for convenience. Dashboards and debug tools asking for cluster-wide secrets are asking for the keys to everything. Scope to the namespace, or redesign the tool's access.
• Confusing RoleBinding-to-ClusterRole scope. Binding a ClusterRole via a RoleBinding grants it only in that namespace — teams regularly believe they granted (or limited) something else. Read the binding kind, not just the role kind.
• Stale bindings. People change teams; projects end; vendor tools are uninstalled and their service accounts and bindings remain, valid and forgotten. The audit diff is how they surface.
• Testing with your own admin account. It works for you, so permissions "work." Test as the workload's identity (--as=system:serviceaccount:ns:name or by running the pod) before the 3 a.m. discovery.

Where this fits in the series

RBAC is the identity layer the rest of the series assumes: the security-basics guide sets the posture, the network policies guide constrains where identities can reach, the ConfigMaps and Secrets guide protects what grants can expose, and the observability and logging guides watch it all — including the audit stream that records every grant being used. Get RBAC boring, and the other controls get to do their jobs.

FAQ

Can RBAC deny a specific action? No. RBAC is additive — a subject's permissions are the union of its grants, and there is no deny rule to carve exceptions. If a grant is too broad, narrow the role or split it; for true deny semantics (blocking an action even for the granted), admission-time policy tools are the layer that does it.

Role or ClusterRole for a cross-namespace tool? If it genuinely acts in many namespaces, a ClusterRole — bound cluster-wide only if it must act everywhere, or via RoleBindings in selected namespaces when its reach should be enumerated namespace by namespace. The second pattern keeps the grant list explicit and reviewable.

Should each pod have its own service account? Each workload identity should: one per deployment/workload is the practical unit, shared by its replicas. Per-pod accounts multiply objects without adding a trust boundary; per-namespace sharing erases the boundary that exists.

How do I see everything a service account can do? kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<name> -n <namespace> for a namespace view, repeated per namespace or answered cluster-wide with RBAC lookup tooling. Make it a habit after every binding change.

Are the built-in admin/edit/view roles safe to use? They are reasonable, maintained defaults — with shapes you should know: view excludes secrets and most write access, edit adds writes but not role management, admin adds role management within the namespace but not namespace deletion or quota changes. The risk is rarely the built-ins; it is the custom wildcard beside them.

TL;DR: Most Terraform changes are still reviewed by reading a plan and hoping. The native test framework (terraform test, stable since Terraform 1.6) changes that: test files declare inputs, run a plan or apply, and assert on the results — and with mock providers (since 1.7) the logic tests run in CI in seconds with no cloud account and no credentials. Test your modules' contracts — outputs, conditional resources, naming and tagging rules — not the cloud provider's behaviour. Keep a small number of real apply tests for critical modules in an ephemeral sandbox, and let linters and scanners do their separate jobs alongside.

Originally published on DevOps Enginer.

The problem: the plan is not a test

Application code gets unit tests as a matter of course. Infrastructure code — which can delete a database just as dead as any application bug — usually gets a human skimming terraform plan output in a pull request. That review catches typos. It does not catch the conditional that inverted last refactor, the variable default that quietly changed a bucket from private to public, or the count logic that creates zero replicas of the thing production needs three of.

The reason was tooling: for years, testing Terraform meant heavyweight frameworks or applying to a real account and inspecting by hand. The native framework removes the excuse. Tests live next to the module, run with one command, and — the decisive part — can run against mocked providers, so the suite executes like unit tests: fast, hermetic, credential-free.

What is worth testing (and what is not)

Test your module's decisions, not the provider's implementation:

• Outputs and contracts. Callers depend on output values and shapes. If bucket_name changes format, downstream code breaks — a test should break first.
• Conditional logic. Every count, for_each, and ternary is a decision your module makes: versioning on/off, optional resources created or skipped, single vs multi-AZ shapes. Each branch deserves a run.
• Naming, tagging, and policy invariants. The rules platform teams actually care about: names compose from project/environment correctly, required tags are always present, encryption is enabled, nothing is public unless an explicit variable says so.
• Variable edge cases. Empty lists, zero counts, nulls — the inputs real callers will eventually pass.

Do not test that AWS creates an S3 bucket when asked, that an attribute AWS generates looks a certain way beyond your control, or other provider internals. Those tests are slow, brittle, and verify someone else's software.

The framework in one module

A small module to test:

# main.tf
variable "project"     { type = string }
variable "environment" { type = string }

variable "enable_versioning" {
  type    = bool
  default = true
}

resource "aws_s3_bucket" "this" {
  bucket = "${var.project}-${var.environment}-assets"
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id

  versioning_configuration {
    status = var.enable_versioning ? "Enabled" : "Suspended"
  }
}

output "bucket_name" {
  value = aws_s3_bucket.this.bucket
}

And its test file (conventionally in a tests/ directory inside the module):

# tests/bucket.tftest.hcl
mock_provider "aws" {}

variables {
  project     = "shop"
  environment = "prod"
}

run "versioning_on_by_default" {
  command = plan

  assert {
    condition     = aws_s3_bucket_versioning.this.versioning_configuration[0].status == "Enabled"
    error_message = "Versioning must be enabled by default"
  }

  assert {
    condition     = output.bucket_name == "shop-prod-assets"
    error_message = "Bucket name must compose project and environment"
  }
}

run "versioning_can_be_disabled" {
  command = plan

  variables {
    enable_versioning = false
  }

  assert {
    condition     = aws_s3_bucket_versioning.this.versioning_configuration[0].status == "Suspended"
    error_message = "Versioning should suspend when explicitly disabled"
  }
}

Run it with terraform test from the module root. The anatomy:

• mock_provider "aws" {} replaces the real provider. Terraform generates fake computed values, so a plan (or even an apply) runs without credentials, network, or a cloud account — in milliseconds.
• variables {} at file level sets shared inputs; a run block can override them per case.
• run blocks are the test cases. command = plan is the default workhorse; command = apply executes against the (possibly mocked) provider for shape checks that only appear after apply.
• assert pairs a condition over any value in the configuration — resources, outputs, variables, locals — with the error_message a human reads when it fails. Write the messages as explanations, not echoes.

Where a mocked computed value needs to be specific — an ARN a policy references, an ID another resource consumes — override_resource and override_data blocks pin those values inside a run. Use them sparingly; a test that overrides everything asserts nothing.

Mock providers: what they change, and their honest limits

Before mocks, a "unit test" of a module still configured the real provider, which meant credentials in CI, provider API calls in every plan, and suites measured in minutes. Mocks invert all three: the pipeline job needs no cloud credentials at all for the test stage (which, per the keyless-CI pattern in the GitHub Actions OIDC guide on this site, is exactly where you want the fewest credentials), tests run at unit-test speed, and nobody's sandbox account accumulates test debris.

The limits are equally clear, and stating them keeps the suite trustworthy:

• Mocks validate your configuration logic against the provider's schema, not against the real API. A value the real AWS API would reject (an invalid name pattern, an unsupported combination) can sail through a mocked plan.
• Data sources return generated values unless overridden — tests depending on a data source's real content need overrides or a real read.
• Provider-side quirks and eventual-consistency behaviour are, by definition, absent.

So the strategy is layered: mocked tests for logic, in bulk; a few real tests for reality, on purpose.

The top of the pyramid: real apply tests, kept few

For a critical module — the network foundation, the shared database module — a mocked plan cannot prove the one thing that matters: it actually builds. Keep a small suite of command = apply runs against the real provider in a sandbox account with three disciplines:

• Ephemeral by design. Tests create, assert, and the framework destroys what it created at the end of the run (that is the framework's contract for test-created resources). Verify the destroy happened; orphaned test resources are a cost and security smell — the state-management guide's hygiene rules apply to test state too.
• Separate credentials, narrow scope. The sandbox role can create and destroy only the resource types under test, in the sandbox account only. The OIDC guide's per-environment role pattern fits exactly here.
• Scheduled, not per-commit. Real applies take minutes and cost real (if small) money. Run them nightly or on module release, while the mocked suite gates every pull request.

Where tests sit among the other tools

Terraform already has a quality neighbourhood, and the test framework does not evict anyone:

• Variable validation blocks — the module rejecting bad input at plan time. First line of defence, and the test suite should cover that they fire.
• terraform fmt / validate — shape and syntax. Free; run always.
• tflint and friends — lint rules, naming conventions, deprecated syntax. Style and hygiene, not behaviour.
• Checkov / tfsec-style scanners — policy and security posture of the configuration. They check what the config says; tests check what the module decides given inputs. Complementary, not overlapping.
• terraform test — behaviour: outputs, conditionals, invariants across input variations.

A sane pipeline order: fmt check → validate → lint/scan → terraform test (mocked) → plan for human review. Each stage is cheap; together they mean the human reviews intent, not correctness arithmetic.

Modules are where tests pay rent

For a root configuration used once, tests are nice. For modules — shared, versioned, consumed by teams who did not write them — they are the contract enforcement mechanism. The modules guide in this series covers structuring reusable modules; tests are what let a maintainer refactor a module's internals while proving its callers see the same outputs, the same resource decisions, the same invariants. That confidence is precisely what a module library runs on, and it is why test files should ship inside the module repository, versioned with the code they describe.

Common mistakes

• Testing the provider. Asserting that a mocked (or real) AWS resource has AWS-chosen properties teaches the suite to fail for other people's changes. Assert your decisions.
• Overriding everything. Heavy override_resource use pins the fake world so completely that the test passes regardless of the configuration. Overrides are scalpels for specific computed values, not a substitute world.
• One giant run block. A failed assert should name the broken decision. Many small runs with descriptive names beat one mega-run with forty asserts.
• Apply tests against shared accounts. Test resources in an account other teams use will eventually collide, linger, or be "cleaned up" by someone mid-run. Sandbox account, always.
• Mocked-only confidence for critical paths. The suite is green, the module cannot actually apply. Keep the few real apply tests for the modules whose failure would hurt.
• Ignoring test state. terraform test manages its own state for runs, but interrupted real applies can leave residue; a periodic sandbox sweep belongs in the routine.
• Writing tests once, at the end. The suite earns its keep when a refactor lands six months later. Add the run block when you add the conditional — that is when the intent is clearest and cheapest to encode.

FAQ

terraform test or Terratest? They answer different questions. terraform test is native, HCL-based, and ideal for module logic and contracts — no Go toolchain, no extra framework. Terratest (Go) remains strong for end-to-end verification that spans tools and real cloud behaviour beyond one module. Many teams use both: native tests in every module repo, Terratest-style end-to-end checks for platform assemblies.

Does terraform test cost money? Mocked and plan-only runs create nothing and cost nothing. command = apply runs against a real provider create real resources for the duration of the test — priced accordingly, then destroyed by the framework. Keep them few, small, and in a sandbox.

Can tests run in CI without cloud credentials? Mocked-provider tests: yes, completely — that is their headline feature. Plan runs against a real provider still need whatever credentials that provider requires to configure itself, which is one more reason the mocked suite should carry the bulk of the assertions.

Where do test files live? In the module root or a tests/ subdirectory (the conventional choice), as *.tftest.hcl. terraform test discovers them in both places; keep one convention per repository.

What about OpenTofu? OpenTofu implements the same test framework model — test files, run blocks, and mock providers are compatible concepts, so the approach in this guide carries across. Check your version's documentation for exact feature parity before assuming a specific block behaves identically.

TL;DR: In Kubernetes, logs die with the pod — and the pods you most need logs from are the ones that just died. The fix is an architecture, not a tool: applications write structured logs to stdout/stderr, a node-level agent (Fluent Bit, usually) collects and enriches them with Kubernetes metadata, and a backend (Loki, OpenSearch, or your cloud's) stores them with retention you chose on purpose. Add correlation IDs so a log line connects to a request and a trace, control the volume before it controls your budget, and test the pipeline by killing a pod and finding its last words.

Originally published on DevOps Enginer.

The problem: the evidence deletes itself

On a VM, a crashed application's logs sat on disk until someone looked. In Kubernetes, the sequence runs the other way: the pod crashes, gets rescheduled or replaced, the node eventually cleans up, and the logs — the primary evidence of what happened — evaporate on a schedule driven by everything except your investigation.

kubectl logs papers over this for exactly one generation: it shows the current container, and with -p the previous one. Before that? Gone. On another node after a reschedule? Gone. From the DaemonSet that was OOM-killed at 3 a.m.? Never existed as far as you can prove.

So the troubleshooting guide in this series starts with logs — and production logging is the discipline of making sure they are somewhere else before the pod takes them away.

The model: stdout is the application's only logging API

The rule from which everything else follows: a containerised application logs to stdout and stderr, and nowhere else. No log files inside the container, no application-managed rotation, no "the file is on the volume somewhere." The container runtime captures those two streams to files on the node (/var/log/containers/…), and the platform takes it from there.

This buys three things:

• Uniformity. Every workload, every language, same contract. The pipeline never needs per-application file paths.
• Rotation for free (with a caveat below): the kubelet and runtime rotate the node-side files; the application cannot fill its own container filesystem with logs.
• Streams stay separable. stderr is not automatically "errors" — plenty of frameworks log everything to stdout — but having both captured beats parsing one merged file inside a dying container.

The application-side obligations that make this work: one event per line, and structured — JSON lines with a timestamp, a level, a message, and the fields a future searcher will filter on (service, version, request ID). Plain-text prose logs are a tax every incident pays at the worst exchange rate.

The collector: a node-level agent

The standard shape is one log collector per node, running as a DaemonSet, tailing the runtime's log files, enriching each record with Kubernetes metadata (namespace, pod, container, labels — fetched from the API), parsing structure, and shipping to the backend.

The usual choice in 2026 is Fluent Bit: small, fast, CNCF-graduated, written for exactly this job. Fluentd (its older sibling) remains capable, especially where a team already runs its plugin ecosystem; a few platforms ship their own agents. What matters more than the brand:

• Metadata enrichment is non-negotiable. A log line without namespace/pod/labels is nearly unsearchable at cluster scale. Verify your pipeline adds them — silently missing metadata is the most common logging bug in production clusters.
• Buffering and backpressure. When the backend is unreachable, the agent must buffer (to disk, with a cap) and retry — and its buffer behaviour must not be allowed to fill the node disk it is protecting you from. Size it, alert on it.
• The agent is production software. It runs on every node, touches every log line, and fails silently when misconfigured. It belongs in the observability stack's alerting (the observability guide covers the wider stack): DaemonSet pods healthy, shipping lag near zero, drop counters at zero.

A note on audit logs: the API server's audit stream is a different animal — who did what to the cluster, the security record from the security-basics guide — collected at the control plane, retained separately, and never mixed into the application pipeline.

Where logs land: pick the backend by the questions you ask

• Loki — indexes only labels, stores compressed log chunks in object storage. Cheap to run, excellent when your queries are "this service, this namespace, this time window, containing this string" — which is most incident queries. Pairs with Grafana next to your metrics. Its discipline: labels are precious and few (service, namespace, level, environment). A user ID or request ID in a label explodes the index (the cardinality trap); those belong in the log line or in structured fields, not labels.
• Elasticsearch / OpenSearch — full-text indexing of everything: expensive in storage and operations, unmatched when you genuinely need arbitrary field search across huge volumes and long retention.
• Cloud-native sinks — CloudWatch Logs, Google Cloud Logging, Azure Monitor: zero extra infrastructure on the matching managed service, pricing that rewards a retention policy, and query languages your team may like less than LogQL. For many EKS/GKE/AKS clusters this is the pragmatic default, with the node agent either replaced by the provider's or forwarding to it.

Any of the three works. The expensive mistakes are the two defaults nobody chose: keeping everything forever, and discovering your per-GB ingestion bill during the incident that tripled log volume.

The sidecar pattern: when node-level isn't enough

Some applications will not log to stdout — legacy software writing to files, multi-line stack traces an application insists on assembling itself, or a workload needing local pre-processing. The pattern: a sidecar container in the same pod that reads the file from a shared volume and writes it to its own stdout (so the node agent picks it up normally), or ships directly to the backend. It works, at the price of a container per pod — resources, complexity, one more thing to upgrade. Reach for it per-workload, knowingly, not as the cluster default.

Log survival details that bite

• Rotation is configured at the node, and it has a size. Runtime/kubelet log rotation (commonly around 10 MiB per file, a handful of files) means a chatty container's history on the node may cover minutes, not hours. That is fine because the agent ships lines as they are written — but it is why "the agent was down for an hour" translates directly into "that hour's logs exist only where the agent later recovered them, if it could." Agent health is log retention.
• Node disk is a shared fate. Logs, images, and emptyDir volumes all eat the same disk; a runaway logger can trigger disk-pressure evictions of innocent pods (the storage guide covers the disk side). Resource discipline — requests and limits from that guide — applies to your loggiest workloads first.
• kubectl logs still matters. For live debugging it is the fastest tool that exists: kubectl logs -l app=api --since=10m --all-containers=true across a label selector, --previous for the crashed instance, --follow during a rollout. The pipeline is for history and search; kubectl is for now. Engineers should be fluent in both.

Correlation: logs that connect to everything else

A log stream alone answers "what did this service print." Production debugging asks a bigger question: "what happened to this request." The bridge is a correlation ID: generated at the edge (or taken from an incoming header), propagated through every service call, and printed in every log line and trace span. With that one discipline, an incident moves from grepping five services by timestamp to one query per system, all keyed by the same ID — and the metrics–traces–logs triad from the observability guide becomes a single investigation instead of three.

Two supporting habits:

• Levels with meaning. ERROR should mean "a human may need to act," not "a retry happened." When everything is an error, nothing is; alert on error rate changes, not on the existence of error lines.
• Never log secrets or personal data. Tokens, passwords, full request bodies with credentials, unnecessary personal fields — logs are copied, retained, and widely readable by design; treat them as the least-trusted storage you own. Redaction at the application beats scrubbing in the pipeline, and both beat explaining a credential found in a log bucket (the ConfigMaps and Secrets guide covers the secret-handling side).

Volume, retention, and the bill

Log volume scales with traffic and verbosity, and both spike during incidents — exactly when you can least afford dropped data or a surprise invoice. The controls, in the order teams usually meet them:

1. Retention tiers. Hot storage (fast search) for days-to-weeks; cheaper object storage for months; deletion after the period your compliance actually requires. "Forever" is a decision too — an expensive one nobody remembers making.
2. Drop or sample the noise. Health-check access logs from internal probes, debug output from one misbehaving workload, a namespace that logs every cache hit — filter at the agent, where dropping is free, rather than at the backend, where you have already paid to ingest.
3. Per-namespace visibility. Know which workloads produce the volume. The cost guide in this series makes the same point about compute; logs deserve the same per-team accounting, because the fix is always a conversation with one workload's verbosity.

Prove it: the kill-a-pod test

Like backups and disruption budgets before it in this series, a logging pipeline is a rumour until exercised. The test is five minutes:

1. Pick a workload. In its logs, find (or generate) a unique marker line — a distinctive request ID works.
2. Delete the pod. Wait for the node to forget it ever existed.
3. From the backend alone, retrieve that pod's final log lines, filtered by namespace, pod name, and the marker. Check the metadata is attached and the timestamps are ordered and correct.

If step 3 fails, you do not have a logging pipeline — you have a logging intention. Run the test after every agent upgrade and every backend change, and monitor the agent continuously in between; silent gaps are this system's natural failure mode.

Common mistakes

• Applications logging to files inside containers. The data is invisible to the platform, dies with the pod, and fills the container filesystem on the way out.
• Unstructured prose logs. Every incident becomes a regex-writing contest. JSON lines with stable fields, from day one.
• High-cardinality labels. Request IDs, user IDs, and pod IPs as index labels: the backend slows, costs climb, and the fix (re-indexing your mental model) is painful. Labels identify the stream; fields describe the event.
• No one owns agent health. The DaemonSet falls behind or dies on two nodes; the gap is discovered mid-incident, which is the only time it cannot be fixed.
• Debug level in production, forever, because one incident needed it once. Verbosity is a per-workload dial with a cost meter attached.
• Secrets in logs. The widest-read storage in your stack is the worst place for the most sensitive strings you own.
• Retention set by the tool's default instead of by your actual compliance and debugging needs — too short loses investigations, too long buys nothing but storage bills.

Where this fits in the series

Logging is the third pillar the observability guide stands on, and the raw material the troubleshooting guide consumes first. It leans on the storage guide's disk discipline, the requests-and-limits guide's resource accounting, the network policies guide's protection for the shipping path, and the security-basics guide's audit and redaction rules. Get it right and it disappears — every other guide in this series quietly assumes the logs were there when needed.

FAQ

Fluent Bit or Fluentd? For a new cluster pipeline, Fluent Bit: lighter, faster, and the current centre of gravity for Kubernetes log collection. Fluentd remains reasonable where its plugin ecosystem or an existing deployment carries weight. The architecture matters more than the choice.

Loki or Elasticsearch? If your queries are service-and-time-window shaped (most incident work), Loki's label model is dramatically cheaper to run. If you need deep, arbitrary full-text analytics over long retention, Elasticsearch/OpenSearch earns its cost. Many clusters start on their cloud provider's sink and graduate to one of these when query needs or bills demand it.

Should the application ever ship logs directly to the backend? Almost never. It couples application deploys to logging infrastructure, duplicates shipping logic in every service, and loses the node agent's uniform metadata and buffering. The stdout contract exists precisely to avoid this.

How long should logs be retained? Long enough to cover your slowest investigation and your compliance floor — commonly 2–4 weeks hot and a few months in cheap archive for application logs, longer for audit logs under policy. Decide per class of log, write it down, and let the tiers enforce it automatically.

Do managed services (EKS, GKE, AKS) solve this for me? They provide a destination (CloudWatch, Cloud Logging, Azure Monitor) and sometimes an agent option, but the architecture decisions remain yours: stdout discipline, metadata, correlation IDs, retention, and cost control. The provider removes the backend operations, not the design.

TL;DR: If your GitHub Actions workflows authenticate to AWS with a stored access key, that key is a permanent credential sitting in a second system, waiting to leak and rarely rotated. OIDC federation removes it entirely: each workflow run gets a short-lived token from GitHub, presents it to AWS, and assumes an IAM role scoped to exactly that repository, branch, or environment. Credentials live for about an hour and are never stored anywhere. Setup is one IAM identity provider, one role with a careful trust policy, and two lines in the workflow. This guide walks the whole thing, including the scoping mistakes that quietly undo the benefit.

Originally published on DevOps Enginer.

The problem: a permanent key in someone else's house

The classic setup is familiar: create an IAM user for CI, generate an access key pair, paste them into GitHub repository secrets, and reference them in the workflow. It works on day one. Then the years happen:

• The key never rotates, because rotating it means touching every repo that copied it — and nobody remembers all of them.
• The key outlives its purpose: the project is archived, the repo is forked into a test, the contractor who set it up left. The credential remains valid through all of it.
• The key's permissions drift upward: it was scoped for one deployment, then someone needed one more action, then it quietly became near-admin because that was faster.
• The key leaks like a secret, not like a session: pasted into a log, a debug dump, a screenshot. When a session token leaks you have an hour of exposure; when this leaks you have until someone notices and rotates — if anyone notices.

The deeper issue is architectural: a long-lived credential is a standing invitation. OIDC federation deletes the standing part. There is nothing to steal from the repository because the repository holds nothing — the workflow proves who it is at runtime instead.

How it works, in one paragraph

GitHub runs an OIDC identity provider. Every workflow run can request a signed JSON Web Token whose claims describe that run: the repository (subject claims like repo:my-org/my-repo:ref:refs/heads/main), the branch or tag, the environment, the workflow name. In AWS, you register GitHub as an IAM OIDC provider once per account, then create an IAM role whose trust policy says: "tokens from GitHub's provider may assume me — but only if the claims match these exact conditions." The workflow exchanges its token via AssumeRoleWithWebIdentity and receives ordinary short-lived AWS credentials (default lifetime one hour). AWS validates the token's signature against GitHub's published keys and checks the audience claim (sts.amazonaws.com) and your conditions. No shared secret exists on either side.

Step 1: register GitHub as an OIDC provider in IAM

One time per AWS account (Console: IAM → Identity providers → Add provider → OpenID Connect):

• Provider URL: https://token.actions.githubusercontent.com
• Audience: sts.amazonaws.com

That is the entire provider setup. Everything else lives on the role.

Step 2: the role and its trust policy — where the security actually is

Create an IAM role for web identity federation. The trust policy is the security boundary of this whole design, so write it deliberately:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

What each part earns:

• The aud condition confirms the token was minted for AWS STS, not for some other service that also trusts GitHub tokens.
• The sub condition is the scoping decision. repo:my-org/my-repo:ref:refs/heads/main means: only runs of this repo, on the main branch. Alternatives you will actually use:
  • repo:my-org/my-repo:environment:production — only runs that deploy to the GitHub environment named production (which can itself require manual approval — a strong pattern for prod).
  • repo:my-org/my-repo:ref:refs/tags/v* with StringLike — only tag pushes, for release roles.
• Never a wildcard like repo:my-org/* on a powerful role, and never omit the sub condition. "Any workflow in any repo I own can assume the production deploy role" is the stored-key problem wearing a new coat.

Then attach the role's permissions policy — the least-privilege part. Scope it to what the pipeline does: update one ECS service, write to one S3 bucket, publish to one ECR repository. The AWS IAM least-privilege guide on this site covers how to build such policies without guessing; the OIDC work here assumes that discipline rather than replacing it.

One role per (repository, environment) pair is the pattern that scales: gha-myrepo-prod, gha-myrepo-staging, each boringly narrow.

Step 3: the workflow side

Two requirements, both small:

name: deploy

on:
  push:
    branches: [main]

permissions:
  id-token: write      # without this, no token is issued
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-myrepo-prod
          aws-region: ap-northeast-1

      - name: Deploy
        run: |
          aws s3 sync ./dist s3://my-prod-site-bucket --delete

Notes that save debugging time:

• permissions: id-token: write is the step everyone forgets. Without it the token request fails before AWS is even involved. Setting any permissions block also means everything not listed drops to none — hence contents: read for checkout.
• There are no secrets to configure: AWS_ACCESS_KEY_ID and friends disappear from the repo settings entirely.
• The configure action exports the short-lived credentials to subsequent steps; tools that read the standard AWS credential chain (the AWS CLI, Terraform, SDKs) work unchanged.

Scoping patterns that matter

Environments are the strongest lever. GitHub environments can require a human approval before a job runs, and the token's claims carry the environment name. Combine them: the production role trusts only environment:production, so even a maintainer pushing to main cannot reach prod credentials until the environment's approval gate passes. Branch protection plus environment approval plus a scoped role is about as strong as CI identity gets without extra tooling.

Pull requests are a different animal. A pull_request run's claims do not look like a branch push — and for forks, the workflow runs in a context you should treat as untrusted. Practical rule: give PR workflows a read-only role if they need AWS at all (fetching artifacts, running plan), and never let a PR-triggered run assume a role that can write to production. terraform plan on PRs with a read-only role, apply only from main with the production role, is the canonical split — it pairs naturally with the Terraform state setup from this site's state-management guide.

Multiple accounts, one pattern. Staging and production in separate AWS accounts is common; OIDC handles it cleanly because the provider registration and role live per account. The same workflow can assume the staging role in the staging account and the production role in the production account, each trust policy scoped to its own environment claim.

Reusable workflows inherit the caller's identity carefully. If you use reusable workflows (this site's guide covers them), the token claims describe the caller repository and context — which is what you want, but it means a shared deploy workflow used by ten repos needs trust-policy entries or separate roles per caller repo. Plan for that instead of discovering it as a wildcard temptation.

Migrating off stored keys without breaking Friday's deploy

1. Inventory. Search your repos' secrets and workflows for AWS_ACCESS_KEY_ID. Note which IAM user each key belongs to and what it can do — this list is usually longer and scarier than expected, which is the argument for the project.
2. Create the provider and roles, one repo/environment at a time, permissions mirroring what the key was supposed to do (not what it accidentally can).
3. Switch the workflow to the OIDC pattern and deploy. Verify in CloudTrail that calls now arrive as the assumed role session, not the old user.
4. Watch, then delete. Leave the old key deactivated (not deleted) for a short window while you confirm nothing else uses it — scheduled jobs and forgotten repos surface quickly when a key stops working. Then delete the key, and if the CI IAM user has no remaining purpose, delete the user.
5. Sweep the leftovers: organisation-level secrets, third-party tools holding the same keys, and local copies in developer tooling.

Order matters less than completeness: a migration that moves nine repos and forgets the tenth has kept exactly the risk it meant to remove.

Common mistakes

• Wildcard sub conditions. repo:my-org/* or a missing sub check turns one role into every repo's master key. The trust policy is the product; write it narrowly.
• One role for everything. A single powerful role assumed by every workflow recreates the shared-key blast radius with better marketing. Role per repo per environment.
• Forgetting id-token: write. The failure appears before AWS is contacted and confuses everyone who starts debugging in IAM.
• Leaving the old keys active "just in case". The fallback is the vulnerability. Deactivate, watch, delete.
• Granting write roles to pull-request contexts. Plan on PRs, apply from protected branches and environments — never both from the same trigger.
• Skipping the audience check. The aud condition costs one line and prevents tokens minted for other purposes from being replayed at STS.
• Assuming OIDC fixes permissions. Federation decides who may assume the role, not what the role may do. An admin role via OIDC is still an admin role; least privilege is a separate, continuing discipline.

FAQ

Does this cost anything? No. OIDC providers and roles are free IAM features, and STS calls for this pattern are not a line item you will ever notice.

What if a workflow needs to run longer than the token lifetime? Role sessions default to one hour and can be configured up to the role's maximum session duration (up to 12 hours). Design for re-assumption or shorter jobs instead of reaching for the maximum by default — long-lived sessions quietly reintroduce the exposure you removed.

Can self-hosted runners use this? Yes — runners you host still request the token from GitHub's OIDC endpoint at runtime, so the same trust policy works. (GitHub Enterprise Server deployments use their own provider URL; check your instance's docs for its exact value.)

Does Terraform in GitHub Actions work with OIDC credentials? Yes. The configure-aws-credentials step exports standard environment credentials, and the Terraform AWS provider consumes them through its normal credential chain — no provider block changes needed. Remote state locking and the backend setup stay exactly as they were.

What about other clouds? The same pattern exists across the major providers — Azure and Google Cloud both accept GitHub-issued OIDC tokens for workload identity federation with no stored secrets. Once a team learns the shape on one cloud, the others are variations, and the destination of all of them is the same: CI with no permanent keys anywhere.

TL;DR: Kubernetes constantly disrupts your pods on purpose — node drains, cluster upgrades, autoscaler scale-downs, rollouts. Pod Disruption Budgets (PDBs) are how you negotiate those disruptions: "you may take one of my three replicas at a time, never two." But a PDB only covers the eviction handshake. The other half of graceful shutdown is the pod itself: handling SIGTERM, finishing in-flight work, and leaving the load balancer before the SIGKILL lands. Get both halves right and maintenance stops being an outage source; get either wrong and every upgrade is a small incident.

Originally published on DevOps Enginer.

The problem: most pod deaths are voluntary

When people imagine pods dying, they imagine failures — a node dying, an OOM kill, a crash. Those are involuntary disruptions, and Kubernetes handles them the only way it can: after the fact, by starting replacements.

But watch a healthy cluster for a week and most pod terminations are voluntary: an admin drains a node for maintenance, the cluster autoscaler removes an underused node, an upgrade rolls the nodes, a deployment replaces pods, the cluster itself gets upgraded version by version (the upgrades guide in this series is essentially a long, careful series of voluntary disruptions). Nothing failed — the platform is doing its job.

The trouble is the intersection: a drain that evicts all three replicas of your API in the same minute because nobody told it not to. A pod killed mid-request because it never got the memo — or got it and ignored it. Voluntary disruptions are scheduled, polite, and negotiable. You just have to actually negotiate.

Pod Disruption Budgets: the negotiation

A PDB is a small object that tells the eviction API how many pods of a workload may be voluntarily taken down at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: shop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

This says: of the pods labelled app: api in shop, at least 2 must stay available; evictions that would break that are refused with an error, and the drain waits. Two ways to express the budget:

• minAvailable — an absolute number (2) or a percentage (67%) of desired replicas that must remain up.
• maxUnavailable — how many may be down at once (1, or 34%). Usually the better choice, because it tracks replica count: if you scale from 3 to 6 replicas, "at most 1 down" still means what you meant, while a stale minAvailable: 2 silently got weaker.

Rounding is asymmetric and worth knowing: percentages for minAvailable round up (67% of 3 = 3, not 2), and for maxUnavailable round down (34% of 3 = 1). If your PDB seems stricter than the percentage suggests, this is why.

How the handshake works in practice: kubectl drain, cluster upgrades, and well-behaved controllers evict pods through the eviction API, which checks the PDB first. If the budget allows it, the pod gets its termination grace period and shuts down; if not, the eviction is rejected and the tool retries later. Tools that delete pods directly (a raw kubectl delete pod) bypass PDBs entirely — a PDB is a contract with polite tooling, not a force field.

The trap: a PDB that can never be satisfied

Set minAvailable equal to your replica count — a very common "extra safe" instinct — and you have built a deadlock, not a safeguard. The node can never drain, because draining requires evicting a pod, and the budget forbids it. Cluster upgrades stall, autoscaler scale-downs stall, and someone eventually force-deletes the pod at the worst possible time, defeating the entire exercise under pressure.

The subtler version: the budget is satisfiable on paper, but the workload is unhealthy. If only 1 of 3 replicas is ready, a minAvailable: 2 PDB blocks all evictions — including evicting the broken pods — because the budget counts healthy pods, and there is no slack. Recent Kubernetes versions address this with spec.unhealthyPodEvictionPolicy (set it to AlwaysAllow so unhealthy pods can always be evicted; the default IfHealthyBudget is stricter). A blocked drain is information: it usually means the application was already below its own availability floor before maintenance started.

A working rule: budget for one disruption at a time. maxUnavailable: 1 for replicated stateless services is the sensible default — drains proceed pod by pod, replacements come up before the next eviction is allowed, and upgrades stay boring.

The other half: what the pod does when told to die

A PDB buys an orderly sequence. It does not make an individual shutdown graceful. That part happens inside the pod, on a clock:

1. The pod is marked Terminating. It is removed from Service endpoints — mostly. Endpoint removal and the kube-proxy / ingress updates that follow are fast but not instantaneous, so a terminating pod can receive a trickle of new connections for a short window. This race is normal and known.
2. Any preStop lifecycle hook runs.
3. The container receives SIGTERM. This is the application's cue: stop accepting new work, finish in-flight requests, flush and close.
4. After terminationGracePeriodSeconds (default 30), if the process is still alive, it receives SIGKILL — no negotiation, mid-whatever.

Graceful shutdown is an application feature, not a platform one. The checklist:

• Handle SIGTERM in the process. Many frameworks do this out of the box now (Spring Boot's graceful shutdown, Go's signal.NotifyContext, Node's server close on signal) — but verify yours does, and that the handler drains: stop taking requests, wait for in-flight ones, close database connections cleanly.
• Size the grace period to the work. A web API that finishes requests in 2 seconds is fine at 30s. A worker with 5-minute jobs at the default 30s grace period gets SIGKILLed mid-job every rollout — set terminationGracePeriodSeconds to match the longest unit of work you intend to finish, and make the work checkpoint or idempotent for the cases where it still gets cut.
• The preStop sleep, honestly labelled. A common pattern is a preStop hook that sleeps a few seconds, giving endpoint removal time to propagate before SIGTERM makes the app stop listening. It looks like a hack because it is one — but it addresses the real endpoint race in step 1, and it prevents the classic "502s during every deploy" bug. Use it knowingly, keep it short.
• Workers and queues: on SIGTERM, stop consuming, finish the current message, then exit — and make sure unprocessed messages return to the queue (visibility timeout / ack discipline) rather than dying with the pod.

Stateful and single-replica workloads: where PDBs can't save you

PDBs assume redundancy. Apply the arithmetic to a single-replica workload and minAvailable: 1 (or maxUnavailable: 0) means "never voluntarily evict this pod, ever" — the drain deadlocks on your one database pod, and the eventual forced eviction takes the downtime anyway, unscheduled. For singletons, the honest choices are:

• Accept the disruption window: no PDB (or maxUnavailable: 1), maintenance takes the pod down briefly, planned and announced. Better a scheduled 60-second blip than a blocked upgrade escalating into an unplanned one.
• Make it not a singleton if the availability requirement is real — which for databases means the database's own replication story, not a Kubernetes trick.

For StatefulSets, PDBs work normally (they select pods by label like anything else), but pair them with the storage and backup thinking from earlier in this series: an evicted stateful pod comes back to its own volume, so the shutdown risk is about in-flight writes — the application-level SIGTERM handling matters more, not less.

One more placement rule: do not put PDBs on batch Jobs. A Job's pods are supposed to terminate; a budget on them protects nothing and can wedge completions and node drains alike. PDBs are for long-running services whose absence a user would notice.

Proving it: test the drain before the upgrade does

Like every control in this series, a PDB is a rumour until exercised. The test fits in a staging afternoon:

1. Run a light, continuous request loop against the service (a small script recording status codes is enough).
2. Cordon and drain a node running the workload: kubectl cordon <node> then kubectl drain <node> --ignore-daemonsets.
3. Watch: evictions should proceed one at a time, the replacement should be Ready before the next eviction is allowed, and the request loop should show zero — or near-zero — failed requests. Failed requests with graceful PDB behaviour usually point at the application half: SIGTERM ignored, grace period too short, or the endpoint race.
4. Then test the blocked path deliberately: scale the service down to its floor and try another eviction. Being refused, and seeing the clear PDB error, is the control working. An upgrade night is the wrong time to learn this behaviour for the first time.

The cluster autoscaler respects the same handshake when it scales nodes down (covered in the autoscaling guide in this series) — which is why a too-strict PDB shows up twice in a cluster's life: once blocking your upgrade, once blocking your cost savings.

Common mistakes

• minAvailable equal to the replica count. The deadlock special. Budget one disruption at a time.
• Percentages that don't mean what you think. minAvailable: 50% of 3 replicas is 2 pods (it rounds up) — and of 1 replica it is a permanent eviction ban.
• Expecting PDBs to cover failures. They govern voluntary evictions only. Node death, crashes and OOM kills are answered by replicas, topology spread and anti-affinity — the deployment-strategies guide covers that side.
• Default 30-second grace for slow work. Long requests, report generation, queue jobs: if the work takes minutes, the grace period must too, or every rollout is a small data-loss event.
• Ignoring SIGTERM "because Kubernetes restarts us anyway". It does — elsewhere. The in-flight request that died with the old pod does not get restarted; the client just sees an error.
• PDBs on everything, including Jobs and singletons. Indiscriminate budgets create blocked operations across the cluster and train the team to force-delete — the worst possible outcome.
• Never alerting on blocked evictions. A drain stuck for hours is a signal the app is below its availability floor. It belongs in the observability stack (the observability guide), not in tomorrow's surprise.

Where this fits in the series

PDBs are the politeness protocol that several earlier guides quietly assume: the upgrades guide's node-by-node dance only works when evictions are budgeted, the autoscaling guide's cluster autoscaler asks PDB permission before removing nodes, Argo Rollouts and the deployment-strategies guide manage the same risk from the release side, and the backup guide is the admission that even perfect politeness has limits. Voluntary disruption done well is invisible — which is precisely the point of it.

FAQ

Does a Pod Disruption Budget protect against node failure? No. PDBs only constrain voluntary evictions through the eviction API. Involuntary loss — node death, kernel panic, a cloud reclaiming the instance — ignores budgets entirely. Redundancy (replicas, topology spread constraints, anti-affinity) is the tool for that side.

Should I use minAvailable or maxUnavailable? maxUnavailable in most cases, typically 1. It states the disruption you tolerate rather than a floor that silently weakens or deadlocks as replica counts change. Reserve minAvailable for workloads whose floor is a hard contractual number.

Can a PDB block an upgrade forever? Yes, if the budget can never be met — either by construction (floor equals replicas) or because the workload is too unhealthy to have slack. That is the PDB doing its stated job; the fix is to repair the workload's health or consciously relax the budget, not to force-delete pods under pressure.

Do Deployments create PDBs automatically? No. A Deployment manages replicas; the budget is a separate object you write, select, and own. Some Helm charts ship one (check before assuming — the value may not match your replica count), and the Helm guide in this series shows where such templates live.

What happens to in-flight requests when a pod is evicted? The pod leaves the Service endpoints, gets SIGTERM, and has its grace period to finish. Requests still in flight when the grace period expires die with the SIGKILL. Both halves of this guide exist to shrink that window to zero: PDBs sequence the evictions, and the application's shutdown handling empties the pod before the deadline.

TL;DR: High availability is not backup. A three-replica Deployment survives a node failure, but kubectl delete namespace deletes all three replicas just as happily, and no amount of HA brings back a dropped database. Real protection has three layers — cluster resources, persistent volume data, and the things that live outside the cluster — plus a restore you have actually performed and timed. Velero is the standard open-source tool for the first two layers. The work is not installing it; the work is proving, on a schedule, that a restore produces a working application.

Originally published on DevOps Enginer.

The problem: Kubernetes feels safer than it is

Kubernetes heals. Kill a pod and it comes back. Lose a node and workloads reschedule. This self-healing is real — and it quietly convinces teams they are protected against data loss, which is a different thing entirely.

Self-healing protects against failures of the machinery. Backups protect against everything else: an engineer deleting the wrong namespace, a bad migration dropping a table, a compromised account wiping resources, a storage failure corrupting a volume, a cloud account problem taking the whole region's control plane with it. In every one of those cases, the cluster faithfully heals itself into the new, broken state. The disaster is not that Kubernetes stops working — it is that it keeps working, on empty data.

The second misconception: "etcd is backed up, so we are covered." On a self-managed cluster, an etcd snapshot captures cluster state — the resource definitions. It does not contain the bytes inside your PersistentVolumes, and restoring it is a cluster surgery procedure, not an application recovery. On managed services (EKS, GKE, AKS) you cannot snapshot etcd at all — the provider runs it — which settles the question: your backup strategy lives at the resource and volume level whether you chose it or not.

What actually needs protecting: three layers

1. Cluster resources. Deployments, Services, ConfigMaps, Secrets, RBAC, NetworkPolicies — the declarative state. Cheap to back up (it is text in etcd), and the layer GitOps already half-covers: if everything is applied from Git, Git is a backup of intent, though not of generated state like Secrets created by operators or resources created by controllers.
2. Persistent data. The contents of PersistentVolumes: databases, uploaded files, queues on disk. This is the layer that hurts to lose and the layer most "we ran a backup tool once" setups quietly miss, because resource backup and volume backup are separate switches.
3. Outside the cluster. Container images (a registry that garbage-collects the tag you need is a DR failure), external databases and queues, DNS, certificates, and the cloud resources the cluster depends on — IAM roles, load balancer config, the object storage bucket your backups land in. A restore plan that discovers a missing layer at 3 a.m. is not a plan.

Backups also need their own threat model: they must survive the loss of the cluster and the loss of the account or credentials that administer it. Backup data stored only in the same account, deletable by the same compromised credential, is a copy, not a backup. Cross-account or cross-region copies with separate retention are the grown-up version.

Two numbers that decide the design: RPO and RTO

Before tooling, two definitions do all the real work:

• RPO (recovery point objective): how much data you can afford to lose, measured in time. Hourly backups mean up to an hour of data loss. If the business answer is "five minutes", nightly volume snapshots are the wrong tool no matter how reliable they are — you need replication or database-native continuous backup.
• RTO (recovery time objective): how long the business tolerates being down while you restore. This is the number restore tests measure, and it includes everything: provisioning a cluster, restoring resources, restoring data, DNS, verification. Teams routinely discover their restore takes six hours against a one-hour RTO — better discovered in a drill than in an incident.

Write both numbers down per application tier. They turn backup arguments ("should we also snapshot the volumes?") into arithmetic.

Layer 1 + 2 with Velero: the working setup

Velero is the standard open-source tool for Kubernetes backup: it copies cluster resources to object storage and, when asked, snapshots or copies the data inside PersistentVolumes. Two facts shape every Velero setup: backups land in object storage outside the cluster (S3, GCS, Azure Blob, or S3-compatible), and volume data needs an explicit choice — it is not automatic.

Install against your provider's plugin (AWS shown; bucket and credentials are placeholders):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws \
  --bucket my-cluster-backups \
  --secret-file ./credentials-velero \
  --backup-location-config region=ap-northeast-1 \
  --use-node-agent

--use-node-agent deploys the agent that performs file-system-level volume backups — needed for the second volume option below.

There are two ways Velero captures volume data:

• CSI snapshots — the storage driver snapshots the volume (on EKS, an EBS snapshot). Fast, storage-native, and the right default where the CSI driver supports snapshots. Since Velero 1.15 the CSI integration is built in rather than a separate plugin.
• File-system backup — Velero copies the files out of the volume through the node agent (Kopia under the hood in current releases). Slower, but works on storage with no snapshot support and produces data that lives in the same object storage as the resource backup, which simplifies cross-cluster restores.

A scheduled backup of one namespace, with volume snapshots, is one object:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: shop-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - shop
    snapshotVolumes: true
    ttl: 720h

And the manual equivalents you will use in drills:

velero backup create shop-drill --include-namespaces shop --snapshot-volumes --wait
velero restore create --from-backup shop-drill --namespace-mappings shop:shop-restore
velero backup describe shop-drill      # check phase: Completed, and read the warnings

Restoring into a different namespace (the --namespace-mappings above) is the drill pattern: the original workload keeps running while you prove the copy works.

The consistency problem: volumes are crash copies

A volume snapshot taken under a running database is a crash-consistent copy — equivalent to pulling the plug. Most databases recover from that on startup, which is why the pattern usually works, but "usually" is doing heavy lifting. For the workloads where consistency matters, you have three honest options:

• Backup hooks. Velero can run a command in a pod before or after the snapshot (annotations on the pod template) — the classic pattern is freezing or flushing the database (for example fsfreeze or a database-specific lock) for the seconds the snapshot takes.
• Database-native backup. For a critical PostgreSQL or MySQL, the serious answer is often the database's own tooling — WAL archiving / point-in-time recovery — with Velero protecting the manifests and configuration around it. Point-in-time recovery also beats any snapshot schedule on RPO.
• Quiesce the workload. Scale to zero, snapshot, scale back. Only acceptable where a maintenance window exists — but for small internal systems it is the simplest correct answer.

Pick per workload, and write the choice down next to the RPO it achieves. The failure mode is assuming the default (crash-consistent) covers the database it does not.

The GitOps shortcut, and its limits

If your manifests live in Git and Argo CD (or equivalent) applies them, layer 1 mostly restores itself: point a new cluster at the repo and the workloads reappear. This is a genuine DR advantage of GitOps — the Argo CD guide in this series covers the setup. But know exactly what Git does not contain: Secrets (usually stored encrypted or externally — your restore needs the key material, e.g. the Sealed Secrets private key, or the external secret store), data created inside the cluster, and resources generated by operators. The standard production answer is both: GitOps for the declarative layer, Velero for the generated state and the volume data.

Prove it: restore drills are the actual product

Every guide in this series lands on the same principle — an untested control is a rumour — and backups are where it matters most, because backup failures are silent. Velero will happily report schedules running while the bucket policy expired months ago. The only verification that counts is a restore.

A workable drill rhythm:

1. Monthly: restore last night's backup of one real namespace into a scratch namespace on the same cluster. Verify the application answers, data row counts match expectations, and record how long it took — that number is your honest RTO for that tier.
2. Quarterly: restore into a different cluster (the migration path). This catches the things intra-cluster restores never touch: missing CRDs, storage class names that differ, image pull secrets, ingress differences.
3. Always: alert on backup failure, not just on schedule age. Wire backup status into the same observability stack as everything else — the observability guide in this series is where the backup metrics and alerts belong.

Notice what the drill produces beyond safety: a measured restore time, a list of missing pieces fixed while nothing was on fire, and a team that has performed the procedure before the night it counts.

Common mistakes

• Backing up resources and calling volumes done. snapshotVolumes off (or no file-system backup configured) means your backup is the recipe, not the meal. Check a backup's contents, not just its green status.
• Backups in the blast radius. Same account, same credentials, no versioning or retention lock on the bucket: one compromise deletes production and its backups in the same session.
• No TTL / no cost control. Snapshots accumulate forever; the storage guide in this series covers the orphaned-snapshot problem from the cost side. Every schedule needs a ttl.
• Secrets that cannot be restored. The backup contains the Sealed Secrets resources but the cluster that sealed them is the one that died. Keep the unsealing keys in the DR plan, stored outside the cluster.
• Restoring everything at once. A full-cluster restore recreates namespaces in dependency order whether you plan it or not — and load balancers, ingress and DNS fight you on the way. Restore per-application, most critical first; that ordering is the RTO plan.
• One backup tool for the database it cannot help. Crash-consistent volume copies of a busy database, with no hooks and no native backup, presented as a DR strategy.
• Testing backups by reading their status. Green "Completed" is Velero's opinion. The restore is the test.

Where this fits in the series

Backup is the layer underneath several earlier guides: the storage guide explains the volumes being protected, the upgrades guide's blue/green cluster pattern is a restore-shaped procedure (and the quarterly cross-cluster drill doubles as upgrade rehearsal), the observability guide is where backup success and failure get monitored, and the network policies and security-basics guides cover the access control that keeps backup credentials and buckets out of an attacker's reach. Recovery is not a feature you add at the end; it is the property the rest of the stack is for.

FAQ

Does Velero back up etcd? No — deliberately. Velero works at the API level, copying resources and volume data; etcd snapshots are a separate, control-plane procedure for self-managed clusters. Velero's model also restores into a cluster running a different Kubernetes version, which an etcd restore cannot do.

We run GitOps. Do we still need Velero? Yes, for the parts Git cannot see: volume data, generated resources, and in-cluster state. GitOps shrinks the restore problem; it does not remove it.

How often should backups run? As often as your RPO demands, minus honesty about restore cost: a common shape is daily schedules with a 30-day TTL for general namespaces, more frequent (or database-native continuous backup) for the tier whose RPO is measured in minutes. More schedules than that mostly buy storage cost.

Is a second cluster in another region enough — active-active instead of backup? Replication solves some RPO problems and no deletion problems: a replicated kubectl delete is a deletion in two regions. Multi-cluster setups still need backups, and most teams should get backups plus practiced restores working before paying for a second always-on cluster.

Can Velero migrate between clusters? Yes — backup on one cluster, restore on another is a first-class workflow, and the standard way to move workloads during a cluster rebuild or the blue/green upgrade pattern. That dual use is another reason the quarterly drill pays off: it rehearses two procedures at once.

TL;DR: By default, every pod in Kubernetes can talk to every other pod — a flat network with no internal walls. Network Policies are the built-in way to change that: declare which pods may talk to which, on which ports, and everything else is denied. Start with a default-deny policy per namespace, add back exactly the flows your app needs (including DNS — everyone forgets DNS), and test with real connection attempts, not hope. One catch: your CNI plugin must actually enforce policies, or you are writing rules nobody obeys.

Originally published on DevOps Enginer.

The problem: a flat network inside the cluster

Kubernetes networking guarantees that pods can reach each other. What it does not give you, out of the box, is any restriction on that reachability. A compromised frontend pod can scan the database. A batch job can reach the payment service. A pod in the dev namespace can talk to a pod in prod if they share a cluster.

Network Policies fix this at the platform layer. They are firewall rules expressed as Kubernetes objects: selectors pick the pods a policy applies to, and rules list the allowed sources (ingress), destinations (egress), and ports. Anything not explicitly allowed is denied — but only once at least one policy selects a pod. A pod with no policy selecting it remains wide open, which is the single most misunderstood part of the whole feature.

First, the prerequisite nobody checks: your CNI

NetworkPolicy is an API, not an implementation. The enforcement is done by your CNI plugin or network stack. Calico and Cilium enforce policies fully. Flannel, historically, does not enforce them at all — you can apply perfect policies and traffic flows anyway. On managed services it varies: Amazon's VPC CNI supports Network Policy enforcement in current EKS versions, GKE offers Dataplane V2 or Calico-backed policy, and AKS has Azure CNI powered by Cilium or its own policy manager options.

Before writing a single rule, verify enforcement in a scratch namespace: apply a default-deny, try a connection that should fail, and watch it fail. A policy that has never been tested is a rumour, not a control. (The security-basics guide in this series covers where Network Policies sit among the other cluster controls.)

The anatomy of a policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432

Read it as a sentence: Pods labelled app: api in namespace shop may accept TCP 8080 from pods labelled app: web, and may initiate TCP 5432 to pods labelled app: postgres. Everything else for these pods is denied.

Details that matter:

• Empty podSelector: {} selects every pod in the namespace — that is how you write a namespace-wide default-deny.
• policyTypes is a promise. If you list Egress but write no egress rules, all egress is denied. If you omit Egress from policyTypes while writing an ingress-only policy, egress stays open.
• from/to entries in one list item are OR'd; separate list items are also OR'd. Within one item, a podSelector and namespaceSelector together are AND'd — the classic way to say "pods with this label, in namespaces with that label."
• Rules are additive only. There is no deny rule and no ordering. Policies from multiple objects union together, which is why the default-deny-plus-allows pattern works.
• Policies are namespaced. A policy in shop cannot select pods in another namespace directly; cross-namespace rules go through namespaceSelector.

Step 1: default-deny per namespace

The baseline move in every namespace that runs real workloads:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Apply this and the namespace goes dark: nothing in, nothing out. You have not secured the app yet — you have made the allowed list explicit and empty. Now you earn each flow back deliberately. Do this in a staging namespace first, because the next thing that happens is always the same.

Step 2: allow DNS, the flow everyone forgets

The first casualty of default-deny egress is DNS. Pods resolve names through the cluster DNS service (kube-dns/CoreDNS in kube-system), and if egress to it is not allowed, every lookup fails and the app errors look like five unrelated bugs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Two subtleties: when a rule item contains both a namespaceSelector and a podSelector, they combine with AND — exactly what you want here. And allow TCP 53 as well as UDP 53; large responses and retries fall back to TCP, and "DNS mostly works" is a miserable state to debug.

Step 3: allow the real flows, one hop at a time

Now walk your architecture diagram and convert each arrow into a rule. For a typical three-tier app in one namespace — web → api → postgres — the api-allow policy at the top of this guide is the middle hop. The web tier needs ingress from the ingress controller's namespace (via namespaceSelector) and egress to the api. The database needs ingress only from the api and should usually have no egress at all beyond DNS.

Work namespace by namespace, and keep the policies in Git next to the deployments they describe. If you run GitOps (the Argo CD guide in this series), a policy change then gets the same review trail as a code change — which is precisely the audit story you want for a security control.

Testing policies: prove the negative

The failure mode of Network Policies is silent — both kinds. Too open, and nothing complains while the risk stays. Too strict, and the app breaks in ways that look like application bugs. Test both directions deliberately:

1. Positive test: the allowed flow works (web reaches api, api reaches postgres, names resolve).
2. Negative test: a forbidden flow fails. Exec into a pod — or spin up a throwaway debug pod in the namespace — and attempt the connection that should not work: the database port from the web tier, an external IP, another namespace's service. A timeout or refusal is the policy doing its job; a successful connect means the policy is not selecting what you think it is, or the CNI is not enforcing.

A debug pod recipe that covers most cases:

kubectl run nettest --rm -it --image=nicolaka/netshoot -n shop -- bash
# inside: dig api.shop.svc.cluster.local, nc -vz postgres 5432, curl to the api

(netshoot is a well-known troubleshooting image that bundles the usual network tools.) Label the debug pod like the tier you are testing — policies select by labels, so an unlabelled debug pod tests the default, not the app's reality.

Common mistakes

• Applying policies where the CNI ignores them. Test enforcement first; a policy on Flannel-era setups is decoration.
• Forgetting policyTypes. If you write egress rules but do not list Egress in policyTypes, your egress rules are not applied the way you expect — the API defaults policyTypes based on what you wrote, and the surprises are all in the direction of "more open than intended" or "mysteriously closed." Be explicit.
• Denying DNS and chasing ghosts for an afternoon. Any default-deny egress rollout needs the DNS allow in the same change.
• Selecting by namespace name strings. kubernetes.io/metadata.name is the label to match for a namespace; hand-rolled name labels drift.
• Trying to write deny rules. There are none in the native API. Deny-by-exception designs belong to the CNI-specific policy types (Calico and Cilium both offer higher-level policy objects with ordering and explicit deny) — use them deliberately, not by accident.
• One giant policy per namespace. Small, per-tier policies named after the flow (api-allow, postgres-ingress-from-api) stay readable and reviewable. A 300-line policy is where mistakes hide.
• Ignoring egress. Ingress-only thinking leaves data exfiltration and command-and-control paths wide open. Locking egress to known destinations is the higher-value half of the work, and the half most teams skip.

Where this fits in the series

Network Policies are one layer, not the whole story. They sit alongside the controls from the security-basics guide (RBAC, Pod Security Standards, secrets handling from the ConfigMaps and Secrets guide), the namespace and quota discipline from the resource requests and limits guide, and the upgrade hygiene that keeps the CNI itself current. No single layer carries the cluster; the layers are the point.

FAQ

Do Network Policies replace a service mesh? No. Policies operate at L3/L4 — pods, namespaces, ports. A mesh adds identity-based, L7-aware controls (which service may call which path). Many clusters run policies alone happily; regulated or multi-tenant environments often run both.

Can a policy allow traffic from outside the cluster? Ingress rules accept ipBlock entries (CIDR ranges), which is how you allow a load balancer range or an office VPN. Prefer selecting the ingress controller's pods by label when the traffic enters through it — CIDRs of cloud load balancers change more often than labels do.

What about traffic between nodes, or to the node itself? Native policies govern pod traffic. Host and node-level rules are CNI-specific territory again (Calico's GlobalNetworkPolicy and Cilium's host policies) — reach for those when you need them, knowingly.

How do I roll this out on a live cluster without an outage? Namespace by namespace, staging first: apply default-deny where a copy of the workload runs, add the flows, run the positive/negative tests, then repeat in production during a quiet window with the Git revert one command away. The rollout is a series of small, reversible changes — treat it with the same respect as the upgrades in this series' cluster-upgrades guide and it stays boring, which is the goal.

TL;DR: EKS gives you a managed Kubernetes control plane and leaves everything else — nodes, networking shape, IAM wiring, add-ons — as your design decisions. The choices that matter: nodes come as managed node groups (the sensible default), self-managed groups (when you need control), or Fargate (serverless pods, with real constraints); Karpenter increasingly replaces static groups for teams that want nodes to follow demand. Networking runs on your VPC — pods get real VPC IPs from your subnets, which makes the VPC design a Kubernetes decision. IAM meets Kubernetes through Pod Identity (the current, simpler answer to "how does this pod get AWS permissions"), and the cluster is only production-shaped once the add-ons land: EBS CSI for storage, the Load Balancer Controller for ingress, external-dns and cert-manager for the public face. What EKS removes is the control plane burden. Everything in the rest of this series still applies — it just runs on AWS now.

Originally published on DevOps Enginer.

Most teams choosing EKS are not really choosing Kubernetes — they already know it. They are choosing to stop operating a control plane. What surprises them is how much remains theirs: EKS is managed Kubernetes, not managed everything, and the gap between "cluster created" and "cluster production-ready" is a series of decisions this guide walks through.

What AWS manages, and what it pointedly does not

AWS runs the control plane — API servers and etcd across multiple AZs, patched and highly available — and charges a flat per-cluster hourly fee for it. That is the entire managed surface. Nodes are your EC2 instances (or Fargate tasks), the VPC is yours, IAM is yours, and the add-ons that make a cluster useful are yours to install and maintain. Understanding this boundary prevents the most common EKS disappointment: expecting an app platform and receiving excellent infrastructure primitives.

Cluster creation itself (eksctl, Terraform, console) is the easy afternoon. The decisions below are the ones with consequences.

Nodes: three ways to run pods, and a fourth that changes the question

Managed node groups — AWS handles the EC2 lifecycle (provisioning, AMI updates, draining on termination) behind a group definition: instance type, size range, subnets, labels. This is the default recommendation: boring, supported, and sufficient for most clusters.

Self-managed node groups — you own the Auto Scaling group and the bootstrap. Justified by custom AMIs, unusual instance configuration, or requirements the managed path doesn't expose. You also own every patch and every upgrade chore — the upgrades guide applies with extra steps.

Fargate — no nodes at all: each pod runs in an isolated micro-VM with a per-pod price. The constraints are the fine print: no DaemonSets, no persistent volumes beyond EFS, pod sizes from a fixed menu, and startup latency that batch and CI workloads forgive and latency-sensitive services don't. Excellent for spiky, stateless, low-count workloads; a poor default for a whole cluster.

Karpenter — the open-source autoscaler from AWS that increasingly replaces static groups and the cluster-autoscaler pattern: it watches pending pods and launches right-sized instances (type, family, AZ chosen per workload) in under a minute, then consolidates under-used nodes away. For clusters whose demand moves — most clusters — Karpenter is where the autoscaling story lands on AWS, and its cost effect is the rare one that shows up on the bill uninvited, in the good direction.

Networking: your VPC is the cluster network

EKS uses the AWS VPC CNI by default, which means pods receive real IP addresses from your VPC subnets. The consequences are pleasant (pods are first-class network citizens; security groups can even attach per-pod) and constraining (pod density per node is limited by ENI/IP capacity, and subnet IP space is a real planning input — small subnets plus big clusters is a known way to run out of addresses). Prefix delegation raises pod density substantially where needed. This is why the subnet sizing in the VPC guide leaned generous: Kubernetes spends IP addresses like it means it.

The public face of the cluster is the AWS Load Balancer Controller: annotate an Ingress or Service and it provisions an ALB/NLB with target groups wired to your pods — the AWS-native version of the ingress patterns covered earlier. Add external-dns (Route 53 records follow your ingresses automatically) and cert-manager with ACM or Let's Encrypt, and "deploy an app, get a public HTTPS URL" becomes the paved road rather than a manual ritual.

IAM inside the cluster: Pod Identity is the current answer

Pods need AWS permissions — read this bucket, publish to that queue — and the cardinal rule is unchanged from everywhere else in AWS: per-workload roles, never node-wide credentials. A node role big enough for every pod on the node is a privilege-escalation welcome mat, per the IAM least-privilege guide.

EKS has had two mechanisms. IRSA (IAM Roles for Service Accounts) works via OIDC and annotations and remains everywhere in existing clusters. EKS Pod Identity is the newer, simpler successor: associate an IAM role with a service account through the EKS API, an agent on the node handles credentials, and the annotation archaeology of IRSA goes away. New clusters should start on Pod Identity; existing ones can run both while migrating. Either way, the design is identical: one role per workload, scoped to the resources that workload touches, with the cluster components (CSI driver, LB controller, Karpenter, external-dns) each holding their own deliberately-sized roles.

The add-ons that make it production-shaped

A fresh EKS cluster is deliberately minimal. The standard load-out, whether installed as EKS add-ons or via Helm/Argo CD:

• EBS CSI driver — without it, PersistentVolumes on EBS don't exist; the storage model (gp3 classes, Retain policies, snapshots) hangs off this driver on AWS.
• AWS Load Balancer Controller — ingress and services, as above.
• CoreDNS and kube-proxy — shipped as EKS add-ons; keep their versions in step with cluster upgrades rather than discovering skew during one.
• Metrics and logs — metrics-server or the Prometheus stack for the observability baseline, plus a log route (Fluent Bit to CloudWatch Logs or your aggregator). A cluster nobody can see into is a cluster on borrowed time.
• Cluster access — modern EKS uses access entries (API-based authentication mapping IAM principals to Kubernetes permissions), replacing the hand-edited aws-auth ConfigMap that caused a generation of "I locked everyone out of the cluster" incidents. Whoever provisions the cluster should set access policy deliberately on day one; the creator-admin trap (only the creating identity can administer, ever) is an old rite of passage best skipped.

Everything after installation is the standard Kubernetes operating model — GitOps delivery, progressive rollouts, security basics. EKS changes the substrate, not the discipline.

Upgrades and cost: the two recurring appointments

Upgrades follow the standard Kubernetes rhythm with AWS holding the control-plane end: you trigger the control-plane upgrade one minor version at a time, then roll node groups (or let Karpenter drift-replace nodes onto the new AMI), then bump add-ons. Support windows matter here: standard support for a Kubernetes version on EKS runs about 14 months from its EKS release, with paid extended support beyond — the calendar in the upgrades guide applies verbatim.

Cost has three components people chronically under-model: the per-cluster control-plane fee (multiplied by every forgotten dev cluster — see the Kubernetes cost guide for why cluster sprawl is the quiet line item), the nodes themselves (where Karpenter's right-sizing and consolidation do real work, plus Spot for fault-tolerant capacity), and the AWS perimeter — NAT gateways, load balancers and data transfer that the VPC design decided before the cluster existed.

Common mistakes

1. Expecting a platform. EKS is infrastructure with a managed control plane. Teams that budget zero platform work discover the add-on, IAM and networking decisions arriving anyway, unscheduled.
2. The node role that does everything. One broad role for the node group means every pod inherits it. Per-workload identity via Pod Identity is the entire fix, and it's cheap.
3. Subnets sized for the demo. Pods consume VPC IPs directly; small subnets cap cluster growth in a way that surfaces mid-incident. Size for the cluster you expect, with the VPC's spare range held back.
4. aws-auth archaeology. Hand-editing the access ConfigMap (on clusters still using it) locks teams out with one typo. Prefer access entries; if you must edit aws-auth, do it from a session that stays open until a second login is verified.
5. Add-ons drifting behind the cluster version. CSI and LB controller versions have compatibility windows; treat add-on upgrades as part of the cluster upgrade runbook, not a separate someday.
6. A cluster per team per environment, forever. The per-cluster fee plus duplicated add-ons and perimeter makes sprawl a budget line. Consolidate with namespaces and per-team IAM where isolation requirements genuinely allow it — and keep dedicated clusters where they don't (compliance, blast radius), knowingly.

FAQ

EKS vs self-managed Kubernetes on EC2? Self-managed gives total control and total responsibility: control-plane HA, etcd backups, and patching are yours. EKS removes exactly that layer for a flat fee. Teams choose self-managed for specific control or cost-at-scale reasons; everyone else is buying back their weekends with EKS, correctly.

EKS vs ECS Fargate? If the team knows Kubernetes or needs its ecosystem (Helm charts, operators, the tooling in this series), EKS. If the need is "run these containers on AWS with minimal concepts," ECS is simpler and entirely respectable. The honest test is which ecosystem your hiring, tooling and existing workloads already speak.

Can I run a cluster with no managed node groups at all? Yes — Fargate-only clusters exist and suit small, spiky, stateless estates. The DaemonSet and storage constraints decide for most teams; mixed setups (node groups for the baseline, Fargate for burst jobs) are common.

How many clusters should we have? Fewer than instinct suggests, more than one for anything serious: separate production from non-production at minimum. Beyond that, let blast radius, compliance boundaries and team autonomy argue it out — with the per-cluster and perimeter costs on the table while they do.

Conclusion

EKS is a trade stated plainly: AWS operates the control plane; you keep the architecture. Choose node strategy deliberately (managed groups by default, Karpenter when demand moves), design the VPC like the cluster depends on it — because it literally runs on it — wire IAM per workload with Pod Identity, and install the add-ons that turn a cluster into a platform. Do those things and EKS delivers what it promises: Kubernetes without the control-plane pager. The rest of running Kubernetes well is, as ever, on you — and it's what the other fifteen parts of this series are for.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the production-readiness items this guide builds toward — access, backups, observability and upgrade readiness.

TL;DR: A VPC is your own slice of AWS networking, and almost everything about it is decided by five choices: the CIDR block (bigger than you think you need, because it can never grow), the subnet layout (public for load balancers, private for workloads, isolated for data — spread across at least two, ideally three AZs), the route tables (a subnet is only "public" because its route table points at the internet gateway), NAT gateways (how private subnets reach the internet — one per AZ, because a shared NAT is both a bottleneck and a bill), and the two security layers (security groups, stateful and attached to instances; network ACLs, stateless and attached to subnets). Add VPC endpoints for AWS services before the NAT bill teaches you why, enable flow logs from day one, and never use the default VPC for anything you care about.

Originally published on DevOps Enginer.

Networking is the part of AWS everyone clicks through on the way to launching something — and the part that is nearly impossible to change later. A CIDR block cannot be resized. A subnet layout chosen in an afternoon shapes every architecture decision after it. This guide is the VPC as DevOps engineers actually need to understand it: the design decisions, in the order they become permanent.

The CIDR block: the one-way door

Your VPC gets an IPv4 range like 10.20.0.0/16 — 65,536 addresses. Choose it like it matters, because:

• It cannot be enlarged later. You can attach secondary CIDRs, but the clean answer is to start generous: a /16 per VPC is the common, sane default. Address space is free; running out is not.
• It must not collide with anything it will ever talk to — your other VPCs (peering and Transit Gateway demand non-overlapping ranges), your office network, your VPN ranges. Collisions discovered at integration time are miserable.
• Pick from private ranges everyone recognises (10.x is conventional) and keep a simple allocation register: which block belongs to which account/environment/region. A spreadsheet beats a memory, and it will be consulted at the worst possible moment if it doesn't exist.

Subnets carve the block per AZ and tier: a workable pattern is /20s per subnet (4,096 addresses each) — public, private and data tiers across three AZs consumes nine subnets and still leaves most of a /16 unallocated for the future. Remember AWS reserves five addresses in every subnet (network, router, DNS, future use, broadcast), so a subnet's usable count is its size minus five.

Subnets: three tiers, and what "public" actually means

The tier names describe routing, not intention:

• Public subnets — route table has a route to an Internet Gateway (IGW). For: load balancers, NAT gateways, bastions — the small set of things that face the internet. Nothing else earns a public IP.
• Private subnets — no IGW route; outbound internet goes via a NAT gateway in a public subnet. For: application servers, containers (your EKS/EC2 workloads), everything that should be reachable only through a load balancer.
• Isolated (data) subnets — no internet route at all, in or out. For: databases, caches, queues — resources that should only ever talk to the application tier inside the VPC.

The sentence worth internalising: a subnet is public or private solely because of its route table. There is no checkbox. Misunderstanding this is the root of most "why can/can't my instance reach the internet" mysteries — the answer is always in the route table, never in the subnet's name.

VPC 10.20.0.0/16
├── AZ a: public 10.20.0.0/20 · private 10.20.16.0/20 · data 10.20.32.0/20
├── AZ b: public 10.20.48.0/20 · private 10.20.64.0/20 · data 10.20.80.0/20
└── AZ c: public 10.20.96.0/20 · private 10.20.112.0/20 · data 10.20.128.0/20
    (remainder of the /16 left unallocated, on purpose)

Routing and NAT: where the money and the outages live

Route tables attach per subnet (with one main table as the default). The public table points 0.0.0.0/0 at the IGW; private tables point 0.0.0.0/0 at a NAT gateway; data tables have only the local route. Two operational truths follow:

NAT gateways are per-AZ infrastructure, not a shared utility. One NAT gateway costs a fixed hourly fee plus per-GB processing — and routing three AZs through one NAT in AZ-a means (a) an AZ-a failure takes outbound internet from the whole VPC, and (b) you pay cross-AZ data transfer for the privilege. One NAT per AZ is the production pattern; the cost guide explains the bill either way. Dev environments can share one NAT and accept the risk knowingly.

VPC endpoints are the other half of the traffic bill. Without them, your private subnets reach S3, DynamoDB and every other AWS API through the NAT gateway, paying per-GB for traffic that never needed to leave AWS's network. Gateway endpoints for S3 and DynamoDB are free — there is no excuse for their absence. Interface endpoints (PrivateLink) for the likes of ECR, CloudWatch Logs and Secrets Manager cost an hourly fee per AZ but keep agent and image-pull traffic fully private — on a busy EKS cluster, pulling images through NAT is a cost and failure-domain decision, not a default.

The two security layers (they are not interchangeable)

Security groups attach to instances/ENIs, are stateful (reply traffic is automatically allowed), support allow-rules only, and can reference other security groups — "the database accepts 5432 from the app tier's group" is the single most useful pattern in AWS networking, because it tracks membership automatically as instances come and go.

Network ACLs attach to subnets, are stateless (return traffic needs its own rule — the ephemeral-port gotcha), support allow and deny rules evaluated in numbered order, and apply to everything in the subnet whether you remembered it or not.

The working model: security groups are the primary control, designed per-tier with group-to-group references; NACLs stay near-default (allow all) unless you need a subnet-level guardrail — a deny rule that stands even if someone edits a security group badly. Teams that build elaborate NACL logic usually discover the statelessness tax during their first debugging session at 2 a.m. And least privilege applies to who can edit these as much as to the rules themselves — network changes belong behind the same IAM discipline as everything else.

Flow logs: turn them on before you need them

VPC Flow Logs record accepted/rejected traffic metadata to CloudWatch Logs or S3. They answer the questions every network incident asks — "is traffic even reaching this subnet?" "what is talking to the database?" — and they cannot be back-filled. Enabling them at VPC creation costs almost nothing and is the difference between evidence and vibes during debugging; the observability habits from the Kubernetes side apply here unchanged. While you're in the console: DNS resolution and DNS hostnames on (private DNS names make internal life bearable), and no workloads in the default VPC — it exists for experiments, its layout is nobody's design, and its flat structure has surprised many auditors.

Connecting beyond one VPC

Sooner or later the VPC needs neighbours, and each option has its shape:

• VPC peering — a direct link between two VPCs: cheap, fast, and non-transitive (A↔B and B↔C does not give A↔C), with CIDRs that must not overlap. Fine for a pair; a mesh of many VPCs becomes unmanageable surprisingly fast.
• Transit Gateway — the hub: VPCs and VPNs attach to one router, routes propagate sensibly, and the architecture stays legible at ten VPCs and beyond. It costs more than peering and earns it the moment "many" enters the sentence.
• VPN / Direct Connect — to your office or data centre. Start with site-to-site VPN (fast to stand up, internet-bound); Direct Connect is the committed, consistent-bandwidth option with lead times measured in weeks, ordered when the VPN's variability becomes a real complaint rather than a theoretical one.
• For Kubernetes specifically: EKS nodes live in your private subnets and inherit all of this — the cluster's outbound internet, endpoint choices and NAT layout are VPC decisions made before the first node joins, per the networking thinking in the Kubernetes networking guide.

And define it all as code. A VPC clicked together in the console is a snowflake nobody can rebuild; the same network as a Terraform module — CIDR, tiers and AZ count as variables — is an asset every new environment inherits.

Common mistakes

1. The too-small CIDR. A /24 VPC feels tidy until the second AZ or the EKS cluster arrives (Kubernetes networking is address-hungry). You cannot grow it; you can only rebuild. Start at /16.
2. Overlapping ranges across VPCs or with the office. Peering, Transit Gateway and VPN all refuse to cooperate with collisions. Allocate from a register, not from memory.
3. Workloads in public subnets "temporarily". Temporary is forever. If it doesn't face the internet for a living, it's private.
4. One NAT gateway for three AZs — a single point of failure plus cross-AZ transfer charges. One per AZ in production; shared only where the risk is understood and accepted.
5. No S3/DynamoDB gateway endpoints. Free to add, and their absence sends AWS's own traffic through the metered NAT. The most expensive checkbox in AWS is the one that costs nothing.
6. Elaborate NACL rule sets. Stateless rules with ephemeral ports turn routine debugging into archaeology. Security groups do the real work; NACLs are guardrails.
7. Flow logs "later". Later is during the incident, when the missing history would have answered the question. On at creation, always.

FAQ

How many subnets do I actually need? Per AZ: one public, one private, one data — so 6 for a solid 2-AZ setup, 9 for 3 AZs. More tiers (a separate management tier, say) are legitimate; more AZs than your region offers in your account is the usual surprise, so check AZ availability before finalising the layout.

Can I change the CIDR after creating the VPC? You can add secondary CIDR blocks, but you cannot remove or shrink the primary one. Treat the primary block as permanent — that's why the sizing advice leans generous.

Public IP on an instance in a private subnet — possible? No, and that's the point: no IGW route means no public reachability, whatever the instance settings say. Reach the internet via NAT (outbound) or not at all (data tier); reach the instance via load balancer, bastion or SSM Session Manager — which, not incidentally, is the modern answer to "how do I SSH in": usually, you don't; Session Manager gives audited shell access without any inbound ports.

VPC peering or Transit Gateway? Two VPCs that need each other: peering. Anything beyond a pair, or anything involving a shared services VPC or on-prem: Transit Gateway. The cost difference is real but small next to the operational difference at scale.

Conclusion

A VPC rewards the teams who treat its first afternoon as design, not setup: a generous non-overlapping CIDR, three tiers across multiple AZs, route tables understood rather than assumed, NAT per AZ, free gateway endpoints in place, security groups referencing each other, and flow logs recording from the first packet. None of it is glamorous, and almost none of it can be fixed cheaply later — which is precisely why it's worth a guide, and worth doing once, properly.

More AWS and Kubernetes guides are published weekly on DevOps Enginer — the free Kubernetes Production Checklist is available via the newsletter on the homepage.

TL;DR: A Terraform module is infrastructure with an interface: variables in, outputs out, opinions inside. Good modules wrap one coherent thing (a network, a cluster, an application stack), expose only the decisions callers genuinely need to make, give sensible defaults for the rest, and publish outputs that let other modules compose on top. Version them like software — semantic versions, pinned references, a changelog — because a module edit silently changes every environment that consumes it on the next plan. The failure modes are consistent: the god-module that tries to cover every case and satisfies none, the abstraction leak that forces callers to know the internals anyway, and the unpinned main branch reference that turns a routine apply into a surprise upgrade. Modules done well are how one engineer maintains ten environments; done badly, they're how a copy-paste problem becomes a shared copy-paste problem.

Originally published on DevOps Enginer.

Every growing Terraform codebase reaches the same moment: the staging VPC definition looks exactly like the production one except for six values, someone copies the folder, and from that day the two networks drift apart one forgotten edit at a time. Modules are Terraform's answer — and like most answers, they have their own failure modes. This guide covers the design decisions that separate a module library from a junk drawer.

What a module actually is

Strip the ceremony away: any directory of .tf files is a module. The root module is the one you run terraform apply in; child modules are directories (or registry entries, or git repos) that the root calls with a module block:

module "network" {
  source  = "git::https://example.com/infra-modules.git//network?ref=v1.4.0"
  # or: source = "./modules/network"   (local)
  # or: source = "terraform-aws-modules/vpc/aws"  (registry)

  cidr_block        = "10.20.0.0/16"
  az_count          = 3
  enable_flow_logs  = true
}

output "private_subnet_ids" {
  value = module.network.private_subnet_ids
}

The contract is the whole idea: variables are the questions the module asks, outputs are the promises it makes, and everything in between is its private business. A caller should never need to read a module's internals to use it — the moment they do, the abstraction has leaked and the module has failed at its one job.

Designing the interface (where modules are won or lost)

Expose decisions, not resources. Callers should pass intent — "3 AZs, flow logs on, this CIDR" — not raw resource arguments copied from the AWS provider docs. If your variable list mirrors the underlying resource's argument list one-to-one, you have not written a module; you have written a forwarding service. The module's value is its opinions: the subnet layout, the tag set, the naming scheme, the secure defaults — decided once, encoded once.

Defaults carry the opinions. A good module can be called with three or four inputs by a team that trusts its defaults:

variable "az_count" {
  type        = number
  default     = 3
  description = "How many availability zones to spread subnets across."
  validation {
    condition     = var.az_count >= 2 && var.az_count <= 4
    error_message = "az_count must be between 2 and 4 for this network design."
  }
}

variable "enable_flow_logs" {
  type    = bool
  default = true   # secure by default; turning it off is a decision, not an accident
}

Note the shape: typed variables, a description a stranger can act on, validation that fails at plan time with a human sentence, and defaults that encode the safe choice. Every default you set is a caller who never has to think about that knob — and a team convention that can never be forgotten.

Outputs are the composition layer. Publish the identifiers other modules will need (subnet IDs, the cluster endpoint, the bucket ARN) and nothing else. Outputs are a public API: teams will build on whatever you publish, so publish deliberately, and treat removing an output as a breaking change.

Size: one coherent thing

The temptation is the mega-module — module "platform" that builds the network, the cluster, the databases and the applications in one call. It feels convenient and ages terribly: every consumer inherits every decision, a change to any layer re-plans all of them, and the state file becomes a blast radius the size of your whole estate.

The working rule: a module wraps one layer of the stack that changes together and gets consumed together. Network. Cluster. Data store. Application service. Those compose:

module "network" { source = "./modules/network"  /* ... */ }
module "cluster" {
  source     = "./modules/cluster"
  subnet_ids = module.network.private_subnet_ids
}
module "app" {
  source        = "./modules/app-service"
  cluster_id    = module.cluster.id
  image         = "registry.example.com/api:${var.api_version}"
}

Composition through outputs keeps each module independently testable, independently versionable, and independently replaceable — when the cluster design changes, the network module neither knows nor cares. The same thinking, incidentally, is why state management per layer keeps blast radius small: module boundaries and state boundaries want to be the same lines.

Versioning: modules are software, treat them that way

A module consumed from a git repo at ?ref=main is an unpinned dependency on a moving target: every consumer silently upgrades whenever anyone merges anything. The first time that merge changes a default, every environment in the company plans a diff nobody asked for — during somebody's unrelated change window.

The disciplined version:

• Tag releases semantically. v1.4.0 tells consumers what kind of change they're taking: patch for fixes, minor for additive inputs/outputs, major for breaking changes (renamed variables, changed defaults, removed outputs).
• Pin every reference. ?ref=v1.4.0, never a branch. Upgrading a module becomes a deliberate, one-line, reviewable act per environment — staging takes it this week, production next week, and the diff in between is the changelog.
• Keep a changelog. Three honest lines per release beats a generated novel: what changed, who is affected, what to check after upgrading.
• Test before tagging. A module's test surface is terraform validate, a plan against a sandbox account, and — for the modules that matter — an ephemeral apply-and-destroy in CI. The registry ecosystem (terraform test, now built in) supports real test runs with mocked providers for the logic-heavy modules. Whatever the depth, a release nobody has planned against is a beta, labelled or not.

Local path modules (./modules/...) skip versioning entirely — fine for one repo's internal layers, where the "consumers" ship in the same commit. The moment a module is consumed from another repo or by another team, it has users, and users deserve versions.

Registry modules: stand on the community carefully

The public registry holds mature modules for the common shapes — the AWS VPC module alone encodes years of edge cases. Using them is good engineering, with two cautions. First, read the interface before committing: a famous module with 200 input variables demands you learn its opinions, which may not be yours. Second, a registry module is still a dependency: pin its version, and re-check its defaults on upgrade (a default flip in a popular module has surprised more than one team). When a registry module fits, it saves months; when it almost fits, a thin in-house wrapper module — your interface on the outside, the registry module inside — keeps your callers insulated from its surface.

Common mistakes

1. The god-module. Network + cluster + data + apps in one call: unreviewable plans, maximal blast radius, and consumers forced to accept every layer's choices. Split by layer; compose through outputs.
2. Variables that mirror resource arguments. A module whose inputs are the provider's docs restated provides no leverage — the opinions are the product.
3. Unpinned sources. ref=main turns every merge into a fleet-wide event. Pin versions; upgrade deliberately.
4. Everything is an output. Outputs are API. Publishing internals invites consumers to build on them, and then "internal" refactors become breaking changes.
5. One module for every environment difference. A variable count that grows with every new consumer is a god-module assembling itself. When a consumer's shape is genuinely different, a second module (or composition of smaller ones) beats a flag for every fork in the road.
6. No validation, failure at apply time. An invalid CIDR or an az_count of 7 should fail in seconds at plan, with a sentence explaining why — not fifteen minutes into an apply, midway through creating a network.

FAQ

When should code become a module? At the second honest reuse — or the first, if the thing is a recognised layer (network, cluster) whose shape you already know. Extracting on first use for everything produces speculative abstractions; waiting until the fourth copy produces archaeology.

Local modules or a shared module repo? One repo's layers: local paths, shipped together. Modules with consumers in other repos or teams: their own repo (or a registry), versioned and released. The boundary is about who consumes it, not about elegance.

How do I refactor a module without breaking consumers? The same way you change any API: additively first (new optional variables with defaults, new outputs), deprecate in the changelog, and reserve removals/renames for a major version. Consumers pinned to the old major are untouched until they choose to move.

Modules vs just using Terragrunt/workspaces? Different layers of the problem. Workspaces and wrapper tools manage environments and repetition at the root level; modules manage reusable components. Most estates of any size end up with both, and the state guidance in the state management guide applies either way. If you're still choosing the IaC tool itself, the Terraform vs Pulumi vs CloudFormation comparison is the wider view.

Conclusion

A Terraform module is a promise: give me these few decisions, and I will give you infrastructure built the way this team agreed — every time, in every environment, from one definition. Keep the interface small and opinionated, compose layers through outputs, version like software, and pin like you mean it. Do that, and the tenth environment costs you an afternoon of variables instead of a week of copy-paste — which was the entire point.

Setting up your first environments? The Kubernetes Production Checklist and the other free resources are on DevOps Enginer — new practical guides are published weekly.

TL;DR: ConfigMaps hold non-sensitive configuration (feature flags, endpoints, tuning values); Secrets hold sensitive values (passwords, tokens, keys) — with one warning that must be said out loud: a Kubernetes Secret is only base64-encoded, not encrypted, so anyone with API read access to it has the value. Configuration reaches pods two ways — environment variables (simple, but frozen at pod start) and volume mounts (files the kubelet keeps updated, but only for apps that re-read them). The production pattern most teams converge on: keep config in ConfigMaps, keep real secrets in an external vault (AWS Secrets Manager, Vault, and friends) and sync them in with the External Secrets Operator, encrypt etcd at rest, scope access with RBAC, and treat rotation as a designed event, not an emergency. Configuration is where "works in staging" goes to die — this guide makes it boring instead.

Originally published on DevOps Enginer.

Rebuild an image to change a log level once, and you understand why configuration exists as its own discipline. Rebuild one to change a database password during an incident, and you never design config badly again. Kubernetes gives you two objects for this job — ConfigMaps and Secrets — and a decade of teams have learned the same few lessons about them. Here they are, in one place.

ConfigMaps: the plain-text workhorse

A ConfigMap is key-value data, nothing more exotic:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  FEATURE_CHECKOUT_V2: "true"
  app.properties: |
    pool.size=20
    pool.timeout=5s

It can carry simple values and whole files (the app.properties block above lands as a file when mounted). Creation from files or literals via kubectl create configmap or your GitOps pipeline — as with everything else on this site, the manifest in Git is the version worth having; ad-hoc kubectl creates are config drift with extra steps. ConfigMaps are namespace-scoped, capped around 1 MiB, and — importantly — they are not a database: hundreds of tiny ConfigMaps are a smell pointing at a config service or a file baked sensibly into the repo, not more objects.

Secrets: base64 is not encryption (the paragraph to actually read)

A Secret looks like a ConfigMap with type: Opaque and values that appear encoded:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:          # write plaintext here; the API server encodes it
  username: app
  password: "s3cr3t-value"

That encoding is base64 — a transport format, not protection. The defences that actually protect a Secret, in the order they matter:

1. RBAC: who can get/list Secrets in the namespace. Read access to a Secret is the secret. This is the control that does the most work, and it is covered properly in the security basics guide.
2. Encryption at rest: etcd stores Secrets; enable envelope encryption (KMS-backed on managed clusters, often a toggle) so a stolen etcd backup doesn't hand over every credential in the cluster.
3. Not committing plaintext Secrets to Git. The most violated rule in this entire article. If manifests live in Git — they should — Secrets in plaintext manifests means the Git repo is your vault, with Git's access list, clone history and all. The fixes are the next section.

Secret types beyond Opaque exist for specific consumers (TLS secrets for ingress, docker-registry secrets for image pulls), and those integrate with their consumers directly — an ingress TLS secret, for instance, is the standard way to terminate HTTPS per the networking guide.

Getting configuration into the pod: env vars vs files

Two delivery mechanisms, with opposite trade-offs:

    env:
      - name: LOG_LEVEL
        valueFrom:
          configMapKeyRef: { name: app-config, key: LOG_LEVEL }
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef: { name: db-credentials, key: password }
    volumeMounts:
      - name: config
        mountPath: /etc/app
  volumes:
    - name: config
      configMap: { name: app-config }

• Environment variables are set at container start and never change for the life of the pod. Update the ConfigMap and nothing happens until pods restart. Simple, predictable, and the right default for values that shouldn't mutate under a running process anyway.
• Volume mounts present the data as files, and the kubelet updates the files when the source changes (within a minute or so, via an atomic symlink swap). But an updated file only matters if the application re-reads it — most don't. So volume-mounted config gives you eventual files, not live reload, unless the app watches the files (some frameworks do, via the ..data symlink change).

The operational consequence people trip on: changing a ConfigMap does not roll your Deployment. If the change must take effect, trigger the rollout (kubectl rollout restart) — or use the Helm trick of checksumming the config into a pod-template annotation so a config change is a template change and rolls automatically, per the Helm guide. Decide deliberately which behaviour each config deserves; "sometimes it reloads, sometimes not" is the worst of both.

Secrets in Git, solved properly: external stores and sync

Since plaintext Secrets must not live in Git, but workloads need Secrets in the cluster, the industry converged on one architecture: the vault is the source of truth; a controller syncs values into the cluster as Secrets.

The common implementation is the External Secrets Operator (ESO): you commit an ExternalSecret manifest that contains references — no values — and ESO fetches the real values from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault or HashiCorp Vault using the cluster's own cloud identity:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets, kind: ClusterSecretStore }
  target: { name: db-credentials }
  data:
    - secretKey: password
      remoteRef: { key: prod/app/db, property: password }

What Git shows anyone: that a password exists and where it lives. What it never shows: the password. Rotation happens in the vault; ESO refreshes the Secret; pods pick it up on their next restart (or sooner, for the file-watchers). Alternatives in the same family: the cloud providers' own CSI drivers (mounting secrets as in-memory files, never persisted as API objects at all — arguably the cleanest model) and Sealed Secrets, which encrypts values with a cluster key so the ciphertext is safe to commit — simpler than a vault, with key management becoming your job instead. For a small team with no vault yet, Sealed Secrets is a legitimate step; for anything multi-cluster or compliance-flavoured, the external-store pattern wins.

Two hygiene rules whichever route you choose:

• Different secrets per environment. The staging credential and the production credential being identical is how a staging compromise becomes a production incident.
• Audit access. Secret reads should be logged (CloudTrail on the vault side, API audit logs in-cluster) — per the observability guide, the time to learn who read a secret is not during the postmortem.

Rotation: the event you design before it designs you

Credentials rotate — on a schedule if you're organised, during an incident if you're not. A rotation that requires editing manifests in three repos and restarting everything by hand is not a rotation procedure; it's a hope. The designed version:

1. Vault holds the new value alongside the old (most stores support versions; databases can hold two valid users during a transition window).
2. ESO syncs; a controlled rollout restarts pods against the new value (kubectl rollout restart, staged by deployment strategy so a bad credential fails a canary, not the fleet).
3. Old value revoked after the new one is verified serving traffic.

Notice what this procedure never involves: a human seeing the value, a value in a chat message, or a rebuild. If any of those appear in your rotation story, the design has a hole in it.

Common mistakes

1. Plaintext Secrets in Git. The classic. History keeps them forever, every clone keeps them everywhere. Vault + sync, or sealed ciphertext — those are the options.
2. Believing base64 is encryption. It is encoding, readable by anyone in milliseconds. RBAC and etcd encryption are the actual protections.
3. One giant shared Secret for everything. Every pod that mounts it can read all of it. Split by consumer: a pod should hold the credentials for its own job and nothing else.
4. Expecting ConfigMap edits to reload apps. Env vars never update; mounted files update but apps rarely re-read. Roll the pods, or wire the checksum annotation, deliberately.
5. The same credentials in every environment — and in the load-test cluster, and in that old namespace nobody deleted. Scope per environment, per workload.
6. Config in the image, secrets in the wiki. Config baked into images forces rebuilds for tuning; secrets documented in shared notes are secrets in name only. Both are the same mistake: the value lives where it isn't managed.

FAQ

ConfigMap or Secret for a database URL? The URL itself (host, port, database name) is configuration — ConfigMap. The username and password are Secrets. Splitting them is normal and makes rotation cleaner: credentials change without touching the endpoint config.

How big can a ConfigMap be? Around 1 MiB per object (etcd object limits). If your config approaches that, it isn't config anymore — it's data, and it belongs in object storage or a database, mounted or fetched by the app.

Should pods read config from the API directly instead? Almost never. It couples the app to Kubernetes APIs and permissions it shouldn't hold. Env vars and files are the interface; keep the app portable and the cluster's API unexposed to workloads.

What's the difference between Sealed Secrets and External Secrets? Sealed Secrets stores the encrypted value in Git and decrypts it in-cluster — the repo remains the distribution mechanism. External Secrets stores only a reference in Git and fetches the value from a vault at sync time. The vault model scales to rotation, audit and multi-cluster; the sealed model is simpler to adopt. Both beat plaintext.

Conclusion

Configuration is a design surface, not a chore: ConfigMaps for the values you'd print on a whiteboard, Secrets for the ones you wouldn't, RBAC and etcd encryption as the real protections, a vault with sync as the source of truth, and rotation as a rehearsed procedure. Get those five decisions right and config changes become the least interesting part of your week — which, for anything involving production credentials, is exactly the ambition.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the configuration and secrets pre-flight items — RBAC scoping, backup tests and rotation readiness — alongside the rest of the production readiness checks.

TL;DR: Kubernetes releases three minor versions a year, and each is supported for roughly a year — fall two versions behind and upgrading stops being routine maintenance and becomes a project. Safe upgrades follow a fixed order: check for removed APIs before you touch anything, upgrade the control plane first (one minor version at a time, never skip), then rotate nodes with surge capacity so workloads always have somewhere to land, respecting PodDisruptionBudgets the whole way. The two facts that shape everything: you cannot skip minor versions on the control plane, and there is no downgrade button — your rollback plan is an etcd backup and node images you can still boot. Do the preparation and an upgrade is a boring afternoon. Skip it, and the boring afternoon finds you anyway, at a worse time.

Originally published on DevOps Enginer.

Every cluster is quietly ageing. The Kubernetes release train does not stop because a team is busy, and the gap between "we should upgrade soon" and "our version just went end-of-life" is where upgrade projects turn into incidents. This guide is the upgrade as a routine: the checks, the order, and the traps — written for the person who wants upgrades to be boring.

The rules of the road

Four constraints govern every upgrade, and planning means accepting them rather than negotiating:

1. One minor version at a time for the control plane. Going 1.30 → 1.32 directly is not supported for kube-apiserver. Each step gets applied, verified, and only then does the next begin. (Managed control planes enforce this for you; the steps still happen.)
2. Version skew is a policy, not a vibe. The supported combinations are specific: kubelets may be up to three minor versions behind the API server (since 1.28), but never newer than it; kubectl works within one minor version either way. "It connected" is not the same as "it's supported."
3. APIs get removed. Every minor release can retire beta APIs. Workloads still referencing a removed API fail at apply time — or worse, at the moment you need to redeploy during an unrelated incident. This is what the preparation phase exists to catch.
4. There is no downgrade. The API server and etcd do not roll back. If an upgrade goes wrong mid-flight, recovery means restoring etcd from backup or rebuilding — which is why backups are step zero, not an afterthought.

Phase 1: Preparation (where upgrades are actually won)

Inventory the target. Read the target version's changelog and the release's "urgent upgrade notes," and check your add-ons against it: ingress controller, CNI, CSI drivers, cert-manager, cluster-autoscaler/Karpenter, metrics-server. Add-on incompatibility breaks more upgrades than Kubernetes itself does — an ingress controller that doesn't support the new version will not announce this until traffic stops.

Hunt for removed APIs. Two tools make this mechanical:

# Scans manifests / Helm releases for deprecated and removed APIs
pluto detect-helm -o wide
pluto detect-files -d ./manifests -o wide

# Scans the live cluster for APIs in use that are scheduled for removal
kubent

Fix every finding by moving to the replacement API (the tools name it) before the upgrade window. While you are in there, check the same hygiene for admission webhooks: a dead webhook with failurePolicy: Fail can block the whole cluster's writes after a control-plane change, so inventory webhooks and know which ones are load-bearing.

Back up etcd — and know how you'd restore it. On self-managed clusters, snapshot etcd and copy the snapshot off the node. On managed clusters (EKS, GKE, AKS), the provider handles the control plane, but your workload state is your problem: if everything is in Git (see my Argo CD tutorial), the cluster is rebuildable; if it isn't, a volume-level backup tool is your safety net, per the storage guide.

Check cluster health honestly. Upgrading a cluster that already has failing nodes, Pending pods, or a flapping control plane converts one problem into two. kubectl get nodes, component statuses, and a look at recent events: fix first, upgrade second.

Phase 2: Control plane first

With preparation done, the control plane goes first — always. On managed services this is one action per minor version (console, CLI or Terraform), followed by a pause: watch API server availability, controller-manager and scheduler health, and confirm nodes still report Ready before proceeding.

On self-managed clusters with kubeadm, the shape is:

# On the first control-plane node, per minor version:
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.31.x-*   # target version
kubeadm upgrade plan
kubeadm upgrade apply v1.31.x
# Then upgrade kubelet + kubectl on the node and restart kubelet

kubeadm upgrade plan is the unsung hero here: it reports component versions, warns about what it can and cannot do, and refuses some genuinely bad ideas. Read its output like a pre-flight checklist, because that is what it is. Repeat for the remaining control-plane nodes with kubeadm upgrade node.

Phase 3: Nodes — rotate, don't patch in place

Worker nodes upgrade by replacement wherever possible: bring up a node at the new version, drain an old one, let workloads reschedule, remove the old node. On managed node groups this is the built-in rolling update, with one setting that decides whether it is graceful:

• Max surge: allow the group to temporarily run extra nodes (e.g. +1 or +33%) during the update. Surge means a drained node's workloads already have a home; without surge, the update cordons a node and hopes capacity suffices — on a tightly packed cluster, it doesn't, and pods sit Pending mid-upgrade. This is the same capacity logic as autoscaling: replacement needs somewhere to land.
• PodDisruptionBudgets are the contract that makes draining safe: drain respects voluntary-disruption budgets, so a workload with minAvailable set cannot be fully evicted by an upgrade. Workloads without PDBs can be taken to zero by a drain — if that outage would hurt, the PDB was the missing pre-work (Phase 1 again).

The manual shape, when you are rotating nodes yourself:

kubectl cordon <old-node>
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
# wait: confirm every workload is Running elsewhere
kubectl delete node <old-node>   # after the node itself is terminated/replaced

Watch for the two classic drain blockers: pods with local storage (--delete-emptydir-data acknowledges losing it — be sure you mean that) and pods whose PDB forbids the eviction, in which case the drain waits, correctly, and you investigate the workload rather than forcing it. --force exists; reaching for it during an upgrade is how upgrades become incidents.

On self-managed nodes where replacement isn't practical, kubelet upgrades in place: install the new kubelet (and container runtime, checking its own compatibility), restart, verify the node rejoins at the new version within skew policy.

Phase 4: Verify like you don't trust it (because you shouldn't)

"Upgrade completed" from a console is the beginning of verification, not the end:

1. kubectl version and kubectl get nodes — every node at the intended version, all Ready.
2. Core add-ons healthy: DNS resolution from a test pod, ingress serving, metrics flowing (kubectl top nodes) — the observability guide lists the signals worth watching in the hours after.
3. One real deployment: roll a canary workload, confirm scheduling, networking and storage attach all behave.
4. Add-on upgrades you deferred: now is when ingress, cert-manager and friends get their matching bumps, one at a time, while the window is still staffed.

Then leave the cluster alone for a soak period before starting the next minor version. One version, one verification, one soak — the rhythm that keeps a three-version catch-up from becoming a single three-version mystery.

Managed vs self-managed: what actually differs

The order and the rules are identical; only the hands change. Managed services move the control plane for you and offer node-group rotation with surge built in — your job concentrates on Phase 1 (API removals, add-on compatibility) and Phase 4 (verification), which is honestly where the risk always lived. Self-managed adds the kubeadm mechanics and etcd backup duty. In both worlds, the teams that get hurt are hurt by preparation gaps, not by the upgrade command itself.

Common mistakes

1. Skipping the removed-API scan. Everything works until the first redeploy against a removed API — during a different incident, naturally.
2. Trying to jump two minor versions. The control plane refuses or, worse, half-applies. One step at a time is not a suggestion.
3. Zero surge on node rotation. Drained workloads with nowhere to go; Pending pods; an upgrade window that ends in a rollback of the schedule rather than the version.
4. No PDBs on important workloads. Drain takes them to zero, voluntarily, as designed — the design just never got told they mattered.
5. Upgrading an unhealthy cluster. Pre-existing node or control-plane faults become upgrade casualties and misdirect the whole investigation.
6. Forgetting the add-ons. Kubernetes upgrades fine; the two-year-old ingress controller doesn't. Inventory includes everything with a version number, not just Kubernetes.

FAQ

How often should we upgrade? Often enough to never be more than one minor version behind the oldest supported release — in practice, most teams settle into upgrading every 2–4 months, which keeps each event small. The cost of upgrading grows with the distance jumped; frequency is the discount.

Can we skip a version on managed Kubernetes? No — the provider performs the same one-minor-at-a-time steps; the console just makes them look like one action. Node groups likewise step through versions.

What if something breaks mid-upgrade? For nodes: stop the rotation, keep workloads on healthy nodes, fix forward or replace the bad node. For the control plane: there is no downgrade, which is why the pre-upgrade etcd snapshot (self-managed) or your Git-backed workload state (everywhere) is the real rollback plan.

Do workloads need downtime? Properly built ones don't: multiple replicas, PDBs, surge capacity and rolling drains mean pods move, not stop. Single-replica workloads without PDBs will have a gap during their node's drain — the upgrade is merely revealing a property the workload always had. The deployment strategies guide covers building workloads that survive exactly this.

Conclusion

A Kubernetes upgrade is a checklist wearing a scary costume: scan for removed APIs, back up, control plane one minor at a time, nodes rotated with surge and PDBs respected, then verify like a sceptic. Run it a few times a year and it shrinks to what it should be — routine maintenance with a beginning, a middle, and an end. Let it slide for a year, and the same work returns wearing its real costume: a project, with a deadline set by an end-of-life date.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the upgrade pre-flight items — health checks, backup verification and disruption budgets — alongside the rest of the production readiness checks.

TL;DR: Kubernetes storage confuses people because it splits one idea — "my pod needs a disk" — into three objects. A PersistentVolumeClaim (PVC) is the pod's request: "I need 10Gi, writable by one node." A PersistentVolume (PV) is the actual disk that satisfies it. A StorageClass is the recipe that creates those disks on demand, with a chosen type, reclaim behaviour and expansion policy. Stateless apps should stay on Deployments with no volumes at all. Apps that genuinely own data — databases, queues, anything whose identity is its disk — belong on StatefulSets, which give each replica a stable name, a stable network identity and its own claim that survives rescheduling. Most storage incidents come from three places: the wrong reclaim policy deleting data, a volume bound to one availability zone while the pod reschedules into another, and nobody testing a restore until the day it matters.

Originally published on DevOps Enginer.

Pods are cattle; disks are pets. Everything awkward about Kubernetes storage lives in that sentence. This guide walks the moving parts in the order a request flows through them, then covers StatefulSets, expansion, snapshots, and the failure modes that actually bite.

The three objects, in the order they meet

When a pod declares it needs storage, this is the chain:

1. PersistentVolumeClaim — the request, written by you (or generated by a StatefulSet). It asks for a size, an access mode, and optionally a StorageClass.
2. StorageClass — the template. It names the provisioner (your cloud's disk service, a CSI driver, a local provisioner), the parameters (disk type, IOPS tier), the reclaim policy, and whether volumes can be expanded later.
3. PersistentVolume — the reality. With dynamic provisioning (the normal case on any cloud cluster), the StorageClass's provisioner creates a real disk and a PV object representing it, bound one-to-one to your claim.

You almost never create PVs by hand anymore. You write PVCs, you choose (or set a default) StorageClass, and provisioning does the rest. If a claim sits in Pending, the conversation between those three objects broke somewhere — usually the class doesn't exist, the provisioner can't create disks in that zone, or quota ran out. kubectl describe pvc <name> prints the events that tell you which.

Access modes: the line that decides your architecture

Every claim declares how the volume may be mounted, and this choice constrains everything downstream:

• ReadWriteOnce (RWO) — one node at a time. The default for cloud block disks (EBS, Azure Disk, GCE PD). Fine for a database with one writer; it also means a rolling update can deadlock if the new pod lands on a different node while the old pod still holds the volume — the classic multi-attach error.
• ReadWriteMany (RWX) — many nodes at once. Requires a shared filesystem (NFS, EFS, Azure Files, Filestore). Needed when several pods across nodes must see the same files; you pay for it in performance characteristics and complexity.
• ReadWriteOncePod (RWOP) — one pod, not just one node (Kubernetes 1.29+ for CSI volumes). The stricter guarantee single-writer workloads actually wanted.

Read the mode before you design around a volume. Half of all "Kubernetes storage is broken" stories are a workload assuming RWX semantics on an RWO disk.

StorageClasses: the decisions hiding in the defaults

One cluster usually offers several classes — a fast SSD class, a balanced class, maybe an HDD class. The fields that matter most are the quiet ones:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-retain
provisioner: ebs.csi.aws.com        # example: AWS EBS CSI driver
parameters:
  type: gp3
reclaimPolicy: Retain               # not Delete — see below
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

• reclaimPolicy decides what happens to the disk when the PVC is deleted. Delete destroys the underlying volume — convenient for throwaway environments, catastrophic for a production database whose claim gets deleted by a namespace cleanup or a careless Helm uninstall. For anything whose data matters: Retain. Retain leaves the PV in a Released state; the data survives until a human deliberately deals with it.
• volumeBindingMode: WaitForFirstConsumer delays provisioning until a pod actually schedules. Without it, a disk can be created in zone A before the scheduler places the pod in zone B — and a zonal disk can never attach across zones, so the pod is stuck forever. This single field prevents one of the most common storage dead-ends on multi-zone clusters.
• allowVolumeExpansion: true is only settable at class creation in effect — you cannot retroactively make a non-expandable class expandable for existing volumes in most setups. Set it now, thank yourself later.

Check what your cluster defaults to before writing a single claim:

kubectl get storageclass
kubectl describe pvc <claim> -n <namespace>
kubectl get pv

StatefulSets: when the pod's identity IS the disk

Deployments treat replicas as interchangeable — perfect for stateless services, wrong for a database cluster where db-0, db-1 and db-2 are different members with different data. StatefulSets exist for those workloads and give three guarantees Deployments don't:

• Stable identity: pods are named name-0, name-1, … and keep that name across restarts, with a matching stable DNS entry per pod via a headless Service.
• Per-replica storage: a volumeClaimTemplates block generates one PVC per replica (data-db-0, data-db-1, …). When a pod is deleted and recreated, it reattaches to its claim — the data follows the identity, not the node.
• Ordered behaviour: replicas start and stop in order, which clustered systems that elect leaders and replicate in sequence genuinely need.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db-headless
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: db
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-retain
        resources:
          requests:
            storage: 20Gi

Two operational truths about StatefulSet claims. First, scaling down does not delete the claims — scale from 3 back to 5 later and the old replicas reattach to their existing data. That is deliberate and usually what you want; it also means forgotten claims quietly keep disks (and bills) alive, which is why storage belongs in the cost optimization review. Second, deleting a StatefulSet leaves its PVCs behind too — cleanup is a separate, explicit act. Annoying on a tidy-up day; a lifesaver on a bad day.

And the honest question to ask first: does this workload need to be stateful in Kubernetes at all? A managed database service outside the cluster is often the right answer for small teams. Run state in-cluster when you have a real reason — latency, cost at scale, data locality — not by default. If you do run it, the deployment strategies guide covers how updates roll for these workloads, and it is not "like a Deployment."

Growing and protecting volumes

Expansion is the happy path, when it was prepared: with allowVolumeExpansion: true on the class, you edit the PVC's requested size upward and the provisioner grows the disk; most filesystems expand online once the pod restarts or remounts. Shrinking is not supported — a smaller disk means a new volume and a data migration, full stop.

Snapshots are the backup primitive: a VolumeSnapshot asks the CSI driver for a point-in-time copy, and a new PVC can be created from a snapshot to restore or clone. Combined with a schedule (via your backup tooling or a CronJob-driven process), this is how in-cluster state gets real backups. Two warnings from experience-shaped convention:

• A snapshot is not a backup strategy until a restore has been tested. Untested restores fail in creative ways: wrong class, wrong zone, permissions, application-level inconsistency.
• For databases, storage-level snapshots are crash-consistent at best unless the application quiesces writes. Where correctness matters, pair snapshots with the database's own backup tooling.

Security applies to disks too: volume contents inherit your RBAC and Pod Security posture — whoever can create a PVC from a snapshot can read the data in it, so snapshot permissions deserve the same least-privilege care as the volumes themselves.

Common mistakes

1. reclaimPolicy: Delete on data that matters. The claim gets deleted — by a person, a script, a namespace teardown — and the disk goes with it. Retain for production state, always.
2. No WaitForFirstConsumer on multi-zone clusters. Disks provisioned in the wrong zone; pods Pending forever with attach errors that look like scheduler problems.
3. Assuming RWX from an RWO disk. Multi-attach errors during rollouts, or worse, silent assumptions in the app design that only surface under failover.
4. Forgetting StatefulSet claims on cleanup. Orphaned PVs holding real disks for months — visible in the bill, invisible in kubectl get pods.
5. Snapshots nobody has restored from. A backup you have never tested is a hypothesis. Schedule a restore drill like you schedule the backup.
6. Running stateful workloads on Deployments with a shared claim. Losing stable identity and per-replica storage is how databases end up with two writers and one corrupted dataset.

FAQ

Should my web app have a PersistentVolume? Almost never. If the app is stateless, uploads and user files belong in object storage (S3/GCS/Azure Blob) or a managed service, not a pod-local disk. PVCs are for workloads whose data is the workload — databases, brokers, search indexes.

What happens to my data if the pod is deleted? With a PVC, nothing — the volume outlives the pod and reattaches to its replacement. With no volume (emptyDir or the container filesystem), everything is gone with the pod. That distinction is the entire subject of this article.

Can I move a volume to another cluster? Not by copying YAML. The path is snapshot/restore (CSI snapshots, or backup tools built on them) or application-level replication/export. Plan migrations around the data tooling, not the manifests.

Why is my PVC stuck in Pending? Read its events: kubectl describe pvc. The usual causes are a StorageClass that doesn't exist, a provisioner that can't create volumes in the needed zone, exhausted quota, or WaitForFirstConsumer waiting (correctly) for a pod that itself can't schedule — in which case the pod's events, covered in the troubleshooting guide, hold the real reason.

Conclusion

Kubernetes storage stops being scary when you keep the chain straight: claims ask, classes define, volumes provide — and StatefulSets bind identity to data for the workloads that need it. Choose the access mode honestly, set Retain and WaitForFirstConsumer before they save you, grow volumes instead of replacing them, and test restores before you need them. Disks are pets; treat them with the deliberate care pets require, and they will stop biting.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the storage pre-flight items — reclaim policies, backup tests and disruption budgets — alongside the rest of the production readiness checks.

TL;DR: A default Kubernetes cluster is far more open than most teams realise: any pod can talk to any other pod, many workloads run with more API permission than they need, and "the app works" often quietly means "the app runs as root with a token mounted it never uses." You do not need a security platform to fix the worst of it. Three controls close most of the real-world holes: RBAC scoped to what each identity actually does, NetworkPolicies that end the flat any-to-any network, and Pod Security Standards that stop privileged, root-running workloads at admission time. Add disciplined secrets handling and image hygiene, and a default cluster becomes a defensible one — with changes your team can roll out namespace by namespace, without a big-bang migration.

Originally published on DevOps Enginer.

Kubernetes security advice tends to arrive as a 200-item checklist, which is why so much of it never gets implemented. This guide takes the opposite approach: the small set of controls that remove the most risk per hour of effort, in the order I would apply them to a real cluster, with the manifests that matter.

Start from the uncomfortable defaults

Before the controls, be honest about what a fresh cluster permits:

• The network is flat. With no NetworkPolicies, every pod can open a connection to every other pod and service in the cluster. One compromised frontend can reach your database directly.
• Every pod gets a service account token mounted by default (unless you turn it off), whether or not the application ever calls the Kubernetes API.
• Containers frequently run as root, with the default capabilities, on images pulled by a mutable tag like latest.
• RBAC is often bypassed in practice — not by attackers first, but by teams handing out cluster-admin because a narrower role "took too long to debug."

None of this is exotic. Attackers do not need zero-days when the defaults hand them lateral movement and credentials. The good news: each default above has a specific, boring fix.

Control 1: RBAC — least privilege for humans and workloads

RBAC answers one question: who is allowed to do what to which resources? Two identities matter — the people (and CI systems) who operate the cluster, and the service accounts workloads run as.

For workloads: give each application its own ServiceAccount, and set automountServiceAccountToken: false unless the app genuinely calls the API. Most applications never do — the mounted token is pure attack surface: steal the pod, steal a credential.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
automountServiceAccountToken: false

When an app does need API access, bind a Role that names the exact verbs and resources — get and list on configmaps in one namespace is a role; * on * is an incident waiting for a date:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: config-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

For humans and CI: scope roles per namespace, treat cluster-admin like root on a production database — a named, audited exception, not a team default — and audit what you have. Two commands tell you most of the story:

kubectl get clusterrolebindings -o wide
kubectl auth can-i --list --as=system:serviceaccount:production:api -n production

The second is the one to run for every workload identity: it prints exactly what that token can do, in plain text. If the list surprises you, that is the finding. This is the same least-privilege discipline as IAM on the cloud side — my AWS IAM least privilege guide works through the identical thinking for AWS policies.

Control 2: NetworkPolicies — end the flat network

A NetworkPolicy is a firewall rule for pods: it selects pods and declares which traffic may reach them (ingress) and where they may connect (egress). Policies are enforced by your CNI plugin — an important caveat: if your CNI does not enforce NetworkPolicy (the basic kubenet setup is the classic example), the objects exist and silently do nothing. Cilium, Calico and the major cloud CNIs enforce them; verify yours before trusting the YAML.

Roll out in this order, namespace by namespace:

Step 1 — default deny. One policy that selects all pods in the namespace and allows nothing changes the posture from "open unless closed" to "closed unless opened":

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Step 2 — allow what the app actually needs. Then add narrow allowances: the frontend may reach the API on its port; the API may reach the database on 5432 and DNS on 53 (forgetting DNS egress is the classic self-inflicted outage — allow it explicitly):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

Do this in a staging namespace first, watch what breaks, and promote the same policies with the application — if you run GitOps, they belong in the same repository as the workloads, reviewed like any other change (my Argo CD tutorial covers that workflow). A compromised pod inside a default-deny namespace is a contained problem; in a flat network it is a cluster tour.

Control 3: Pod Security Standards — stop bad workloads at the door

RBAC limits identities and NetworkPolicies limit traffic, but neither stops a deployment that asks to run privileged, mount the host filesystem, or run as root. That is admission-time work, and Kubernetes has it built in: Pod Security Standards, enforced through Pod Security Admission with three profiles:

• Privileged — effectively no restrictions. For system components that genuinely need host access, and nothing else.
• Baseline — blocks the known privilege escalations (privileged containers, host namespaces, dangerous capabilities) while staying compatible with most typical workloads.
• Restricted — the hardened target: no root, no privilege escalation, drop all capabilities, seccomp on. What user-facing workloads should aim for.

Enforcement is a namespace label — no webhook to install:

kubectl label namespace production \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

That combination is the pragmatic rollout: enforce Baseline today (violating pods are rejected), while audit and warn on Restricted shows you — in API warnings and audit logs — exactly which workloads would fail the stricter profile, so you can fix them deliberately instead of breaking production on day one. At the workload level, the settings Restricted asks for are small and belong in every Deployment template:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

The honest blocker is usually the image: an app whose image only runs as root needs a Dockerfile fix, not an exception. Treat exceptions as named, expiring decisions — a namespace labelled back to Privileged "temporarily" has a way of becoming permanent.

Secrets: the rules that prevent the embarrassing incidents

Kubernetes Secrets are base64-encoded, not encrypted, and anyone with the right RBAC verb can read them — which loops straight back to Control 1. The working rules:

• Encryption at rest on for etcd (a cluster-level setting; managed clusters differ in defaults — verify yours, don't assume).
• Never bake secrets into images or commit them to Git. Use environment injection or mounted volumes from a secrets source, and scan repositories in CI so a committed credential fails the pipeline — that scanning habit is part of the wider pipeline discipline in my DevSecOps checklist.
• Prefer an external secrets store (cloud secret managers, or an external-secrets operator syncing from one) as the source of truth, so rotation and audit live in one place instead of being scattered across namespaces.
• Rotate like you mean it. A secret that has never been rotated should be assumed permanent — and anything permanent eventually leaks.

Image hygiene: know what you are actually running

The last basic: the artefact you deploy. Four habits cover most of it:

1. Pin by digest, or at least by specific version tag. latest means "whatever exists at pull time," which is neither reproducible nor auditable.
2. Scan in CI and watch the results. A scanner that reports into a void changes nothing; gate on fixable critical vulnerabilities for internet-facing workloads.
3. Use minimal bases. A small base image is fewer packages to patch and fewer surprises — the same discipline as my Docker image optimization guide, where smaller and more secure turned out to be the same work.
4. Pull from registries you control or trust, with image pull policies that do not silently fall back to stale local copies.

Common mistakes

1. NetworkPolicies on a CNI that doesn't enforce them. The YAML applies cleanly, everyone feels safer, nothing is filtered. Verify enforcement with a test connection that should fail.
2. cluster-admin as a convenience. Every standing cluster-admin binding is a standing invitation. Scope it, name it, review it.
3. Default-deny without DNS egress. The policies "work" and every lookup fails. Always allow UDP/TCP 53 to your DNS service explicitly.
4. Enforcing Restricted on day one. Rejected pods across production, followed by a panicked rollback to no standards at all. Baseline-enforce + Restricted-audit gets you to the same place without the outage.
5. Service account tokens mounted everywhere. One flag — automountServiceAccountToken: false — removes a credential from every pod that never needed it.
6. Secrets in Git "just for staging." Staging secrets have a habit of being production secrets. Scan, block, rotate if it already happened.

FAQ

Where do I start if the cluster has none of this? In this order: inventory RBAC (clusterrolebindings and can-i --list for your workloads), turn off automatic token mounting where the API isn't used, then default-deny NetworkPolicies in one staging namespace, and Baseline Pod Security in audit mode cluster-wide. Each step is an afternoon, not a quarter.

Do NetworkPolicies replace a service mesh or an external firewall? No — they cover pod-to-pod traffic inside the cluster at L3/L4. Egress to the internet, L7 rules and identity-based policy are other layers' jobs. NetworkPolicies are the floor, not the whole building.

Managed Kubernetes (EKS/GKE/AKS) — is this handled for me? Partially. The provider secures the control plane and offers defaults, but RBAC bindings, NetworkPolicies, Pod Security labels and your images remain yours. The shared-responsibility line runs right through the middle of this article.

How do I prove it's working? Test negatively: from a frontend pod, try to reach the database and watch it fail; deploy a privileged test pod in a Baseline namespace and watch it get rejected. A control you have never seen block anything is a hope, not a control. For the wider detection picture, see the Kubernetes observability guide.

Conclusion

Kubernetes security basics are not a product you buy; they are five defaults you deliberately change: scoped RBAC, no free tokens, a network that denies by default, admission that refuses privileged workloads, and secrets and images handled like they matter. Roll them out namespace by namespace, enforce gradually, and test that each control actually blocks something. A cluster secured that way will not make headlines — which is precisely the point.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the security pre-flight items — RBAC review, token mounting, disruption budgets and rollback paths — alongside the rest of the production readiness checks.

TL;DR: Kubernetes has four different autoscaling layers and most problems come from expecting one of them to do another one's job. The Horizontal Pod Autoscaler (HPA) changes how many pods you run. The Vertical Pod Autoscaler (VPA) changes how big each pod is. Cluster autoscaling (Cluster Autoscaler or Karpenter) changes how many nodes exist for pods to land on. KEDA scales on events — queue depth, Kafka lag, cron schedules — instead of CPU. In production you usually need at least two of these together: HPA for replicas plus node autoscaling underneath, or your new pods sit Pending forever. And none of them work without honest resource requests, because every percentage-based decision is calculated from the requests you set.

Originally published on DevOps Enginer.

Getting autoscaling right is the difference between a cluster that absorbs a traffic spike quietly and one that pages you at 3 a.m. while pods pile up unscheduled. This guide walks each layer in the order traffic actually flows through your cluster, with the configuration that matters and the mistakes that quietly break each one.

Why autoscaling fails quietly

Autoscaling rarely fails with an error. It fails silently, in three classic ways:

1. Pods scale, nodes don't. HPA adds replicas, but there is no room for them. They sit in Pending while the load that triggered them keeps growing.
2. Nodes scale, pods can't use them. The cluster adds capacity, but pod requests are set so high that each node fits only one or two pods — you pay for nodes that are mostly reserved air.
3. The metric lies. HPA targets 70% CPU, but the app is memory-bound, or requests were set years ago and no longer resemble reality. The scaler makes confident decisions from bad inputs.

Every layer below exists to close one of these gaps. If you haven't set requests and limits deliberately yet, start with my guide to Kubernetes resource requests and limits — autoscaling built on guessed requests just automates the guessing.

Layer 1: Horizontal Pod Autoscaler — more copies of the pod

HPA is the layer most teams already have. It watches a metric — by default CPU or memory utilisation relative to pod requests — and adjusts the replica count of a Deployment, StatefulSet or similar workload between a minimum and a maximum.

A production-shaped HPA looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

Three details in that manifest do most of the work:

• minReplicas: 3, not 1. A single replica is a single point of failure and a slow cold start on the first spike. Three gives the scaler something to work with and survives a node loss.
• The behavior block. Without it, scale-down is eager: load dips, replicas vanish, load returns, pods churn. A 300-second stabilisation window on scale-down and one pod per minute keeps the replica count calm instead of flapping. Scale-up stays aggressive — when you need capacity, you need it now.
• The target is utilisation of requests. 65% of a 500m CPU request is a real number; 65% of a request nobody has reviewed since 2023 is not. This is why the requests guide matters more than the HPA manifest itself.

Prerequisites people forget: HPA needs metrics. Resource metrics come from the metrics-server add-on, which many clusters install by default but minimal or self-managed clusters may not have. If kubectl top pods returns nothing, HPA is flying blind and will do nothing — check that first before debugging the HPA object itself. Custom and external metrics (requests per second, queue length) need an adapter such as the Prometheus Adapter, which is where KEDA, below, often becomes the simpler path.

Layer 2: Vertical Pod Autoscaler — the right size for each pod

VPA attacks a different problem: the pod itself is the wrong size. It observes actual usage over time and recommends — or automatically applies — better requests.

VPA runs in three modes, and choosing the mode is the whole decision:

• Off — it only shows recommendations (kubectl describe vpa becomes a right-sizing report). Safe everywhere, and honestly the mode most teams should start with for a few weeks.
• Initial — requests are set when a pod is created and never changed afterwards. Low disruption, slow to adapt.
• Auto — VPA evicts and recreates pods to apply new requests. Correct sizes, but your pods restart when VPA decides — never point Auto mode at something that cannot tolerate restarts.

The classic conflict: never let VPA in Auto mode and HPA fight over the same resource. If HPA scales on CPU utilisation while VPA keeps resizing the requests that utilisation is calculated from, the two controllers chase each other. The standard peace treaty: HPA on CPU (or on custom metrics), VPA in Off mode as an advisor — or VPA adjusting memory only while HPA watches CPU. My Kubernetes cost optimization guide covers why this right-sizing loop is where most of the money is: oversized requests are a silent tax on every node you run.

Layer 3: Node autoscaling — room for the pods to land on

HPA can decide it wants 20 replicas, but if the cluster has capacity for 8, the other 12 do not exist yet — they are Pending. Node autoscaling is the layer that notices unschedulable pods and adds machines, and later removes empty ones.

You have two mainstream choices:

Cluster Autoscaler is the older, conservative option. It watches for pending pods, asks the cloud provider for another node from an existing node group or instance pool, and scales groups down when nodes sit under-utilised. It is predictable and battle-tested, and its weakness is rigidity: it scales within node groups you pre-defined, so the instance types and sizes are your homework.

Karpenter (open source, originally built at AWS and now a CNCF project) takes the opposite approach: instead of scaling fixed groups, it watches pending pods, reads their actual requirements — CPU, memory, zones, labels, taints — and provisions a node shaped to fit them, often in under a minute. When nodes empty out, it consolidates: shuffles pods onto fewer, cheaper nodes and terminates the rest. The practical difference shows up exactly where Cluster Autoscaler is rigid — mixed workloads with very different shapes — and in consolidation, which steadily squeezes wasted capacity out of the cluster instead of only reacting to spikes.

Whichever you run, three settings decide whether it behaves:

• PodDisruptionBudgets on important workloads. Node scale-down and consolidation drain nodes. A PDB is how a workload says "you may move me, but never take me below this many healthy replicas." Without PDBs, autoscaling will eventually evict something at a bad moment. (My deployment strategies guide covers PDBs alongside rolling updates.)
• Do-not-evict annotations for the truly pinned pods — the singleton, the job mid-run. Both controllers respect signals that a pod must not be moved; use them sparingly and deliberately.
• A floor under the cluster. Keep enough baseline capacity for your steady state. Autoscaling is for variance, not for running the cluster at zero slack and hoping provisioning is always fast.

Layer 4: KEDA — scaling on events, not CPU

Some workloads have no meaningful CPU signal. A worker that drains a queue is idle at 2% CPU with 10,000 messages waiting; a report generator should exist twice a day and never at night. Scaling these on CPU is scaling on a lie.

KEDA (Kubernetes Event-Driven Autoscaling, a CNCF graduated project) extends HPA rather than replacing it: it feeds external signals — SQS or Kafka backlog, RabbitMQ queue depth, database query results, cron schedules, Prometheus queries — into the scaling decision as metrics. Its two signature abilities:

• Scale to zero. HPA's floor is 1 replica (well, minReplicas of 1 in practice). KEDA can take a workload to 0 when the queue is empty and bring it back when work arrives. For intermittent workers, this is the difference between paying for idle pods all month and paying for none.
• Scale on the signal that predicts the work. Queue depth rises before CPU does. Scaling on the cause instead of the symptom removes most of the lag that makes CPU-based scaling feel sluggish.

A typical pattern: a ScaledObject declares the target deployment, the trigger (say, messages visible in a queue), and a threshold (one pod per 50 messages, between 0 and 15 replicas). KEDA maintains the underlying HPA for you. If you already run Argo CD, ScaledObjects are just more YAML in Git — the same review-and-rollback story as everything else.

Putting the layers together

A sane default stack for a typical web service:

And the observability to prove it works: watch Pending pod count, HPA's current-vs-desired replicas, and node utilisation after consolidation. If you cannot see autoscaling deciding, you are trusting it blindly — the metrics to wire up are covered in my Kubernetes observability guide.

Common mistakes

1. Autoscaling on CPU for a memory-bound app. The metric must be the resource the workload actually runs out of — or a custom metric closer to the real demand, like requests per second.
2. No metrics-server, no scaling. kubectl top returning nothing means HPA sees nothing. Verify the plumbing before tuning the policy.
3. HPA + VPA Auto on the same resource. Two controllers, one variable, endless churn. Pick who owns CPU.
4. maxReplicas as an afterthought. Too low and the spike wins; absurdly high and a runaway loop or a bad deploy can scale you into a very large cloud bill. The max is a budget decision, not a formality.
5. Scaling pods without scaling nodes. The Pending pile-up. Every HPA rollout should be paired with the question: where do the new pods land?
6. No PodDisruptionBudgets. Node consolidation and scale-down will eventually evict the wrong pod at the wrong time.
7. Scale-down as eager as scale-up. Flapping replicas churn connections, caches and readiness. Stabilisation windows are free stability.

FAQ

Do I need all four layers? No. Most services need HPA plus node autoscaling. Add VPA in advisory mode when you suspect requests are wrong (they usually are), and KEDA when a workload's real demand signal is a queue, a schedule or another external event rather than CPU.

Karpenter or Cluster Autoscaler? If you are starting fresh on AWS, Karpenter's just-in-time, right-shaped provisioning and consolidation usually win. Cluster Autoscaler remains a solid, predictable choice — especially outside AWS or where node groups are deliberately fixed for compliance reasons.

Can KEDA replace HPA? KEDA uses HPA under the hood for most triggers. What it replaces is your hand-rolled metrics plumbing, and it adds scale-to-zero, which plain HPA does not give you.

Why are my pods Pending even though autoscaling is on? Work through the layers: HPA may have scaled (check desired replicas), but if nodes are full and the node autoscaler cannot add capacity — quota limits, no matching node group, Karpenter constraints — pods wait. kubectl describe pod on a pending pod names the failing condition; my Kubernetes troubleshooting guide walks that diagnosis step by step.

Conclusion

Autoscaling is not one feature you turn on; it is four decisions stacked on top of each other — how many pods, how big each pod is, how many nodes exist, and which signal you trust. Get the requests honest first, pair every pod scaler with node capacity underneath it, protect workloads with disruption budgets, and scale event-driven work on the events themselves. Do that, and traffic spikes become a graph you glance at — not an incident you get paged for.

Running autoscaling in production? The Kubernetes Production Checklist (free PDF via the newsletter on the homepage) covers the pre-flight checks — requests, probes, disruption budgets and rollback paths — that make scaling safe to automate.

TL;DR: A plain Kubernetes Deployment can roll an update out, but it cannot send 5% of traffic to the new version, watch the error rate, and roll itself back when the numbers turn bad. Argo Rollouts can. It replaces your Deployment with a Rollout resource that adds weighted canary steps, blue-green switching, metric-driven analysis, and automatic rollback — and it slots neatly next to Argo CD if you already run GitOps. This guide installs it, ships a first canary, wires metrics into the promotion decision, and covers the mistakes that bite.

Originally published on DevOps Enginer.

My Kubernetes deployment strategies guide explains the patterns: rolling, blue-green, canary, recreate. Patterns are the easy half. The hard half is execution — who watches the new version while it serves its first real traffic, and who has the nerve (and the speed) to pull it back at 2 a.m. when the error graph twitches?

A stock Deployment answers "nobody, really." It replaces pods on a schedule of percentages of replicas, not traffic. It has no idea whether the new version is healthy in any sense that matters to users. Rollback is a human noticing, deciding, and typing.

Argo Rollouts is the controller that closes that gap. And if you already run Argo CD — I do, and my Argo CD tutorial walks through it — Rollouts is from the same Argo family and feels like the missing half: Argo CD gets the right version to the cluster; Rollouts decides how that version earns the rest of the traffic.

Why a plain Deployment is not enough

Three limits show up the first time a release goes wrong:

1. Replica math is not traffic math. "25% of pods are new" does not mean 25% of users see the new version — behind a Service, it is roughly proportional, but you cannot set 1%, or 5%, or hold a step deliberately.
2. There is no verify step. A Deployment promotes on pod readiness. A pod can be Ready and still return 500s, leak memory, or take 4 seconds per request. Readiness is not health.
3. Rollback is manual and slow. kubectl rollout undo works — after a human has noticed, paged, decided, and typed. The blast radius grows the whole time.

Progressive delivery is the discipline of fixing exactly these three: small traffic steps, automated verification between steps, and rollback that does not wait for a human. That is what Rollouts implements.

What Argo Rollouts actually is

Three pieces:

• The Rollouts controller — a deployment in your cluster that watches Rollout resources and drives ReplicaSets, Services, and (optionally) your ingress or mesh.
• The Rollout resource — a near drop-in replacement for a Deployment. Same pod template, same selectors; the difference is the strategy block, which describes how versions rotate.
• The kubectl plugin — kubectl argo rollouts … for watching, promoting, pausing, and aborting rollouts from the terminal (and a web dashboard if you want one).

One honest nuance up front: precise traffic percentages need a traffic router. With a plain Service, canary steps scale ReplicaSets proportionally, so weights are approximate (with 10 replicas, your steps are effectively 10% chunks). Integrate Rollouts with NGINX Ingress, ALB, Traefik, Istio, or another supported router and the weights become real traffic weights, independent of replica count. Replica-based canary is fine to learn on; traffic-based is what you want for user-facing services.

Install Rollouts and the kubectl plugin

The controller:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# watch it come up
kubectl -n argo-rollouts get pods

The plugin (Linux example — grab the matching binary for your OS from the same releases page):

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

kubectl argo rollouts version

Using latest in a learning cluster is fine. In production, pin the release version in the URL and manage the install through your GitOps repo like everything else — a controller that controls your releases should itself be released carefully.

Your first canary: a Rollout with steps

Take an ordinary Deployment, change kind: Deployment to kind: Rollout (and the apiVersion to argoproj.io/v1alpha1), and add a strategy. This example rolls a new version through 20% → pause for a human look → 50% → indefinite pause → 80% → full:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop-api
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: shop-api
  template:
    metadata:
      labels:
        app: shop-api
    spec:
      containers:
        - name: shop-api
          image: ghcr.io/example/shop-api:2.4.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: {}          # waits for a human promote
        - setWeight: 80
        - pause: { duration: 10m }

Apply it, trigger an update by changing the image tag, and watch:

kubectl argo rollouts get rollout shop-api -n production --watch

You will see the step list with the current step highlighted — this little tree view is, honestly, half the reason people adopt Rollouts. The state of a release stops being something you reconstruct from three commands and becomes something you can see.

Promote past a manual pause, or bail out:

kubectl argo rollouts promote rollout shop-api -n production
kubectl argo rollouts abort rollout shop-api -n production   # back to stable, now
kubectl argo rollouts undo rollout shop-api -n production     # roll back a finished one

Let metrics decide: analysis runs

Manual pauses are a fine start, but they still depend on a human watching a dashboard. The feature that turns Rollouts from "nicer rolling update" into progressive delivery is the AnalysisRun: during a canary step, Rollouts queries your metrics provider on an interval; if the numbers breach your conditions, it aborts and rolls back automatically.

A minimal success-rate check against Prometheus:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 30s
      count: 10
      successCondition: result[0] >= 0.95
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{ args.service-name }}",status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{ args.service-name }}"}[5m]))

Reference it from the Rollout's canary strategy, and the analysis starts automatically at the step you choose:

  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        args:
          - name: service-name
            value: shop-api
        startingStep: 2
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }

Now the 2 a.m. scenario changes shape: the new version starts failing, the success rate dips under 95% twice, and Rollouts aborts and restores the stable ReplicaSet before anyone's phone buzzes. The human reviews a finished, contained event in the morning instead of performing an emergency.

Two disciplines make or break this:

• Analyse what users feel, not what machines feel. Success rate and latency percentiles of real requests beat CPU graphs. (This is the same symptom-over-cause rule from my Kubernetes observability guide — analysis is only as good as the signals you feed it.)
• Your metrics must already exist and be trustworthy. If the service emits no usable request metrics, step one is observability, not Rollouts. An automated rollback wired to a meaningless query is worse than a manual one — it fails with confidence.

Blue-green with Rollouts

Prefer an all-or-nothing switch with an instant way back? Blue-green is a strategy block away:

  strategy:
    blueGreen:
      activeService: shop-api-active
      previewService: shop-api-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300

The new version comes up behind the preview Service, where you can smoke-test it against real infrastructure with zero user traffic. Promote it, and the active Service flips to the new ReplicaSet; the old one stays warm for scaleDownDelaySeconds — your undo window. With autoPromotionEnabled: false, the flip waits for kubectl argo rollouts promote. Set an analysis as a prePromotionAnalysis and even blue-green gets a metric gate before the switch.

My rule of thumb, same as in the strategies guide: canary for stateless user-facing services where gradual is possible; blue-green where a version must switch atomically (or where testing the exact production-bound version beforehand matters more than gradual exposure).

Rollouts + Argo CD: who owns what

If you run GitOps, the division of labour is clean:

• Argo CD syncs the desired state from Git — including the Rollout resource and its new image tag. A release is a commit, with the audit trail Git gives you for free.
• Rollouts owns the progression of that version inside the cluster: steps, pauses, analysis, promotion, abort.

Argo CD understands Rollouts natively — it reports the Rollout's health (Progressing / Healthy / Degraded) instead of the binary in-sync view you'd get with a Deployment mid-rollout, so a canary sitting at 20% doesn't show up as a permanently "progressing" mystery.

One trap: do not let two systems fight. Your CI pipeline should update the image tag in Git (or the Rollout manifest) and then stop — no kubectl promote step in CI for releases you want gated, and no second controller trying to manage the same ReplicaSets. One owner per decision: Git for what version, Rollouts for how fast it earns traffic, humans for the pauses you deliberately kept.

Mistakes that bite

• Canary by replicas, expectations by traffic. With few replicas and no traffic router, a "20%" step can be one pod behind a Service — noisy and not really 20% of anything. If precise percentages matter to the release decision, wire up a traffic router first.
• Analysis wired to vanity metrics. An AnalysisRun checking cluster CPU will happily promote a version that is erroring on every tenth request. Gate on per-service success rate and latency, measured on the canary itself.
• No baseline. Analysis compares the canary against an absolute condition (like the 95% above). If normal for this service is 96% on a good day, a 95% gate aborts healthy releases and trains everyone to ignore aborts. Tune the gate to the service's real baseline — my requests and limits guide makes the same point about numbers copied from other people's clusters.
• Promoting during an incident. If the platform is already on fire, a promote command is not a fix. Rollouts will even keep an in-flight canary paused politely while everything degrades around it. Abort first; release later.
• Forgetting the HPA. Rollouts scales the stable and canary ReplicaSets to honour weights; an HPA targeting the Rollout works (point it at the Rollout, not the ReplicaSets), but an HPA someone left aimed at the old Deployment setup will confuse everyone. One more reason the migration should be a deliberate manifest change, not a weekend experiment.
• revisionHistoryLimit left huge. Old ReplicaSets are your rollback targets — keep a few (2–3). Keep thirty and every get rollout becomes an archaeology dig.

When NOT to use Rollouts

Honestly: plenty of services should stay on plain Deployments.

• Internal tools and low-traffic services where a bad release inconveniences three colleagues, not three thousand customers.
• Batch and cron workloads — there is no traffic to graduate.
• Anything whose metrics are too thin to analyse; a canary you cannot verify is just a slower rolling update.
• Teams still fighting basic stability. Progressive delivery amplifies a release process; it does not repair one. Get deployments boring first (the troubleshooting guide is the place to start), then add graduation.

The measure I use, from the project-manager side of my desk: does a bad release of this service cost more than the machinery to catch it early? For the public, revenue-adjacent, or reputation-carrying services, Rollouts pays for itself the first time it aborts at step one. For everything else, it's ceremony.

FAQ

Is Argo Rollouts a replacement for Argo CD? No — they are siblings, not competitors. Argo CD delivers desired state from Git to the cluster. Rollouts controls how a new version progresses once it's there. Many teams run both; Rollouts also works without Argo CD if your delivery is plain kubectl or another CD tool.

Can I use Rollouts without an ingress controller or service mesh? Yes. You get replica-proportional canary steps through Services, which is approximate but functional — fine for learning and for low-stakes services. Precise, replica-independent traffic weights need a supported traffic router (NGINX, ALB, Traefik, Istio, and others).

What happens to my Deployment manifests? You convert them: kind: Rollout replaces kind: Deployment, the pod template carries over unchanged, and you add the strategy block. Rollouts manages its own ReplicaSets from there. The conversion is small; the thinking about steps and gates is the real work.

Does an aborted canary count as downtime? No — aborting shifts traffic back to the stable ReplicaSet, which never went away. Users saw the canary's behaviour only for the share of traffic it held, which is precisely the blast-radius reduction you adopted this for.

Can analysis roll back a blue-green release too? Yes — pre-promotion analysis gates the switch, and post-promotion analysis can trigger a rollback to the previous ReplicaSet while it's still within its scale-down window.

Conclusion

A Deployment rolls software out. Argo Rollouts rolls software out on probation: a small share of traffic, real metrics watching, automatic retreat when the numbers say so. Install the controller, convert one service, start with replica-based canary steps and a manual pause — then wire in analysis and graduate to "the release promotes itself, and pages nobody."

Ship the versions. Let the evidence promote them.

Keep going: Deployment Strategies · Argo CD Tutorial · Kubernetes Observability · Kubernetes Production Checklist (free PDF)