Stories From the Trenches
Field notes from production systems that did not behave the way the docs said they would.
Names, repos, dates, and specifics have been filed off. Every one of these is real. Each was painful exactly once.
The Three-Headed Message Broker
# Three operators, two registries, zero reconciliation.
A staging message broker cluster managed by three different control surfaces that didn't know about each other. It rotted in plain sight for ten months, then a mystery kubectl apply rotated the brokers onto a new major version and broke every client still pinned to a two-year-old library.
Deleting a Variable Doesn't Delete It
# An overlay file that's actually a state file pretending to be a snapshot.
If you delete a variable from your environment values file and redeploy, the variable will still be set in production. This is not a bug in your project. It is, however, a bug in the merge semantics that the entire deploy pipeline is built around.
Sixty Services, Ninety Pull Requests, One Silent Regression
# How a routine deploy-all silently rolled a service backward, where it still sits.
A staging deploy-all of sixty services produced ninety auto-merged pull requests, saturated the runner pool, cancelled builds mid-flight, and silently rolled at least one service backward to an older chart version. The downgrade is still live in staging.
Two Graveyards, One Repository
# 00-DISABLED/ and disabled/. Nobody documented the difference.
A single repo contained two parallel directories of soft-deleted services, several dormant environments still wired into auto-merge, template READMEs from the Helm v2 era, silent symlink aliases, and two mysterious zero-byte files at the root that nobody could explain.
Four Ways to Make ArgoCD Notice You Exist
# Three ApplicationSets, plus "I wrote the Application by hand."
There wasn't one way services got registered with ArgoCD. There were three ApplicationSets and a hand-rolled escape hatch. Each one globbed for a different trigger filename. Forget the trigger file and the deploy silently does nothing.
Just... Why?
# An entire piece of infrastructure whose documentation is a single shrug.
There was an internal chart repository that nobody could explain. The official documentation, in its entirety, was the question "Just... why?"
The Forced Upgrade
# The cloud provider upgraded the cluster for us. It did not go well.
An older Kubernetes minor version reached end-of-life. The cloud provider forcibly upgraded the production clusters. Several components had not been prepared for the new minor version, message broker credentials evaporated, persistent volume claims got stuck, and customer-facing services were down for hours.