Build fast and break things: A lesson in Archeology

<p>My business infrastructure started as a homelab and scaled into the backbone of my work. ingress-nginx recently hit end-of-life, and what I thought was going to be an ingress migration turned into an audit of my infrastructure.</p><h3 id="the-deadline-forced-the-move-the-audit-changed-the-job">The Deadline Forced the Move. The Audit Changed the Job. </h3><p>ingress-nginx hit end-of-life in March 2026. No more releases, no more patches, no more security fixes. That forced the move. On its own, not much of a story. Controllers get replaced all the time. I had already started to outgrow ingress-nginx as my service architecture evolved, so the migration made sense anyway. The story started when I checked the cluster instead of trusting the version of it I had in my head.</p><p>This is a k3s cluster running self-hosted services behind Cloudflare, with Flux managing most of the GitOps. I went into the migration thinking I was swapping ingress controllers: install Istio, move apps over in phases, leave the critical stuff alone for now, clean up later. That plan didn't survive the audit.</p><p>The documented cluster and the real cluster had drifted apart as I had expanded it from a toy to my backbone. The repo described one ingress path and a manageable set of apps. The cluster had three controllers in play, more Ingress resources than I'd planned for, and a few pieces of infrastructure that couldn't move without turning the whole project into a larger outage risk. ingress-nginx was going to have to stay around longer than I wanted.</p><p>But the resource count was the surface problem. Underneath it, the cluster had organized itself around nginx in ways I hadn't properly kept track of during my early days. Network policies trusted the ingress-nginx namespace directly. cert-manager annotations were built around nginx handling ACME challenges. Auth flows were buried in controller-specific annotations. The migration had to account for all of it before any install work could start.<br><br>I had built a system with more hidden coupling than I realized, and it was time to untangle.</p><h3 id="why-istio-ambient">Why Istio Ambient</h3><p>Gateway API was the obvious direction for my growing mix of services.</p><p>A gateway-only replacement - Envoy Gateway, Traefik, NGINX Gateway Fabric - would have solved the immediate problem. That would have been fine if this cluster were going to stay mostly edge-routed apps forever. It isn't. The architecture is moving toward microservices, and once that happens the hard traffic problems are no longer just at the boundary. Identity between services matters. mTLS matters. Policy and observability inside the cluster matter. When I started, SLAs were not part of my decision-making process. Now they are.</p><p>Part of the problem with scaling a business yourself is that you become the everything engineer. That works fine until you hit a part of the stack that, in FAANG world, has entire teams devoted to it. Istio is a monster: too much overhead, too much ceremony, too much commitment up front. But it has strong selling points, and ambient mode blunts some of its sharper edges. Ambient mode replaces sidecars with per-node ztunnel DaemonSets for L4 and optional waypoint proxies for L7. The overhead drops from per-pod to per-node, which makes the cost predictable and the commitment much easier to justify. I can adopt the gateway now and leave the mesh untouched until the architecture actually needs it. The migration only has to happen once.</p><p>I can set up Istio now for the easy parts and go deep where it matters as I expand.</p><h3 id="the-hardest-problems-werent-in-the-install-guide">The Hardest Problems Weren't in the Install Guide.</h3><p>The actual Istio install was fine. Four Helm charts, dependency ordering through Flux, nothing surprising.</p><p>The surprises started where the install met the platform.</p><p><strong>k3s, exactly like k8s, except where it isn't</strong>. Istio’s CNI assumes the config will be at <code>/etc/cni/net.d</code>, exactly where Kubernetes normally stores it. k3s stores it at <code>/var/lib/rancher/k3s/agent/etc/cni/net.d/</code>. CNI pods crashed on most nodes until the Helm values were adjusted. It's documented, but k3s is close enough to standard Kubernetes that it's easy to forget where it isn't, right up until something assumes the usual layout and falls over.</p><p><strong>The spec and the implementation disagree. </strong>Gateway API allows unlimited <code>certificateRefs</code> per TLS listener. Istio caps it at 2. I had services that needed more than two certificates on a single listener. Adding the third threw the gateway into an unprogrammed state - connection refused, all associated apps down. The fix was per-hostname listeners with SNI matching, which is actually the cleaner design. But it caught me off guard because I had trusted the spec over the implementation.</p><p><strong>Resource ordering has invisible dependencies. </strong>Istio's external authorization requires an <code>extensionProvider</code> defined in mesh config before an <code>AuthorizationPolicy</code> can reference it. Apply the policy before istiod reconciles the config update and every request returns 503 - Envoy can't find a provider it hasn't learned about yet. Nothing in Kubernetes stops you from doing this. You just find out when requests fail in a way that's technically correct and operationally obnoxious.</p><p>The cluster had been carrying assumptions I hadn’t realized were there: inconsistent tuning, spec-versus-implementation gaps, and invisible ordering. The new system stopped covering for them.</p><p>I am the everything engineer, and I've learned my trade through headaches and eye strain.</p><h3 id="gateway-api-forced-hidden-behavior-into-the-open">Gateway API Forced Hidden Behavior Into the Open.</h3><p>This is the part I care about most now that the migration is done.</p><p>Under ingress-nginx, routing behavior, auth, and TLS wiring were compressed into annotations. Compact to configure, hard to see. They existed without ever feeling like first-class parts of the system.</p><p>Gateway API changed that.</p><p>HTTP-to-HTTPS redirect stopped being an annotation repeated on every Ingress and became an actual route - one policy, one place, visible with <code>kubectl get httproute</code> instead of buried in config you have to grep for.</p><p>Auth changed more. Forward authentication had been two annotation lines. It looked simple because nginx was compressing the complexity. Under Istio, it expanded into what it actually was: an <code>extensionProvider</code> defining how to reach the auth service, an <code>AuthorizationPolicy</code> scoping which requests get checked, and an <code>HTTPRoute</code> for the OAuth callback path. That forced me to understand that my Authentik outpost is a gatekeeper, not a proxy - it handles auth checks and OAuth flow, it doesn't sit in the request path. Envoy consults it, then routes directly. The old annotations let me avoid thinking about that. The new model didn't. </p><p>Bootstrap ordering moved out of my head into code. The dependency chain, Istio depends on cert-manager config and MetalLB, the gateway config on Istio - all this lives in a Flux <code>dependsOn</code> declaration now. I could never go back to my pre-GitOps days. </p><p>In a cluster moving toward microservices, service-to-service traffic, auth boundaries, and routing policy can't stay buried in controller-specific annotations. Gateway API and Istio forced that behavior into resources that can be inspected, reasoned about, and rebuilt from scratch.</p><hr><p>As I've moved my stack homelab to small business, "I know what has to come up first" isn't good enough, I'm chasing 9s now. </p>