Moving Cloudflare tunnels from E2Edge to E2E

<p></p><h2 id=""></h2><p>I like self hosting, I enjoy managing my own infrastructure, I do not like posting my IP on the internet. Obviously, that creates some issues. Obfuscating your address is normally not too much of an issue, there are plenty of ways to do it, I utilize cloudflare zero-trust tunnels. I get the benefits from Cloduflare DNS, DDoS Protection, etc, and I get to utilize my stack from anywhere via the Cloudflare<em>d</em>. </p><p>Initially I set everything up as E2Edge, I would terminate my TLS at the edge then just navigate internally with plain HTTP; not the most secure, but also not a big deal with the way my architecture. Here's to you, Authentik! Genuinely one of the better services I've deployed. </p><p>This setup worked fine, but as I grew I realized there are genuine benefits to hosting your own container registry - you can utilize intra-cluster speeds instead of ISP bandwidth which can often be painfully slow when uploading containers. By utilizing intracluster communication I was able to increase the speed of the uploads and downloads of my containers with a hairpin DNS setup via coredns, but I still wanted to be able to externally access these containers for remote services or allowing friends of mine to utilize things I've built. </p><p>Sounds reasonable enough?</p><h2 id="the-problem-gitlab-container-registry-pushing-expects-https">The problem: Gitlab container registry pushing <strong><em>expects </em></strong>HTTPS</h2><h2 id="-1"></h2><p>Registry is set up, I'm authenticated, image pulls work, image pushes... fail? </p><pre><code class="language-bash">$ podman push gitlab-registry.example.com/project/image:latest Error: authenticating creds for "gitlab-registry.example.com": authentication required </code></pre><p>I know it's authenticated. I can pull containers that require authentication? What's going on here. </p><h3 id="the-debugging-spiral">The Debugging Spiral</h3><p>I chased a few wrong things, validating the registry configuration, testing ingress timeouts, confirming the pumas are working. Who knew there were so many large cats in a gitlab stack. </p><p>Now, Docker and Podman authenticate to registries using a challenge-response flow:</p><ol><li>Client requests a push</li><li>Registry responds with <code>401 Unauthorized</code> and a <code>Www-Authenticate</code> header pointing to a token endpoint</li><li>Client hits the token endpoint to get a bearer token</li><li>Client retries the push with the token</li></ol><p>Taking a step back I captured the HTTPS exchange directly to diagnose from there and found that the registry was returning the token endpoint URL as <code>http://</code> instead of <code>https://</code>. My client dutifully followed that HTTP URL, but Cloudflare (correctly!) redirected HTTP to HTTPS. The redirect broke the auth flow.</p><h3 id="whats-happening-here">What's happening here? </h3><p>GitLab's container registry generates URLs based on what it sees in the incoming request. When ingress-nginx forwarded the request over plain HTTP, the registry saw:</p><ul><li>Protocol: <code>http</code></li><li>Host: <code>gitlab-registry.example.com</code></li></ul><p>So it generated <code>http://gitlab-registry.example.com/...</code> in its responses.</p><p>The registry had no way to know the original request came in over HTTPS through Cloudflare. From its perspective, HTTP was correct.</p><hr><h2 id="the-workaround-x-forwarded-proto">The Workaround: X-Forwarded-Proto</h2><p>The quick fix was telling the registry what protocol the client actually used. You can configure ingress-nginx to inject headers via annotations on the Ingress:</p><pre><code class="language-yaml">apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gitlab-registry annotations: nginx.ingress.kubernetes.io/configuration-snippet: | proxy_set_header X-Forwarded-Proto https; spec: # ... rest of ingress config </code></pre><p>This tells the registry: "Even though you received this over HTTP, the original client used HTTPS." The registry then generates HTTPS URLs in its responses.</p><p>This works, but if I stopped at "Working" I'd still be hosting individually managing a bunch of boxes directly instead of accidentally becoming qualified enough for a CKA certification. </p><p>The <code>X-Forwarded-Proto</code> approach has problems:</p><ol><li><strong>Every application needs to respect it.</strong> Some do, some don't, some need explicit configuration. I'd have to remember this for every new service.</li><li><strong>It's lying to the application.</strong> The registry <em>thinks</em> it's serving HTTPS, but it's actually serving HTTP. If any component along the path doesn't properly forward the header, things break silently.</li><li><strong>The traffic is actually plaintext.</strong> Between Cloudflare and my cluster, traffic was unencrypted. My ISP, anyone on my local network, or any compromised tunnel component could see it.</li><li><strong>GitLab wasn't the only problem.</strong> I hit similar issues with other applications that generate URLs based on the incoming protocol. OAuth callbacks, webhook URLs, asset links - lots of things break when the protocol doesn't match expectations.</li></ol><p>I wanted actual HTTPS all the way through. Not "pretend HTTPS via headers."</p><h2 id="the-real-fix-tls-to-the-origin">The Real Fix: TLS to the Origin</h2><p>The solution was straightforward in concept: just go fully E2E</p><p>This meant I needed certificates for my internal services. Real ones, not self-signed - Cloudflare validates origin certificates, and I didn't want to deal with certificate trust issues.</p><h3 id="what-i-needed">What I Needed</h3><ul><li><strong>Automated certificate issuance</strong> - I wasn't going to manually request certs for every service</li><li><strong>Automated renewal</strong> - Let's Encrypt certs expire every 90 days</li><li><strong>Works behind the tunnel</strong> - No public port 80 for HTTP-01 challenges (or so I thought)</li><li><strong>Works for internal services</strong> - Some services aren't exposed through Cloudflare at all</li></ul><hr><h2 id="the-solution-cert-manager-with-multiple-issuers">The Solution: cert-manager with Multiple Issuers</h2><p>cert-manager is a Kubernetes add-on that handles certificate lifecycle. You define "Issuers" that know how to get certificates, and cert-manager does the rest.</p><p>I set up two types of issuers for different use cases.</p><h3 id="http-01-for-public-services">HTTP-01 for Public Services</h3><p>HTTP-01 challenges work by Let's Encrypt requesting a specific URL on your domain. cert-manager creates a temporary pod to respond.</p><p>Here's the surprising thing: <strong>this works through Cloudflare Tunnel.</strong> When Let's Encrypt hits <code>http://gitlab.example.com/.well-known/acme-challenge/&lt;token&gt;</code>, Cloudflare proxies the request through my tunnel to ingress-nginx, which routes to cert-manager's solver pod.</p><p>I didn't need port 80 exposed on my router. The tunnel handles it.</p><pre><code class="language-yaml">apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: [email protected] privateKeySecretRef: name: letsencrypt-prod-key solvers: - http01: ingress: class: nginx </code></pre><p><strong>Gotcha:</strong> This only works if Cloudflare is set to "Proxied" (orange cloud) mode. In DNS-only mode, traffic goes directly to whatever IP is in the record - which isn't your cluster.</p><h3 id="dns-01-for-internal-services-and-wildcards">DNS-01 for Internal Services and Wildcards</h3><p>Some services aren't exposed through Cloudflare at all - internal admin UIs, monitoring dashboards, etc. HTTP-01 can't reach them.</p><p>DNS-01 solves this by proving domain ownership through DNS TXT records. cert-manager uses the Cloudflare API to create the records, Let's Encrypt verifies them, done. No HTTP traffic required.</p><pre><code class="language-yaml">apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-cloudflare-prod spec: acme: email: [email protected] server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-cloudflare-key solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token key: api-token selector: dnsZones: - "example.com" </code></pre><p>DNS-01 also unlocks wildcard certificates (<code>*.example.com</code>), which HTTP-01 can't do. </p><p>For an internal service not exposed through Cloudflare, you'd use this issuer instead:</p><pre><code class="language-yaml">apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: internal-dashboard annotations: cert-manager.io/cluster-issuer: letsencrypt-cloudflare-prod spec: ingressClassName: nginx tls: - hosts: - dashboard.internal.example.com secretName: dashboard-tls # ... </code></pre><p><strong>Security note:</strong> The Cloudflare API token needs <code>Zone:DNS:Edit</code> permission. Scope it to only the zones you need. </p><h3 id="using-certificates-in-ingresses">Using Certificates in Ingresses</h3><p>Once the issuers are configured, getting a certificate is one annotation:</p><pre><code class="language-yaml">apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gitlab annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: ingressClassName: nginx tls: - hosts: - gitlab.example.com secretName: gitlab-tls rules: - host: gitlab.example.com http: paths: - path: / pathType: Prefix backend: service: name: gitlab port: number: 8080 </code></pre><p>cert-manager sees the annotation, creates a Certificate resource, handles the ACME challenge, and stores the cert in the <code>gitlab-tls</code> secret. Renewals happen automatically!</p><h2 id="fixing-gitlab">Fixing GitLab</h2><p>With TLS actually terminating at ingress-nginx, I could configure GitLab properly.</p><h3 id="the-gitlabrb-configuration">The gitlab.rb Configuration</h3><pre><code class="language-ruby"># External URL is HTTPS (what users see) external_url 'https://gitlab.example.com' # But GitLab's internal nginx listens on HTTP # because ingress-nginx handles TLS termination nginx['listen_address'] = '0.0.0.0' nginx['listen_port'] = 80 nginx['listen_https'] = false nginx['redirect_http_to_https'] = false # Registry also listens on HTTP internally registry['enable'] = true registry_external_url 'https://registry.gitlab.example.com' registry['registry_http_addr'] = "0.0.0.0:5000" registry_nginx['enable'] = false </code></pre><p>This might look contradictory: external URLs are HTTPS, but nginx listens on HTTP. The key is that ingress-nginx terminates TLS <em>before</em> traffic reaches GitLab. GitLab sees HTTP requests from ingress-nginx, but the client's connection to the Ingress was actually HTTPS.</p><p>The difference from before: the hop between Cloudflare and ingress-nginx is now TLS instead of HTTP. When ingress-nginx sets <code>X-Forwarded-Proto: https</code>, it's telling the truth - the client really did connect over HTTPS. GitLab generates correct HTTPS URLs because the header accurately reflects the client's connection.</p><h3 id="registry-ingress">Registry Ingress</h3><pre><code class="language-yaml">apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gitlab-registry annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/proxy-body-size: "0" # unlimited for large images nginx.ingress.kubernetes.io/proxy-read-timeout: "900" spec: ingressClassName: nginx tls: - hosts: - gitlab-registry.example.com secretName: gitlab-registry-tls rules: - host: gitlab-registry.example.com http: paths: - path: / pathType: Prefix backend: service: name: gitlab-registry port: number: 5000 </code></pre><p>After applying this, the auth failures were gone the registry now generates correct HTTPS URLs because the <code>X-Forwarded-Proto</code> header accurately reflects the client's connection.</p><p>This fixed my HTTPS issues, but I immediately stumbled into another issue with the way that Cloudflare truncates chunked transfers. Docker and Podman stream layer blobs using <code>Transfer-Encoding: chunked</code> and Cloudflare silently cuts them off mid-upload. I wrote about that separately in <a href="https://ghost.scopecreep.productions/blog/cloudflare-chunked-transfer-mystery">Debugging CloudFlare Chunked Upload Limits</a>.</p><p></p><p>This isn't perfect end-to-end encryption - Cloudflare still terminates TLS at the edge and can inspect traffic. But it removes the class of bugs where applications generate wrong URLs because they think they're serving HTTP.</p><p></p><h2 id="troubleshooting-when-certificates-dont-issue">Troubleshooting: When Certificates Don't Issue</h2><p>cert-manager is usually reliable, but when things go wrong, here's how to debug:</p><h3 id="check-the-certificate-resource">Check the Certificate Resource</h3><pre><code class="language-bash">kubectl describe certificate gitlab-tls -n gitlab </code></pre><p>Look for status conditions. "Issuing certificate as Secret does not exist" is normal - it means cert-manager is working on it.</p><h3 id="check-for-challenges">Check for Challenges</h3><pre><code class="language-bash">kubectl get challenges -A </code></pre><p>If there's no Challenge resource, cert-manager couldn't match a solver. Check your ClusterIssuer's ingress class or DNS zone selector.</p><h3 id="for-http-01-verify-the-challenge-url-works">For HTTP-01: Verify the Challenge URL Works</h3><pre><code class="language-bash">curl -v http://gitlab.example.com/.well-known/acme-challenge/test </code></pre><p>If this doesn't reach your cluster:</p><ul><li>Is Cloudflare set to Proxied (orange cloud)?</li><li>Is the tunnel running?</li><li>Is the ingress class correct?</li></ul><h3 id="for-dns-01-check-the-txt-record">For DNS-01: Check the TXT Record</h3><pre><code class="language-bash">dig TXT _acme-challenge.gitlab.example.com </code></pre><p>If the record doesn't exist, check cert-manager logs for Cloudflare API errors:</p><pre><code class="language-bash">kubectl logs -n cert-manager deploy/cert-manager | grep -i cloudflare </code></pre><p></p><h2 id="tldr">tl;dr </h2><p></p><h3 id="1-trusted-internal-http-creates-hidden-dependencies">1. "Trusted Internal HTTP" Creates Hidden Dependencies</h3><p>The header-based workaround (<code>X-Forwarded-Proto</code>) worked, but it required every application to correctly handle the header. Some applications need explicit configuration. Some ignore it entirely. Each new service was a potential landmine.</p><p>Actual TLS to the origin eliminated this class of problems. Applications don't need special configuration to generate correct URLs - they just see HTTPS because that's what the connection actually is.</p><h3 id="2-the-tunnel-doesnt-change-http-01">2. The Tunnel Doesn't Change HTTP-01</h3><p>I assumed I'd need DNS-01 for everything because "port 80 isn't exposed." But Cloudflare Tunnel proxies HTTP just fine. The Let's Encrypt servers talk to Cloudflare's edge, not directly to my IP. HTTP-01 works for any service that's publicly accessible through the tunnel.</p><p>I still use DNS-01 for internal services and as a backup, but HTTP-01 handles most cases.</p><h3 id="3-cert-managers-ingressshim-is-worth-using">3. cert-manager's IngressShim is Worth Using</h3><p>I started by creating Certificate resources manually - separate YAML files for each service. This was tedious and error-prone.</p><p>IngressShim lets you just annotate the Ingress with <code>cert-manager.io/cluster-issuer: letsencrypt-prod</code>. cert-manager automatically creates the Certificate resource. Adding TLS to a new service went from "copy Certificate YAML, edit names, apply" to "add one annotation."</p><h3 id="4-the-registry-problem-would-have-happened-elsewhere">4. The Registry Problem Would Have Happened Elsewhere</h3><p>GitLab's registry was the first thing to break obviously, but the same issue - HTTP internally, HTTPS externally - affects anything that generates URLs:</p><ul><li>OAuth callback URLs</li><li>Webhook endpoints</li><li>Asset URLs in HTML responses</li><li>API responses with <code>Location</code> headers</li></ul><p>Fixing it at the transport layer (actual TLS) was more robust than fixing it application by application.</p><p></p><p></p><h2 id="my-stack">My Stack</h2><p>For reference, here's what I'm running</p> <!--kg-card-begin: html--> <table> <thead> <tr> <th>Component</th> <th>What I'm Using</th> </tr> </thead> <tbody> <tr> <td><strong>Cluster</strong></td> <td>K3s</td> </tr> <tr> <td><strong>Certificate Management</strong></td> <td>cert-manager 1.x</td> </tr> <tr> <td><strong>Ingress Controller</strong></td> <td>ingress-nginx</td> </tr> <tr> <td><strong>Edge/Tunnel</strong></td> <td>Cloudflare Tunnel</td> </tr> <tr> <td><strong>DNS</strong></td> <td>Cloudflare</td> </tr> <tr> <td><strong>CA</strong></td> <td>Let's Encrypt</td> </tr> <tr> <td><strong>GitLab CE</strong></td> <td>Omnibus </td> </tr> </tbody> </table> <!--kg-card-end: html-->