Cloudflare doesn't like chunked uploads, but their workers can help

<h2 id="context">Context</h2><p>While operating a self-hosted GitLab Container Registry behind CloudFlare Tunnel with proxy enabled, I encountered intermittent container image push failures from Podman and Docker. Image pulls functioned normally; only pushes failed with digest verification errors.</p><h2 id="symptoms">Symptoms</h2><p>Push operations returned digest mismatches where the registry computed a hash for empty or truncated content:</p><pre><code class="language-bash">podman push gitlab-registry.scopecreep.productions/project/image:latest Error: writing blob: uploading layer chunked: digest mismatch Expected: sha256:abc123... Actual: sha256:e3b0c4... # SHA256 of empty string </code></pre><p>At first I was convinced I'd broken my registry config. I rolled back a week of changes. Nothing. The "aha" was noticing the zero-byte blobs in the registry logs - no disconnects, no retries, just missing data. The uploads were being silently truncated somewhere between my client and the registry.</p><h2 id="investigation">Investigation</h2><h3 id="tunnel-configuration-check-dead-end-1">Tunnel Configuration Check (Dead End #1)</h3><p>Initial hypothesis focused on CloudFlare Tunnel settings, specifically the <code>disableChunkedEncoding</code> configuration option. I spent about 45 minutes tweaking tunnel configs before realizing this flag only affects tunnel→origin traffic, not client→CloudFlare. The failure was happening before requests even reached my tunnel.</p><h3 id="reproduction-by-payload-size">Reproduction by Payload Size</h3><p>I isolated the transfer mechanism using raw HTTP/1.1 chunked uploads to identify the failure threshold:</p><pre><code class="language-bash"># Small upload (1MB) - SUCCESS dd if=/dev/urandom bs=1M count=1 | curl -X POST \ -H "Transfer-Encoding: chunked" \ --data-binary @- \ https://gitlab-registry.scopecreep.productions/test # Medium upload (3MB) - FAILURE at ~1.2MB dd if=/dev/urandom bs=1M count=3 | curl -X POST \ -H "Transfer-Encoding: chunked" \ --data-binary @- \ https://gitlab-registry.scopecreep.productions/test # 502 Bad Gateway </code></pre><h3 id="control-test-with-content-length">Control Test with Content-Length</h3><p>Using explicit <code>Content-Length</code> headers avoided the failure:</p><pre><code class="language-bash">TEMP=$(mktemp) dd if=/dev/urandom of=$TEMP bs=1M count=5 curl -X POST \ -H "Content-Length: $(stat -c%s $TEMP)" \ --data-binary @$TEMP \ https://gitlab-registry.scopecreep.productions/test # Success - full 5MB uploaded </code></pre><h3 id="test-results-summary">Test Results Summary</h3>  <table> <thead> <tr> <th>Test Type</th> <th>Payload Size</th> <th>Result</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>Chunked encoding</td> <td>1MB</td> <td>Pass</td> <td>Full upload</td> </tr> <tr> <td>Chunked encoding</td> <td>3MB</td> <td><strong>Fail</strong></td> <td>502 at ~1.2MB</td> </tr> <tr> <td>Content-Length</td> <td>5MB</td> <td>Pass</td> <td>Full upload</td> </tr> <tr> <td>HTTP/2</td> <td>5MB</td> <td>Pass</td> <td>No truncation</td> </tr> <tr> <td>Internal (no CF)</td> <td>Any size</td> <td>Pass</td> <td>No limitations</td> </tr> </tbody> </table>  <h2 id="root-cause">Root Cause</h2><p>CloudFlare's proxied edge path truncates HTTP/1.1 chunked-transfer uploads around 1-2MB in this configuration. The registry receives incomplete content, calculates the SHA256 digest of the partial blob, and returns a digest mismatch error. <strong>This behavior is not documented anywhere in CloudFlare's official docs.</strong></p><h3 id="why-container-registries-hit-this">Why Container Registries Hit This</h3><p>Docker, Podman, and Skopeo stream image layer blobs using <code>Transfer-Encoding: chunked</code> because the total size is often unknown upfront. When the proxy silently truncates mid-upload:</p><ol><li>Registry receives partial data</li><li>Registry computes SHA256 of incomplete content</li><li>Digest verification fails</li><li>Client sees only the mismatch error, not the truncation</li></ol><h2 id="solution-options">Solution Options</h2><h3 id="option-1-hybrid-pushpull-paths-what-i-implemented">Option 1: Hybrid Push/Pull Paths (What I Implemented)</h3><p>Use different URLs for push vs. pull operations.</p><ul><li><strong>Push:</strong> Internal Kubernetes service URL (bypasses CloudFlare)</li><li><strong>Pull:</strong> External URL through CloudFlare proxy</li></ul><pre><code class="language-bash"># CI/CD pushes docker tag myimage:latest \ gitlab-registry.gitlab.svc.cluster.local:5000/project/image:latest docker push \ gitlab-registry.gitlab.svc.cluster.local:5000/project/image:latest # Kubernetes deployments pull via external URL containers: - image: gitlab-registry.scopecreep.productions/project/image:latest </code></pre><p><strong>Why I picked this:</strong></p><ul><li>GitLab CI already runs in-cluster, so no VPN needed</li><li>Free, zero infrastructure changes</li><li>Doesn't punch a hole in my threat model (CloudFlare still protects pulls)</li></ul><p><strong>Downside:</strong> Remote pushes from my laptop need SSH tunnel access to the cluster.</p><h3 id="option-2-cloudflare-worker-buffering">Option 2: CloudFlare Worker Buffering</h3><p>Deploy an edge function to buffer chunked requests and re-emit with <code>Content-Length</code>.</p><pre><code class="language-javascript">export default { async fetch(request, env, ctx) { const body = await request.arrayBuffer() const newHeaders = new Headers(request.headers) newHeaders.delete('Transfer-Encoding') newHeaders.set('Content-Length', body.byteLength) return fetch(request.url, { method: request.method, headers: newHeaders, body: body }) } } </code></pre><p><strong>Why I didn't pick this:</strong> Requires CloudFlare dashboard access (corporate account, slow approvals) and adds latency buffering the entire upload before forwarding.</p><h3 id="option-3-dns-only-mode">Option 3: DNS-Only Mode</h3><p>Disable CloudFlare proxy for the registry subdomain.</p><p><strong>Why I didn't pick this:</strong> Exposes my home IP and loses DDoS protection. Not worth it for a hobby project registry.</p><h2 id="production-implementation">Production Implementation</h2><p>I went with the hybrid path approach. GitLab CI config change was straightforward:</p><pre><code class="language-yaml">variables: REGISTRY_PUSH: "gitlab-registry.gitlab.svc.cluster.local:5000" REGISTRY_PULL: "gitlab-registry.scopecreep.productions" build: script: - docker build -t $REGISTRY_PUSH/$CI_PROJECT_PATH:$CI_COMMIT_SHA . - docker push $REGISTRY_PUSH/$CI_PROJECT_PATH:$CI_COMMIT_SHA deploy: script: - kubectl set image deployment/app \ app=$REGISTRY_PULL/$CI_PROJECT_PATH:$CI_COMMIT_SHA </code></pre><p>Pushes use internal cluster networking (no CloudFlare), pulls use the external URL (CloudFlare caching/protection). Zero CloudFlare config changes needed.</p><h2 id="whos-likely-to-trip-over-this">Who's Likely to Trip Over This</h2><p>If you run container registries behind CloudFlare proxy and use Docker/Podman/Skopeo for image pushes, you'll probably hit this when pushing layers larger than ~1-2MB over HTTP/1.1.</p><p>Mysterious "digest mismatch" errors are the tell.</p><h2 id="tldr">tl;dr </h2><h3 id="1-i-assumed-cloudflare-would-pass-chunked-uploads-through-unchanged">1. I Assumed CloudFlare Would Pass Chunked Uploads Through Unchanged</h3><p>That was wrong. The failure mode was silent enough to waste hours chasing registry configs and Docker bugs. Proxies are not transparent - they have their own optimizations and undocumented limits.</p><h3 id="2-for-large-uploads-behind-cloudflare-prefer-http2-or-explicit-content-length">2. For Large Uploads Behind CloudFlare: Prefer HTTP/2 or Explicit Content-Length</h3><p>HTTP/1.1 chunked encoding is a minefield with proxies:</p><ul><li>Prefer HTTP/2 if your stack supports it</li><li>Use explicit <code>Content-Length</code> when you know the size upfront</li><li>Test with realistic payload sizes in staging</li><li>Don't assume chunked encoding will "just work"</li></ul><h3 id="3-i-brute-forced-the-variables-until-it-gave-up">3. I Brute-Forced the Variables Until It Gave Up</h3><p>The breakthrough came from systematic variable isolation:</p>  <table> <thead> <tr> <th>Variable</th> <th>Test Cases</th> </tr> </thead> <tbody> <tr> <td><strong>Size</strong></td> <td>1MB vs. 3MB vs. 5MB</td> </tr> <tr> <td><strong>Encoding</strong></td> <td>chunked vs. Content-Length</td> </tr> <tr> <td><strong>Path</strong></td> <td>through CloudFlare vs. direct</td> </tr> <tr> <td><strong>Protocol</strong></td> <td>HTTP/1.1 vs. HTTP/2</td> </tr> </tbody> </table>  <p>This methodical approach pinpointed the exact failure conditions in about 2 hours. Way faster than randomly tweaking configs.</p><h3 id="4-logs-at-every-layer-were-essential">4. Logs at Every Layer Were Essential</h3><p>I had logging enabled at:</p><ul><li>CloudFlare Tunnel</li><li>Registry application</li><li>Client verbose output</li><li>tcpdump for sanity checks</li></ul><p>Each layer gave me a piece of the puzzle. The registry logs showing zero-byte blobs were the smoking gun.</p><h2 id="technical-environment">Technical Environment</h2><p>This serves my <strong>OTA (Over-The-Air) firmware distribution system</strong> for IoT devices:</p>  <table> <thead> <tr> <th>Component</th> <th>Technology</th> </tr> </thead> <tbody> <tr> <td><strong>Backend</strong></td> <td>Python/Flask REST API</td> </tr> <tr> <td><strong>Storage</strong></td> <td>MinIO S3-compatible object storage</td> </tr> <tr> <td><strong>Registry</strong></td> <td>GitLab Container Registry</td> </tr> <tr> <td><strong>Proxy</strong></td> <td>CloudFlare Tunnel</td> </tr> <tr> <td><strong>Orchestration</strong></td> <td>Kubernetes (k3s)</td> </tr> <tr> <td><strong>CI/CD</strong></td> <td>GitLab CI with multi-stage Docker builds</td> </tr> </tbody> </table>  <p>The firmware service implements authenticated firmware distribution with SHA256 integrity verification, semantic versioning, and automated device updates. Progress is being made on this, I'll post more updates eventually. </p><h2 id=""></h2><h2 id="-1"></h2>