WebSocket "Socket Hang Up" / Empty Reply on Ingress Controller v0.32.5 with Internal LB Upstream

Support
,
1

Environment

  • Pomerium Ingress Controller: v0.32.5
  • Infrastructure: GKE Private Cluster
  • Upstream: Google Internal Application Load Balancer (L7) via Serverless Network Endpoint Group (PSC) to Cloud Run.
  • Traffic Flow: Browser → Regional NLB (L4) → Pomerium Pod → VPC → Google Internal LB → PSC → Cloud Run.

The Issue
WebSocket upgrade requests (Connection: Upgrade, Upgrade: websocket) to any application behind Pomerium fail immediately. The connection is dropped
before any data is exchanged and without appearing in the Pomerium access logs.

Logs & Error Output

  1. Pomerium Ingress Controller Logs:
    The controller successfully recognizes the Ingress annotation and attempts to configure the route for WebSockets:

1 {
2 “level”: “info”,
3 “cluster-id”: “vibe-app-web-socket-other-web-socket-other-web-socket-other-internal-zenai-apps-com-…”,
4 “time”: “2026-04-08T03:10:52Z”,
5 “message”: “forcing http/1.1 due to web socket support”
6 }

  1. Client-Side Failure (curl):
    When attempting the handshake via curl:

1 * using HTTP/1.x
2 > GET /ws HTTP/1.1
3 > Host: web-socket-other.internal.zenai-apps.com
4 > Upgrade: websocket
5 > Connection: Upgrade
6 …
7 * Empty reply from server
8 * shutting down connection #0
9 curl: (52) Empty reply from server

  1. Client-Side Failure (wscat):
    1 error: socket hang up

What Works (Infrastructure Validation)
We have conclusively proven that the underlying Google infrastructure and the application backend support WebSockets.

When running wscat from a debug pod within the GKE cluster and targeting the Internal LB IP directly (bypassing Pomerium), the WebSocket handshake
succeeds and data flows perfectly:

1 # Debug Pod → Internal LB (10.0.1.100)
2 Connected (press CTRL+C to quit)
3 > hello
4 < Echo: hello

Configurations Attempted
We have attempted to resolve the “Empty reply” through the following changes, none of which changed the result:

  1. Ingress Annotations:
  2. Policy Object: Included allow_websockets: true directly inside the ingress.pomerium.io/policy JSON object.
  3. Global Settings (Pomerium CRD):
    • Toggled codecType between http2, auto, and http1.
    • Added runtimeFlags: envoy_websocket_allow_upgrade: true.
    • Set all timeouts (idle, read, write) to 0s (infinite).
  4. Isolation:
    • Enabled allow_public_unauthenticated_access: “true” to rule out the auth/cookie layer.
    • Verified the failure persists even when bypassing the External LB via kubectl port-forward directly to the Pomerium pod.

Question
Is there a known regression in v0.32.5 regarding WebSocket upgrades, or is there a specific Envoy configuration required when the upstream is another L7
Load Balancer? The “Empty reply” suggests Envoy is rejecting the upgrade internally before the request is even logged.

2

We will try to reproduce your issue in our GKE environment. While we’re doing that, few things to clarify:

What type of Kubernetes Service points to the Internal LB?

Is it an `ExternalName` service, or a headless service with manually managed Endpoints? Does it resolve to a hostname or an IP.

Does the Internal ALB frontend expect HTTP or HTTPS from clients in the VPC?

If it expects plain HTTP, `secure_upstream` should be `false`. If HTTPS, what certificate does it present, and is it from a public or private CA?

How exactly did the debug pod test work?

Was it `ws://10.0.1.100/…` or `wss://10.0.1.100/…`? The exact command would tell us whether TLS is the differentiating factor.

Can you try with secure_upstream: "false"?

If the Internal ALB accepts plain HTTP, this bypasses all TLS/ALPN issues and would immediately confirm whether TLS is the problem.

Can you try with tls_skip_verify: "true" (keeping secure_upstream)?

This would isolate certificate validation failures from ALPN issues.

To get proxy-level debug logs:

Enable Envoy debug logging by adding the --debug-envoy flag to the ingress controller container args in the Deployment:

containers:
- name: pomerium
args:
- all-in-one
- --debug-envoy
# ... other args

This will show TLS handshake details, upstream connection errors, and ALPN negotiation results that don’t appear in access logs (note its extremely verbose so if your cluster is carrying normal traffic already that would quickly overwhelm kube logging system).

3

What type of Kubernetes Service points to the Internal LB?
I’ve tried both ExternalName and internal load balancer. Right now I have
EXT. L4 LOAD BAL → POMERIUM PROXY (POD) → INT. APPLICATION LB → GOOGLE CLOUD RUN.

I have confirmed that websockets work directly against the cloud run instance and against the internal app LB, but fail when I get to the pomerium pod directly. We use a combination of cluster.local or a FQDN with valid certs depending on where we are in the above.

How exactly did the debug pod test work?

1. Direct Cloud Run Validation

Goal: Confirm the application and Cloud Run support WebSockets.

kubectl run -it --rm debug --image=node:20-slim --restart=Never -- bash -c " \
  npm install -g wscat && \
  wscat -c https://web-socket-other-hash-uc.a.run.app/ws"
  • Result: Connected. Successfully sent and received echo messages.

2. PSC NEG / Internal LB Validation

Goal: Confirm the Google Internal LB and Private Service Connect NEG support WebSockets.

kubectl run -it --rm debug --image=node:20-slim --restart=Never -- bash -c " \
  npm install -g wscat && \
  wscat -n -H 'Host: web-socket-other.internal.domain.com' \
  -c https://web-socket-other-serverless-backend.vibe-app-web-socket-other.svc.cluster.local/ws"
  • Result: Connected. Successfully sent and received echo messages.
  • Significance: This proves ExternalName is not required to bypass the LB for WebSockets.

When I change to secure_upstream: "false" the entire flow breaks, so I’m not sure it supports http, or maybe I did it wrong. I may try again.

I’ll work on the other questions and ideas.

4

Select logs

{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:43Z","message":"[Tags: \"ConnectionId\":\"179\",\"StreamId\":\"11595815653348128517\"] encoding headers via codec (end_stream=false):\n':status', '200'\n'content-security-policy', 'default-src 'self';base-uri 'self';font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self' 'unsafe-eval';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests;connect-src 'self' http: https: ws: wss:'\n'referrer-policy', 'no-referrer'\n'x-content-type-options', 'nosniff'\n'x-dns-prefetch-control', 'off'\n'x-download-options', 'noopen'\n'x-permitted-cross-domain-policies', 'none'\n'x-powered-by', 'Express'\n'content-type', 'text/event-stream'\n'vary', 'Accept-Encoding'\n'content-encoding', 'br'\n'date', 'Thu, 09 Apr 2026 00:17:43 GMT'\n'x-envoy-upstream-service-time', '28'\n'cache-control', 'no-cache'\n'strict-transport-security', 'max-age=31536000; includeSubDomains; preload'\n'x-accel-buffering', 'no'\n'x-frame-options', 'SAMEORIGIN'\n'x-xss-protection', '1; mode=block'\n'server', 'envoy'\n'x-request-id', '3fd48ca3-a57e-4ba7-9d59-134787e6ccbe'\n"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"354\",\"StreamId\":\"5275074981135447655\"] request headers complete (end_stream=true):\n':method', 'GET'\n':authority', 'web-socket-other-2.internal.domain.com'\n':scheme', 'https'\n':path', '/'\n'cache-control', 'max-age=0'\n'sec-ch-ua', '\"Chromium\";v=\"146\", \"Not-A.Brand\";v=\"24\", \"Google Chrome\";v=\"146\"'\n'sec-ch-ua-mobile', '?0'\n'sec-ch-ua-platform', '\"macOS\"'\n'upgrade-insecure-requests', '1'\n'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'\n'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'\n'sec-fetch-site', 'same-site'\n'sec-fetch-mode', 'navigate'\n'sec-fetch-user', '?1'\n'sec-fetch-dest', 'document'\n'accept-encoding', 'gzip, deflate, br'\n'accept-language', 'en-US,en;q=0.9'\n'priority', 'u=0, i'\n'x-forwarded-for', '170.203.112.130'\n'cookie', '_pomerium=COOKIE'\n"}
{"level":"info","server-name":"all","service":"authorize","request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","check-request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","method":"GET","path":"/","host":"web-socket-other-2.internal.domain.com","ip":"10.0.0.18","session-id":"96873f8d-3a41-4786-a71f-c27d500a178a","user":"108870561382292340048","email":"daniel@domain.com","envoy-route-checksum":1779439935929183085,"envoy-route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","route-checksum":1779439935929183085,"route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":[],"time":"2026-04-09T00:17:47Z","message":"authorize check"}
{"level":"info","server-name":"all","service":"envoy","upstream-cluster":"vibe-app-web-socket-other-2-web-socket-other-2-web-socket-other-2-internal-domain-com-{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","method":"GET","authority":"web-socket-other-2.internal.domain.com","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36","referer":"","forwarded-for":"170.203.112.130,10.0.0.18","request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","duration":40.319228,"size":1879,"response-code":200,"response-code-details":"via_upstream","time":"2026-04-09T00:17:47Z","message":"http-request"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] new connection from 10.1.1.1:1984"}
{"level":"debug","service":"envoy","name":"http2","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] updating connection-level initial window size to 25165824"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] dispatch error: Received bad client magic byte string"}

I tried changing one thing, but get mostly the same

{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:28:59Z","message":"[Tags: \"ConnectionId\":\"462\",\"StreamId\":\"18396777476771781287\"] request headers complete (end_stream=true):\n':method', 'GET'\n':authority', 'web-socket-other-2.internal.domain.com'\n':scheme', 'https'\n':path', '/'\n'cache-control', 'max-age=0'\n'sec-ch-ua', '\"Chromium\";v=\"146\", \"Not-A.Brand\";v=\"24\", \"Google Chrome\";v=\"146\"'\n'sec-ch-ua-mobile', '?0'\n'sec-ch-ua-platform', '\"macOS\"'\n'upgrade-insecure-requests', '1'\n'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'\n'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'\n'sec-fetch-site', 'same-site'\n'sec-fetch-mode', 'navigate'\n'sec-fetch-user', '?1'\n'sec-fetch-dest', 'document'\n'accept-encoding', 'gzip, deflate, br'\n'accept-language', 'en-US,en;q=0.9'\n'priority', 'u=0, i'\n'x-forwarded-for', '170.203.112.130'\n'cookie', '_pomerium=COOKIE'\n"}
{"level":"info","server-name":"all","service":"authorize","request-id":"c32b2090-41d8-4da9-b0ca-d6aaf92d4fba","check-request-id":"c32b2090-41d8-4da9-b0ca-d6aaf92d4fba","method":"GET","path":"/","host":"web-socket-other-2.internal.domain.com","ip":"10.1.1.1","session-id":"96873f8d-3a41-4786-a71f-c27d500a178a","user":"108870561382292340048","email":"daniel@domain.com","envoy-route-checksum":1779439935929183085,"envoy-route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","route-checksum":1779439935929183085,"route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":[],"time":"2026-04-09T00:28:59Z","message":"authorize check"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] new connection from 10.1.1.1:59784"}
{"level":"debug","service":"envoy","name":"http2","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] updating connection-level initial window size to 25165824"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] dispatch error: Received bad client magic byte string"}

If you can tell me more specifically what to look for in the logs, I can try again.