# WebSocket "Socket Hang Up" / Empty Reply on Ingress Controller v0.32.5 with Internal LB Upstream

**URL:** <https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508>\
**Category:** Support\
**Tags:** idp, k8s\
**Created:** [April 8, 2026, 3:46am UTC](https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508 "2026-04-08T03:46:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dwatrous](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/dwatrous/32/189_2.png) [@dwatrous](https://discuss.pomerium.com/u/dwatrous)\
**Post date:** [April 8, 2026, 3:46am UTC](https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508/1 "2026-04-08T03:46:08Z")

</div>

Environment

- Pomerium Ingress Controller: v0.32.5
- Infrastructure: GKE Private Cluster
- Upstream: Google Internal Application Load Balancer (L7) via Serverless Network Endpoint Group (PSC) to Cloud Run.
- Traffic Flow: Browser → Regional NLB (L4) → Pomerium Pod → VPC → Google Internal LB → PSC → Cloud Run.

The Issue  
WebSocket upgrade requests (Connection: Upgrade, Upgrade: websocket) to any application behind Pomerium fail immediately. The connection is dropped  
before any data is exchanged and without appearing in the Pomerium access logs.

Logs & Error Output

1. Pomerium Ingress Controller Logs:  
The controller successfully recognizes the Ingress annotation and attempts to configure the route for WebSockets:

1 {  
2 “level”: “info”,  
3 “cluster-id”: “vibe-app-web-socket-other-web-socket-other-web-socket-other-internal-zenai-apps-com-…”,  
4 “time”: “2026-04-08T03:10:52Z”,  
5 “message”: “forcing http/1.1 due to web socket support”  
6 }

1. Client-Side Failure (curl):  
When attempting the handshake via curl:

1 \* using HTTP/1.x  
2 \> GET /ws HTTP/1.1  
3 \> Host: [web-socket-other.internal.zenai-apps.com](http://web-socket-other.internal.zenai-apps.com)  
4 \> Upgrade: websocket  
5 \> Connection: Upgrade  
6 …  
7 \* Empty reply from server  
8 \* shutting down connection #0  
9 curl: (52) Empty reply from server

1. Client-Side Failure (wscat):  
1 error: socket hang up

* * *

What Works (Infrastructure Validation)  
We have conclusively proven that the underlying Google infrastructure and the application backend support WebSockets.

When running wscat from a debug pod within the GKE cluster and targeting the Internal LB IP directly (bypassing Pomerium), the WebSocket handshake  
succeeds and data flows perfectly:

1 # Debug Pod → Internal LB (10.0.1.100)  
2 Connected (press CTRL+C to quit)  
3 \> hello  
4 \< Echo: hello

* * *

Configurations Attempted  
We have attempted to resolve the “Empty reply” through the following changes, none of which changed the result:

1. Ingress Annotations:
  - [ingress.pomerium.io/allow\_websockets:](http://ingress.pomerium.io/allow_websockets:) “true”
  - [ingress.pomerium.io/secure\_upstream:](http://ingress.pomerium.io/secure_upstream:) “true”
  - [ingress.pomerium.io/tls\_server\_name:](http://ingress.pomerium.io/tls_server_name:) 
  - [ingress.pomerium.io/preserve\_host\_header:](http://ingress.pomerium.io/preserve_host_header:) “true”

2. Policy Object: Included allow\_websockets: true directly inside the [ingress.pomerium.io/policy](http://ingress.pomerium.io/policy) JSON object.
3. Global Settings (Pomerium CRD):
  - Toggled codecType between http2, auto, and http1.
  - Added runtimeFlags: envoy\_websocket\_allow\_upgrade: true.
  - Set all timeouts (idle, read, write) to 0s (infinite).

4. Isolation:
  - Enabled allow\_public\_unauthenticated\_access: “true” to rule out the auth/cookie layer.
  - Verified the failure persists even when bypassing the External LB via kubectl port-forward directly to the Pomerium pod.

Question  
Is there a known regression in v0.32.5 regarding WebSocket upgrades, or is there a specific Envoy configuration required when the upstream is another L7  
Load Balancer? The “Empty reply” suggests Envoy is rejecting the upgrade internally before the request is even logged.

---

<div class="post-metadata">

**Author:** ![denis](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@denis](https://discuss.pomerium.com/u/denis)\
**Post date:** [April 8, 2026, 9:56pm UTC](https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508/2 "2026-04-08T21:56:34Z")

</div>

We will try to reproduce your issue in our GKE environment. While we’re doing that, few things to clarify:

**What type of Kubernetes Service points to the Internal LB?**

Is it an `ExternalName` service, or a headless service with manually managed Endpoints? Does it resolve to a hostname or an IP.

**Does the Internal ALB frontend expect HTTP or HTTPS from clients in the VPC?**

If it expects plain HTTP, `secure_upstream` should be `false`. If HTTPS, what certificate does it present, and is it from a public or private CA?

**How exactly did the debug pod test work?**

Was it `ws://10.0.1.100/…` or `wss://10.0.1.100/…`? The exact command would tell us whether TLS is the differentiating factor.

**Can you try with `secure_upstream: "false"`?**

* * *

If the Internal ALB accepts plain HTTP, this bypasses all TLS/ALPN issues and would immediately confirm whether TLS is the problem.

**Can you try with `tls_skip_verify: "true"` (keeping secure\_upstream)?**

This would isolate certificate validation failures from ALPN issues.

**To get proxy-level debug logs:**

Enable Envoy debug logging by adding the `--debug-envoy` flag to the ingress controller container args in the Deployment:

`containers:`  
` - name: pomerium`  
` args:`  
` - all-in-one`  
` - --debug-envoy`  
` # ... other args`

This will show TLS handshake details, upstream connection errors, and ALPN negotiation results that don’t appear in access logs (note its extremely verbose so if your cluster is carrying normal traffic already that would quickly overwhelm kube logging system).

---

<div class="post-metadata">

**Author:** ![dwatrous](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/dwatrous/32/189_2.png) [@dwatrous](https://discuss.pomerium.com/u/dwatrous)\
**Post date:** [April 8, 2026, 11:51pm UTC](https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508/3 "2026-04-08T23:51:07Z")

</div>

**What type of Kubernetes Service points to the Internal LB?**  
I’ve tried both ExternalName and internal load balancer. Right now I have  
EXT. L4 LOAD BAL → POMERIUM PROXY (POD) → INT. APPLICATION LB → GOOGLE CLOUD RUN.

I have confirmed that websockets work directly against the cloud run instance and against the internal app LB, but fail when I get to the pomerium pod directly. We use a combination of cluster.local or a FQDN with valid certs depending on where we are in the above.

**How exactly did the debug pod test work?**

### 1. Direct Cloud Run Validation

**Goal:** Confirm the application and Cloud Run support WebSockets.

```bash
kubectl run -it --rm debug --image=node:20-slim --restart=Never -- bash -c " \
  npm install -g wscat && \
  wscat -c https://web-socket-other-hash-uc.a.run.app/ws"

```

- **Result:** `Connected`. Successfully sent and received echo messages.

### 2. PSC NEG / Internal LB Validation

**Goal:** Confirm the Google Internal LB and Private Service Connect NEG support WebSockets.

```bash
kubectl run -it --rm debug --image=node:20-slim --restart=Never -- bash -c " \
  npm install -g wscat && \
  wscat -n -H 'Host: web-socket-other.internal.domain.com' \
  -c https://web-socket-other-serverless-backend.vibe-app-web-socket-other.svc.cluster.local/ws"

```

- **Result:** `Connected`. Successfully sent and received echo messages.
- **Significance:** This proves `ExternalName` is **not required** to bypass the LB for WebSockets.

When I change to **`secure_upstream: "false"`** the entire flow breaks, so I’m not sure it supports http, or maybe I did it wrong. I may try again.

I’ll work on the other questions and ideas.

---

<div class="post-metadata">

**Author:** ![dwatrous](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/dwatrous/32/189_2.png) [@dwatrous](https://discuss.pomerium.com/u/dwatrous)\
**Post date:** [April 9, 2026, 12:47am UTC](https://discuss.pomerium.com/t/websocket-socket-hang-up-empty-reply-on-ingress-controller-v0-32-5-with-internal-lb-upstream/508/4 "2026-04-09T00:47:12Z")

</div>

Select logs

```auto
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:43Z","message":"[Tags: \"ConnectionId\":\"179\",\"StreamId\":\"11595815653348128517\"] encoding headers via codec (end_stream=false):\n':status', '200'\n'content-security-policy', 'default-src 'self';base-uri 'self';font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self' 'unsafe-eval';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests;connect-src 'self' http: https: ws: wss:'\n'referrer-policy', 'no-referrer'\n'x-content-type-options', 'nosniff'\n'x-dns-prefetch-control', 'off'\n'x-download-options', 'noopen'\n'x-permitted-cross-domain-policies', 'none'\n'x-powered-by', 'Express'\n'content-type', 'text/event-stream'\n'vary', 'Accept-Encoding'\n'content-encoding', 'br'\n'date', 'Thu, 09 Apr 2026 00:17:43 GMT'\n'x-envoy-upstream-service-time', '28'\n'cache-control', 'no-cache'\n'strict-transport-security', 'max-age=31536000; includeSubDomains; preload'\n'x-accel-buffering', 'no'\n'x-frame-options', 'SAMEORIGIN'\n'x-xss-protection', '1; mode=block'\n'server', 'envoy'\n'x-request-id', '3fd48ca3-a57e-4ba7-9d59-134787e6ccbe'\n"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"354\",\"StreamId\":\"5275074981135447655\"] request headers complete (end_stream=true):\n':method', 'GET'\n':authority', 'web-socket-other-2.internal.domain.com'\n':scheme', 'https'\n':path', '/'\n'cache-control', 'max-age=0'\n'sec-ch-ua', '\"Chromium\";v=\"146\", \"Not-A.Brand\";v=\"24\", \"Google Chrome\";v=\"146\"'\n'sec-ch-ua-mobile', '?0'\n'sec-ch-ua-platform', '\"macOS\"'\n'upgrade-insecure-requests', '1'\n'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'\n'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'\n'sec-fetch-site', 'same-site'\n'sec-fetch-mode', 'navigate'\n'sec-fetch-user', '?1'\n'sec-fetch-dest', 'document'\n'accept-encoding', 'gzip, deflate, br'\n'accept-language', 'en-US,en;q=0.9'\n'priority', 'u=0, i'\n'x-forwarded-for', '170.203.112.130'\n'cookie', '_pomerium=COOKIE'\n"}
{"level":"info","server-name":"all","service":"authorize","request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","check-request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","method":"GET","path":"/","host":"web-socket-other-2.internal.domain.com","ip":"10.0.0.18","session-id":"96873f8d-3a41-4786-a71f-c27d500a178a","user":"108870561382292340048","email":"daniel@domain.com","envoy-route-checksum":1779439935929183085,"envoy-route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","route-checksum":1779439935929183085,"route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":[],"time":"2026-04-09T00:17:47Z","message":"authorize check"}
{"level":"info","server-name":"all","service":"envoy","upstream-cluster":"vibe-app-web-socket-other-2-web-socket-other-2-web-socket-other-2-internal-domain-com-{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","method":"GET","authority":"web-socket-other-2.internal.domain.com","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36","referer":"","forwarded-for":"170.203.112.130,10.0.0.18","request-id":"1185d2eb-faad-4276-9e84-9972429bcfc1","duration":40.319228,"size":1879,"response-code":200,"response-code-details":"via_upstream","time":"2026-04-09T00:17:47Z","message":"http-request"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] new connection from 10.1.1.1:1984"}
{"level":"debug","service":"envoy","name":"http2","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] updating connection-level initial window size to 25165824"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:17:47Z","message":"[Tags: \"ConnectionId\":\"356\"] dispatch error: Received bad client magic byte string"}

```

I tried changing one thing, but get mostly the same

```auto
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:28:59Z","message":"[Tags: \"ConnectionId\":\"462\",\"StreamId\":\"18396777476771781287\"] request headers complete (end_stream=true):\n':method', 'GET'\n':authority', 'web-socket-other-2.internal.domain.com'\n':scheme', 'https'\n':path', '/'\n'cache-control', 'max-age=0'\n'sec-ch-ua', '\"Chromium\";v=\"146\", \"Not-A.Brand\";v=\"24\", \"Google Chrome\";v=\"146\"'\n'sec-ch-ua-mobile', '?0'\n'sec-ch-ua-platform', '\"macOS\"'\n'upgrade-insecure-requests', '1'\n'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'\n'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'\n'sec-fetch-site', 'same-site'\n'sec-fetch-mode', 'navigate'\n'sec-fetch-user', '?1'\n'sec-fetch-dest', 'document'\n'accept-encoding', 'gzip, deflate, br'\n'accept-language', 'en-US,en;q=0.9'\n'priority', 'u=0, i'\n'x-forwarded-for', '170.203.112.130'\n'cookie', '_pomerium=COOKIE'\n"}
{"level":"info","server-name":"all","service":"authorize","request-id":"c32b2090-41d8-4da9-b0ca-d6aaf92d4fba","check-request-id":"c32b2090-41d8-4da9-b0ca-d6aaf92d4fba","method":"GET","path":"/","host":"web-socket-other-2.internal.domain.com","ip":"10.1.1.1","session-id":"96873f8d-3a41-4786-a71f-c27d500a178a","user":"108870561382292340048","email":"daniel@domain.com","envoy-route-checksum":1779439935929183085,"envoy-route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","route-checksum":1779439935929183085,"route-id":"{\"n\":\"web-socket-other-2\",\"ns\":\"vibe-app-web-socket-other-2\",\"h\":\"web-socket-other-2.internal.domain.com\",\"p\":\"/\"}","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":[],"time":"2026-04-09T00:28:59Z","message":"authorize check"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] new connection from 10.1.1.1:59784"}
{"level":"debug","service":"envoy","name":"http2","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] updating connection-level initial window size to 25165824"}
{"level":"debug","service":"envoy","name":"http","time":"2026-04-09T00:29:03Z","message":"[Tags: \"ConnectionId\":\"467\"] dispatch error: Received bad client magic byte string"}

```

If you can tell me more specifically what to look for in the logs, I can try again.