Websocket connections forbidden 403

Support
1

I have a website using WebSockets running in Kubernetes.
Using the website without Pomerium works just fine. As soon as I put Pomerium in the mix, I get a 403 as shown on the image below.

image
image566×692 65.4 KB

I’m using Pomerium v0.19.1.
Below is my configuration.

insecure_server: true
address: :80
cookie_secret_file: /secrets/cookie-secret
idp_provider: azure
idp_provider_url: https://login.microsoftonline.com/<tenant-id>/v2.0
idp_client_id: <client-id>
idp_client_secret_file: /secrets/client-secret
authenticate_service_url: https://auth.<url>

routes:
  - from: https://<url>
    to: http://localhost:8080
    allow_websockets: true
    policy:
      - allow:
          or:
            - domain:
                is: <some-domain>

Here’s a related log line.

http-request authority=<url> duration=25.4898 forwarded-for=<public-ip>,10.80.2.37 method=POST path=/cmd/225/a7s1ffwl/xhr referer=https://<url>/ request-id=<uuid> response-code=403 response-code-details=via_upstream service=envoy size=20 upstream-cluster=route-bf846d07f01e3ded
2

The via_upstream means that it was an application behind Pomerium that responded with 403.

There must be an additional log line from authorize with the same request-id - likely the request was authorized by Pomerium, but the target application rejected it. Most common suspect is Host header - please see Headers Settings | Pomerium

3

Side note is that we generally recommend deploying Pomerium directly and let it act as an ingress controller. See Kubernetes Quickstart | Pomerium

4

I tried adding preserve_host_header: true on the route with no luck.
Here’s the logs with related request ID’s

{
  "level": "info",
  "service": "authorize",
  "request-id": "f2c575e7-4c52-48c2-9934-08f8a27f44fe",
  "check-request-id": "f2c575e7-4c52-48c2-9934-08f8a27f44fe",
  "method": "GET",
  "path": "/cmd/097/n8j825r7/websocket",
  "host": "<url>",
  "query": "",
  "ip": "10.80.0.38",
  "session-id": "c71639f3-952f-4702-b09b-c643275b82b7",
  "allow": true,
  "allow-why-true": [
    "domain-ok",
    "groups-ok"
  ],
  "deny": false,
  "deny-why-false": [
    "valid-client-certificate-or-none-required"
  ],
  "user": "<uuid>",
  "email": "<email>",
  "time": "2022-11-12T13:41:12Z",
  "message": "authorize check"
}

{
  "level": "info",
  "service": "envoy",
  "upstream-cluster": "route-bf846d07f01e3ded",
  "method": "GET",
  "authority": "<url>",
  "path": "/cmd/097/n8j825r7/websocket",
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0",
  "referer": "",
  "forwarded-for": "<public-ip>,10.80.0.38",
  "request-id": "f2c575e7-4c52-48c2-9934-08f8a27f44fe",
  "duration": 27.367261,
  "size": 20,
  "response-code": 403,
  "response-code-details": "via_upstream",
  "time": "2022-11-12T13:41:12Z",
  "message": "http-request"
}
5

But you’re right… I enabled debug logs on the app behind Pomerium and it looks like there’s some CORS issues…

Reject: '<url>' origin is not allowed
Completed 403 FORBIDDEN, headers={masked}

I dont get why Pomerium changes that.

6

I added HTTPS between the NGINX ingress and pomerium and that seems to have fixed the issue. I have no idea why.

Below flow had CORS issues.

Browser --HTTPS-> NGINX --HTTP-> Pomerium --HTTP-> Backend

This does not and works.

Browser --HTTPS-> NGINX --HTTPS-> Pomerium --HTTP-> Backend