OIDC behind pomerium route

Support
,
1

I’m trying to setup a Pomerium instance with an instance of Keycloak behind an unauthenticated route but having some issues (maybe circular dependencies).

Setup

I have pomerium setup as the reverse proxy to *.example.com. Internally, I have a keycloak instance serving traffic on, lets stay 192.168.1.10:8080.

What happened?

While trying to reach another route (verify.example.com). I get upstream request timeout when redirected to https://auth.example.com/.pomerium/sign_in?pomerium_hpke_query=…

What did you expect to happen?

I expect to be redirected to login.example.com to login and authenticate.

What’s your environment like?

  • Pomerium version (retrieve with pomerium --version):
pomerium: 0.22.2-1685134689+6efd1d6b
envoy: 1.25.5+b1095c058415dfb2261e695a0f144311a7dc346b6eb47ecbb0a01b7de2c7299f
  • Server Operating System/Architecture/Cloud: Ubuntu 22.04.2, docker 24.0.4

What’s your config.yaml?

pomerium_debug: true
authenticate_internal_service_url: https://authenticate.localhost.pomerium.io
authenticate_service_url: https://auth.example.com

idp_client_id: [id]
idp_client_secret: [secret]
idp_provider_url: http://login.example.com/realms/pomerium
idp_provider: oidc
autocert: false
certificate_file: fullchain.pem
certificate_key_file: privkey.pem
signing_key_file: signingkey.pem

routes:
  - from: https://verify.example.com
    to: http://verify:8000
    policy:
      - allow:
          or:
            - email:
                is: user@example.com
  - from: https://login.example.com
    to: http://192.168.1.10:8080
    allow_public_unauthenticated_access: true

What did you see in the logs?

pomerium-pomerium-1  | 6:02AM INF authorize check allow=false allow-why-false=["user-unauthenticated"] check-request-id=cf30d80a-ed64-492c-8a66-4de4ecda9293 deny=false deny-why-false=["valid-client-certificate-or-none-required"] email= host=verify.example.com ip=192.168.1.246 method=GET path=/ query= request-id=cf30d80a-ed64-492c-8a66-4de4ecda9293 service=authorize user=
pomerium-pomerium-1  | 6:02AM INF authorize check allow=true allow-why-true=["pomerium-route"] check-request-id=93dfbebf-9ac0-428d-9ca7-093959f22d83 deny=false deny-why-false=[] email= host=auth.example.com ip=192.168.1.246 method=GET path=/.pomerium/sign_in query= request-id=93dfbebf-9ac0-428d-9ca7-093959f22d83 service=authorize user=
pomerium-pomerium-1  | 6:02AM INF authenticate: session load error error="Bad Request: internal/sessions: session is not found" idp_id=4crv8w9HLD4pLey3eKyWqsEgDRnTFeWKYcLGi5km8z9Y ip=127.0.0.1 request-id=93dfbebf-9ac0-428d-9ca7-093959f22d83 user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
pomerium-pomerium-1  | 6:02AM INF http-request authority=authenticate.localhost.pomerium.io duration=9.087308 forwarded-for=172.18.0.2 method=GET path=/.well-known/pomerium/hpke-public-key referer= request-id=d830be35-3993-4414-985f-080b1dca180a response-code=304 response-code-details=via_upstream service=envoy size=0 upstream-cluster=pomerium-control-plane-http user-agent=Go-http-client/2.0
pomerium-pomerium-1  | 6:02AM INF http-request authority=verify.example.com duration=37.221001 forwarded-for=192.168.1.246 method=GET path=/ referer= request-id=cf30d80a-ed64-492c-8a66-4de4ecda9293 response-code=302 response-code-details=ext_authz_denied service=envoy size=1422 upstream-cluster= user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"


pomerium-pomerium-1  | 6:03AM INF http-request authority=auth.example.com duration=15010.132152 forwarded-for=192.168.1.246 method=GET path=/.pomerium/sign_in referer= request-id=93dfbebf-9ac0-428d-9ca7-093959f22d83 response-code=504 response-code-details=response_timeout service=envoy size=24 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"


pomerium-pomerium-1  | 6:05AM ERR httputil: error error="failed to get sign in url: identity/oidc: could not connect to oidc: Get \"https://login.example.com\": dial tcp 24.5.221.125:443: connect: connection timed out" ip=127.0.0.1 request-id=93dfbebf-9ac0-428d-9ca7-093959f22d83 status=500 status-text="Internal Server Error" user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

Ive tried playing around with the authenticate_internal_service_url and authenticate_service_url, but was unable to get anything to get past this issue.

2
  1. are you able to reach https://login.example.com/realms/pomerium/.well-known/openid-configuration from i.e. browser or curl?
  2. are you able to reach it from the Pomerium server host (or container) ?
  3. Your idp_provider_url in the config is set to http, not https - please correct it. Pomerium only serves on 443/https
3

Hey Denis,

Thanks for the reply. To answer your questions:

  1. Yes, I’m able to assess https://login.example.com/realms/pomerium/.well-known/openid-configuration from the public Internet through Pomerium
  2. Yes (assuming you mean just being able to access routes like the above from the public)
  3. You are correct, good catch, I was testing between using the route address vs using the internal 192.168.1.10 address and forgot to switch it back. Let me change it back tonight and just verify. Will update if there is a different error, but I’m am fairly certain I had it as https at one point.
4

No this is not what I meant.

I meant going to the server machine that runs pomerium, or, if it’s running inside a container, shell into the container, and do wget https://login.example.com/realms/pomerium/.well-known/openid-configuration there and see if you can connect from there.

Because the error we observe is that for some reason Pomerium cannot connect to that URL, it receives server timeout.

5

Thanks Denis, that turned out to be the issue. My firewall had a policy to block all access from the host which included loopback policies from the host. I had to create an allow policy to allow loopback from itself to itself which finally allowed me to access the login page!

Thanks for the pointers Denis!