ERR_TOO_MANY_REDIRECTS after authenticating (Traefik + Pomerium + Keycloak)

Support
1

What happened?

After filling in the authentication information on the Keycloak login page, I end up on a blank page with the error ERR_TOO_MANY_REDIRECTS.

What did you expect to happen?

I hope to be redirected to the home page of the protected service.

How’d it happen?

  1. From the web browser (Google Chrome), I fill the URL of the protected service.
  2. I am redirected to the Keycloak login page
  3. I enter the login credentials of an account present in the Keycloak database and I validate.
  4. I end up on a blank page with the ERR_TOO_MANY_REDIRECTS error.

What’s your environment like?

  • Pomerium version : 0.17.3 (container: pomerium/pomerium:v0.17.3 from DockerHub)
  • Server Operating System/Architecture/Cloud: Ubuntu Server 20.04.4 LTS (x86-64)
  • Docker Swarm Cluster (3 managers + 3 workers)
  • Traefik Proxy deployed as container in the Swarm Cluster in Host Mode on the 3 managers.
  • Keycloak 18.0.0 deployed as a container in the Swarm Cluster.
  • In front of the Docker Swarm Cluster, as a front loadbalancer, a VM with Traefik Proxy binary (sticky cookie in place).

What’s your config.yaml?

pomerium_debug: true
# log_level: debug
# log_level: "info"

address: :8443
services: "all"

shared_secret: xxxxx

authenticate_service_url: https://sso.subdomain.domain.tld

forward_auth_url: https://forwardauth.subdomain.domain.tld

cookie_name: _pomerium_sso
cookie_secret: xxxxx
cookie_domain: domain.tld
cookie_secure: true
cookie_http_only: true
cookie_expire: 2m

idp_provider: oidc
idp_provider_url: https://auth.subdomain.domain.tld/realms/mycustomrealm
idp_client_id: xxxxx
idp_client_secret: xxxxx
idp_scopes: openid,profile,email

jwt_claims_headers: email,groups,user,given_name

routes:
  - from: https://tv.subdomain.domain.tld
    to: http://verify:8000
    pass_identity_headers: true
    signout_redirect_url: https://auth.subdomain.domain.tld/realms/mycustomrealm/protocol/openid-connect/logout
    policy: 
      - allow:
          or:
            - domain:
                is: emaildomain.tld
            - domain:
                is: domain.tld

What did you see in the logs?

pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"info","envoy_version":"1.20.2+694e5593dec2d70728dd5064a03c6e4db01bdf801f90d60deaaaf69e356b01d4","version":"0.17.3-1651784655+f4bad550","time":"2022-05-12T12:55:23Z","message":"cmd/pomerium"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"warn","config_file":"/pomerium/config.yaml","keys":["routes[0].signout_redirect_url"],"time":"2022-05-12T12:55:23Z","message":"config contained unknown keys that were ignored"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"warn","time":"2022-05-12T12:55:23Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"info","config_file_source":"/pomerium/config.yaml","service":"all","config":"local","checksum":"d611535d405267f4","time":"2022-05-12T12:55:23Z","message":"config: updated config"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"info","address":"127.0.0.1:45063","time":"2022-05-12T12:55:23Z","message":"dialing"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | {"level":"info","service":"all","config":"databroker","checksum":"d611535d405267f4","time":"2022-05-12T12:55:23Z","message":"config: updated config"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF config: starting databroker config source syncer databroker_urls=["http://127.0.0.1:5443"] outbound_port=45063
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initial sync syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR error during initial sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF metrics: http server disabled service=metrics_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=forwardauth.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initial sync syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR error during initial sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initial sync syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR error during initial sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=sso.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=tv.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=*
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initial sync syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR error during initial sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=forwardauth.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=sso.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=tv.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR cryptutil: no TLS certificate found for domain, using self-signed certificate domain=*
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF server started debug-port=43961 grpc-port=38373
 http-port=39727 metrics-port=43865 outbound-port=45063
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF extracting envoy binary path=/tmp/pomerium-embedded-files/envoy-2381462427/envoy
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initial sync syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR error during initial sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR sync error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:45063: connect: connection refused\"" syncer_id=databroker syncer_type=type.googleapis.com/pomerium.config.Config
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF envoy: starting envoy process
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF running envoy checksum=694e5593dec2d70728dd5064a03c6e4db01bdf801f90d60deaaaf69e356b01d4 path=/tmp/pomerium-embedded-files/envoy-2381462427/envoy
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF dialing address=127.0.0.1:45063
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF enabled authenticate service host=sso.subdomain.domain.tld
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF authorize: signing key Algorithm=ES256 KeyID=07ec746355d40f6636878e26e97d7f243e4cff7640349e2facf03ee076f38eb0 Public Key={"alg":"ES256","crv":"P-256","kid":"07ec746355d40f6636878e26e97d7f243e4cff7640349e2facf03ee076f38eb0","kty":"EC","use":"sig","x":"w3TH7lrZ_BUKSpwRFyVgHJF3hHnhAbs6rf9otVDGxzY","y":"gKaYoO21UR5CxlvSHNdxDBPvjgXDmZ5Kpj8KlbLDzPM"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF envoy: start monitoring subprocess pid=12
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF dialing address=127.0.0.1:45063
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF enabled authorize service
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF authorize: signing key Algorithm=ES256 KeyID=ce06f733f0ea7fdda4aeacf168ce9df737b2cd323cb65aae2e1915bbc99b07e2 Public Key={"alg":"ES256","crv":"P-256","kid":"ce06f733f0ea7fdda4aeacf168ce9df737b2cd323cb65aae2e1915bbc99b07e2","kty":"EC","use":"sig","x":"l2FwhtuCl_iM6oUq9ID-B2HmsoajrzpPhvFOJmWsI2U","y":"LBzhKVCLWqwLk-wvzDxTysT52gxeqCPCrk2f-69EpvA"}
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF initializing epoch 0 (base id=136393317, hot restart version=11.104) name=main service=envoy
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF statically linked extensions: name=main service=envoy
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF   envoy.upstreams: envoy.filters.connection_pools.tcp.generic name=main service=envoy

(...)

pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM WRN failed to refresh directory users and groups error="unknown directory provider oidc" service=identity_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF put id=identity_manager_last_user_group_refresh_errors type=type.googleapis.com/pomerium.events.LastError
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF authenticate: session load error error="Bad Request: internal/sessions: session is not found" X-Forwarded-For=["172.16.0.142, 172.16.99.41,10.10.10.14"] X-Forwarded-Host=["sso.subdomain.domain.tld"] X-Forwarded-Port=["443"] X-Forwarded-Proto=["https"] X-Forwarded-Server=["services-proxy"] X-Real-Ip=["172.16.0.142"] idp_id= ip=127.0.0.1 request-id=d957bdb0-8f47-46e6-a889-eebd8405378a user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=148.88525 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=d957bdb0-8f47-46e6-a889-eebd8405378a response-code=302 response-code-details=via_upstream service=envoy size=668 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=4.545742 forwarded-for="172.16.99.103, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=8c8c6359-8877-4dc0-a466-ed76bde18810 response-code=302 response-code-details=via_upstream service=envoy size=188 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF refreshing directory users service=identity_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM WRN failed to refresh directory users and groups error="unknown directory provider oidc" service=identity_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF put id=identity_manager_last_user_group_refresh_errors type=type.googleapis.com/pomerium.events.LastError
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=579c67b0-d9c2-4d36-9695-66ae0d776b58 type=type.googleapis.com/user.User
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF put id=579c67b0-d9c2-4d36-9695-66ae0d776b58 type=type.googleapis.com/user.User
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF put id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM ERR directory: failed to refresh user data error="rpc error: code = Unknown desc = unknown directory provider oidc" X-Forwarded-For=["172.16.0.142, 172.16.99.41,10.10.10.14"] X-Forwarded-Host=["sso.subdomain.domain.tld"] X-Forwarded-Port=["443"] X-Forwarded-Proto=["https"] X-Forwarded-Server=["services-proxy"] X-Real-Ip=["172.16.0.142"] ip=127.0.0.1 request-id=b25ecc93-defe-4594-818c-cd82b7ec0027 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=154.94321 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/oauth2/callback referer= request-id=b25ecc93-defe-4594-818c-cd82b7ec0027 response-code=302 response-code-details=via_upstream service=envoy size=188 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=9.108961 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=fe5a98e9-514e-4f98-a215-66627f789bd6 response-code=302 response-code-details=via_upstream service=envoy size=747 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=12.557939 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=a52726ff-445b-4a24-a04f-445f58edc90e response-code=302 response-code-details=via_upstream service=envoy size=757 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=13.664892 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=b92d3562-ef3e-4953-9ffb-e03786b4bad7 response-code=302 response-code-details=via_upstream service=envoy size=753 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=11.30218 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=ea03d71c-c4f0-4bd7-8309-faa904f47cd5 response-code=302 response-code-details=via_upstream service=envoy size=753 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=10.430557 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=c11d3fe1-077f-4581-a437-46101e9420c0 response-code=302 response-code-details=via_upstream service=envoy size=750 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=9.570874 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=57d16873-2f28-4640-82f4-b860902501c7 response-code=302 response-code-details=via_upstream service=envoy size=757 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=10.125169 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=38e14a74-f2ab-400c-9898-9622a801fde8 response-code=302 response-code-details=via_upstream service=envoy size=752 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=8.321065 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=6c1050fe-fe86-484f-9edf-36fe6f5e780b response-code=302 response-code-details=via_upstream service=envoy size=756 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=sso.subdomain.domain.tld duration=10.328122 forwarded-for="172.16.0.142, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/sign_in referer= request-id=1128ffe7-bf26-449d-8dcf-abcaca435198 response-code=302 response-code-details=via_upstream service=envoy size=754 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=2.194933 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=e2af1560-1b10-411e-bd1b-3a0be53829be response-code=302 response-code-details=via_upstream service=envoy size=192 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=2.593533 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=32d6f029-9974-4aab-95ac-d4f2a9970bcb response-code=302 response-code-details=via_upstream service=envoy size=192 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=2.725428 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=ae36c14c-5efd-42d2-b23e-776841ef74df response-code=302 response-code-details=via_upstream service=envoy size=192 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=1.718399 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=9be1a537-d020-431e-a42b-81baa9b452fd response-code=302 response-code-details=via_upstream service=envoy size=192 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=1.76962 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=86768200-872e-4958-95f5-f72c47a65a34 response-code=302 response-code-details=via_upstream service=envoy size=190 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=1.861841 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=0afb5e32-ac24-472b-a482-089ec3cbcd61 response-code=302 response-code-details=via_upstream service=envoy size=190 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=2.574546 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=ab3d4cf0-3c52-45d8-aa2f-6888c8c7e6e7 response-code=302 response-code-details=via_upstream service=envoy size=190 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=1.79449 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=0144211d-ba3d-4a2f-851e-3c9eea6ba4ed response-code=302 response-code-details=via_upstream service=envoy size=190 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF http-request authority=forwardauth.subdomain.domain.tld duration=1.828871 forwarded-for="172.16.99.101, 172.16.99.41,10.10.10.11" method=GET path=/ referer= request-id=f1e6d28c-e57d-49ec-92f8-db17e8b26b54 response-code=302 response-code-details=via_upstream service=envoy size=190 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF refreshing directory users service=identity_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM WRN failed to refresh directory users and groups error="unknown directory provider oidc" service=identity_manager
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF put id=identity_manager_last_user_group_refresh_errors type=type.googleapis.com/pomerium.events.LastError
pomerium_gateway.1.8wehwg5o1v5u@smgr1    | 12:55PM INF get id=96ad3b11-160b-45cf-b747-e2e055117848 type=type.googleapis.com/session.Session

Additional context

version: "3.8"

networks:
  swarm-services-net:
    external: true
  swarm-core-net:
    external: true

configs:
  pomerium_config:
    file: ./configs/pomerium-config.yaml

services:

  gateway:
    image: pomerium/pomerium:v0.17.3
    hostname : sso-gateway

    configs:
      - source: pomerium_config
        target: /pomerium/config.yaml
        mode: 0440

    networks:
      - swarm-core-net # To connect to Traefik Proxy service ("services-proxy")
      - swarm-services-net # To connect to Traefik Proxy service ("services-proxy")

    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
      update_config:
        parallelism: 1
        order: start-first
        failure_action: rollback
      placement:
        constraints:
          - node.role == manager

      labels:
        # Connect the "verify" (pomerium-gateway) service to the "swarm-core-net" network and make it reachable by traefik proxy
        - traefik.enable=true
        - traefik.docker.network=swarm-core-net

        ### POMERIUM GATEWAY
        ## Route and protect the trafic with letsencrypt SSL certificate and authentication
        - traefik.http.routers.to-sso-gateway.entrypoints=websecure
        - traefik.http.routers.to-sso-gateway.rule=Host(`sso.subdomain.domain.tld`)
        - traefik.http.routers.to-sso-gateway.tls=true
        - traefik.http.routers.to-sso-gateway.tls.certresolver=letsencrypt-dns-production
        - traefik.http.services.svc-sso-gateway.loadbalancer.server.port=8443
        - traefik.http.services.svc-sso-gateway.loadbalancer.server.scheme=https
        - traefik.http.services.svc-sso-gateway.loadbalancer.serverstransport=skip-ssl-verification@file

        ### POMERIUM FORWARD AUTH
        ## Route and protect the trafic with letsencrypt SSL certificate and authentication
        - traefik.http.routers.to-sso-fwauth.entrypoints=websecure
        - traefik.http.routers.to-sso-fwauth.rule=Host(`forwardauth.subdomain.domain.tld`)
        - traefik.http.routers.to-sso-fwauth.tls=true
        - traefik.http.routers.to-sso-fwauth.tls.certresolver=letsencrypt-dns-production

  verify:
    image: pomerium/verify:latest
    hostname: sso-pom-verify

    networks:
      - swarm-services-net # To connect to Traefik Proxy service ("services-proxy")

    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
      update_config:
        parallelism: 1
        order: start-first
        failure_action: rollback
      placement:
        constraints:
          - node.role == worker

      labels:
        # Connect the "verify" (pomerium-verify) service to the "swarm-services-net" network and make it reachable by traefik proxy
        - traefik.enable=true
        - traefik.docker.network=swarm-services-net

        ## MIDDLEWARE : FORWARDAUTH : mdlw-pomerium-verify
        - traefik.http.middlewares.mdlw-pomerium-verify.forwardauth.address=https://forwardauth.subdomain.domain.tld/?uri=https://tv.subdomain.domain.tld
        - traefik.http.middlewares.mdlw-pomerium-verify.forwardauth.tls.insecureSkipVerify=true
        - traefik.http.middlewares.mdlw-pomerium-verify.forwardauth.trustForwardHeader=true
        - traefik.http.middlewares.mdlw-pomerium-verify.forwardauth.authResponseHeaders=X-Pomerium-Claim-Email,X-Pomerium-Claim-User,X-Pomerium-Claim-Groups,X-Pomerium-Jwt-Assertion

        ### POMERIUM VERIFY : @mdlw-pomerium-verify
        ## Route and protect the trafic with letsencrypt SSL certificate and authentication
        - traefik.http.routers.goto-pomerium-verify.entrypoints=websecure
        - traefik.http.routers.goto-pomerium-verify.rule=Host(`tv.subdomain.domain.tld`)
        - traefik.http.routers.goto-pomerium-verify.tls=true
        - traefik.http.routers.goto-pomerium-verify.tls.certresolver=letsencrypt-dns-production
        - traefik.http.routers.goto-pomerium-verify.middlewares=mdlw-pomerium-verify@docker
        - traefik.http.services.svc-pomerium-verify.loadbalancer.server.port=8000

I think that the problem can come from the handling of the forwardauth public url by Traefik proxy. But I can’t figure out how to do otherwise. All example I found, use internal pomerium container name, but I’m in swarm environment, the container name is randomly generated after stack and service name.

Anyone have an idea ?

Other additional context

the trafic flow in my environment :
Users —> Front Loadbalancer —> Docker Swarm Cluster [ Traefik --forwardauth–> (Pomerium <–> Keycloak) —> Protected Service ]

1 Like
2

Is there a reason for forward auth mode with traefik, and not using pomerium directly?

3

We use traefik as a reverse-proxy because of its routing and middleware functionalities (rate limiting, headers, redirect, …) for the other parts of our environment.
Pomerium (+ Keycloak) is the new brick that comes to integrate to bring the SSO.

4

Pomerium is built on Envoy and exposes most of its functionality part of the route settings.

May I suggest to try to set up Pomerium in a direct mode first to make sure basic configuration is correct?

5

I have never implement “Envoy Proxy/Pomerium” as reverse proxy, and I can try this kind of setup. But but but I don’t have enough on-premise ressource now to build up similar infrastructure for testing.

Is my actual architecture wrong to work with Pomerium in forwardauth mode ?
Maybe do I need to expose the forwardauth endpoint on another port than the “address” setting port ? Or do I need to explicitly split the different part of Pomerium to better handle the communication/trafic flow ?

6

What I’m suggesting is to just use Pomerium in a regular mode, where you let it proxy traffic in full and also perform authentication and authorization.

The only change you need in your config would be to set an address to 443, and provide TLS certificates (i.e. a wildcard in the example below):

address: :443
certificates:
  - cert: "_wildcard.localhost.pomerium.io.pem"
    key: "_wildcard.localhost.pomerium.io-key.pem"

Pomerium is using Envoy internally already as part of the Proxy component; you do not really need yet another proxy (Traefik) in front of the Pomerium, as it only complicates your deployment and introduces an additional network hop.

image
image686×596 32.3 KB

8

Good morning @denis.
Thank you for your help.

As you suggested, I slightly modified the different configurations to direct the protected services trafic through Pomerium in proxy mode, as below :
Users —> Front Loadbalancer —> Docker Swarm Cluster [ Traefik —proxy mode —> (Pomerium <—> Keycloak) —> Protected Service ].
And after authentication on Keycloak login page, I end up on the home page of the first protected service (containous/whoami), as expected, but I end up with “Error 400 Bad Request” with the second protected service (b4bz/homer).

Here the logs for the second service trough Pomerium :

pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF put id=f1baf911-7ab2-4828-a134-45a3a073d0ce type=type.googleapis.com/session.Session
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM ERR directory: failed to refresh user data error="rpc error: code = Unknown desc = unknown directory provider oidc" X-Forwarded-For=["10.81.234.6, 172.16.99.41,10.10.10.11"] X-Forwarded-Host=["authenticate.subdomain.domain.tld"] X-Forwarded-Port=["443"] X-Forwarded-Proto=["https"] X-Forwarded-Server=["services-proxy"] X-Real-Ip=["10.81.234.6"] ip=127.0.0.1 request-id=f4eca523-f4ea-4229-8194-aee2c1f1dd63 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF get id=f1baf911-7ab2-4828-a134-45a3a073d0ce type=type.googleapis.com/session.Session
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF http-request authority=authenticate.subdomain.domain.tld duration=126.481124 forwarded-for="10.81.234.6, 172.16.99.41,10.10.10.11" method=GET path=/oauth2/callback referer= request-id=f4eca523-f4ea-4229-8194-aee2c1f1dd63 response-code=302 response-code-details=via_upstream service=envoy size=226 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF http-request authority=authenticate.subdomain.domain.tld duration=9.040306 forwarded-for="10.81.234.6, 172.16.99.41,10.10.10.11" method=GET path=/.pomerium/sign_in referer= request-id=7b40220e-0046-4d17-b7fd-a9fc600ba78d response-code=302 response-code-details=via_upstream service=envoy size=790 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF http-request authority=go.subdomain.domain.tld duration=1.925349 forwarded-for="10.81.234.6, 172.16.99.41,10.10.10.14" method=GET path=/.pomerium/callback/ referer= request-id=3eba94cd-3186-4ecc-bc62-fab18aeb6b89 response-code=302 response-code-details=via_upstream service=envoy size=51 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF authorize check allow=true allow-why-true=["user-ok"] check-request-id=68574f5d-9e3c-4e7a-ac43-517da20b6ba6 databroker_record_version=37 databroker_server_version=13404075680448437989 deny=false deny-why-false=["valid-client-certificate-or-none-required"] email=me@emaildomain.tld host=go.subdomain.domain.tld method=GET path=/ query= request-id=ecb37ef0-e8a2-4c85-b5e2-e1f7069e70bc service=authorize session-id=f1baf911-7ab2-4828-a134-45a3a073d0ce user=579c67b0-d9c2-4d36-9695-66ae0d776b58
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF http-request authority=homer_startpage duration=17.368577 forwarded-for="10.81.234.6, 172.16.99.41,10.10.10.14" method=GET path=/ referer= request-id=68574f5d-9e3c-4e7a-ac43-517da20b6ba6 response-code=400 response-code-details=via_upstream service=envoy size=345 upstream-cluster=route-79b6737dc9eda504 user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
pomerium_gateway.1.ynwh03jijqp9@smgr1    | 7:17AM INF refreshing session service=identity_manager session_id=11ebc142-8fcc-4a54-88ed-1304a0e2a4e5 user_id=579c67b0-d9c2-4d36-9695-66ae0d776b58

My routes settings in pomerium config.yaml

routes:
  - from: https://tv.subdomain.domain.tld
    to: http://pomerium_verify:80
    pass_identity_headers: true
    allow_any_authenticated_user: true

  - from: https://go.subdomain.domain.tld
    to: http://homer_startpage:8080
    pass_identity_headers: true
    allow_any_authenticated_user: true
9

response_code_details=via_upstream means that 400 was set by the upstream service (http://homer_startpage:8080). please check its logs.

10

No logs.

When the trafic go through Traefik, there is no error. The page loads successfully.

But when the trafic go through Pomerium (in proxy mode), I end up with “Error 400 Bad Request”.

11

could you change route configuration to: https://httpbin.org and see if it works and which headers are sent to the upstream service, by visiting https://go.subdomain.domain.tld/get in your browser or curl.

if httpbin works, it might be your upstream service requiring some extra header, or maybe it expects an original DNS name - check out host-headers Manage | Pomerium

12

Good moning @denis.

I did it. I replaced to:<myservice>:<port> by to: https://httpbin.org and the request ended up correctly on httpbin homepage.
Hum. I definitively have more work to achieve/problem to solve.

About Pomerium in “forwardauth mode” ?

In experimenting, in “proxy mode”, I observed that after login on Keycloak, I’m redirect back first to authenticate.subdomain.domain.tld/oauth2/callback/... and again to tv.subdomain.domain.tld/.pomerium/callback?... and finally to tv.subdomain.domain.tld. Maybe I need to catchup these endpoints (/.pomerium/ and others ?) and route them back to authenticate.subdomain.domain.tld for processing ? And maybe after that the calling of forwardauth.subdomain.domain.tld/?uri=... will respond with HTTP code 200 instead of starting login process (redirect to authenticate.subdomain.domain.tld/.pomerium/sign_in) ?