Pomerium Verify Identity Verification Failed

What happened?

I was following the Pomerium docker quickstart guide and Pomerium Verify cannot identify me. https://authenticate.<base_url> can identify me correctly, but https://verify.<base_url> says it did not get the JSON web key set. I also tried browsing to https://authenticate.<base_url>/.well-known/pomerium/jwks.json, but that returned keys: null. The certificates are valid and I have DNS running such that <base_url> resolves to an IP in my LAN network.

What did you expect to happen?

I expected jwks.json to return a valid key and user information to be shown at the verify endpoint.

How’d it happen?

  1. Visited the verify endpoint.
  2. Redirected to authentication endpoint.
  3. Authentication is properly enforced (user’s that don’t have access get redirected to an access denied page)
  4. Verify endpoint displays error message:
    Identity verification failed – We tried to verify the incoming user, but failed with the following error: couldn't get json web key: empty JSON Web Key Set payload

What’s your environment like?

Ubuntu 22.04.1 LTS
Raspberry Pi
Docker Running Pomerium/Pomerium and Pomerium/Verify containers.

What’s your config.yaml?

# See detailed configuration settings : https://www.pomerium.com/docs/reference/


# this is the domain the identity provider will callback after a user authenticates
authenticate_service_url: https://authenticate.<base_url>

####################################################################################
# Certificate settings:  https://www.pomerium.com/docs/reference/certificates.html #
# The example below assumes a certificate and key file will be mounted to a volume #
# available to the  Docker image.                                                  #
####################################################################################
certificate_file: /pomerium/cert.pem
certificate_key_file: /pomerium/privkey.pem

##################################################################################
# Identity provider settings : https://www.pomerium.com/docs/identity-providers/ #
# The keys required in this section vary depending on your IdP. See the          #
# appropriate docs for your IdP to configure Pomerium accordingly.               #
##################################################################################
idp_provider: auth0
idp_provider_url: <auth0_url>
idp_client_id: <client_id>
idp_client_secret: <client_secret>

# Generate 256 bit random keys  e.g. `head -c32 /dev/urandom | base64`
cookie_secret: <cookie_secret>
# https://pomerium.com/reference/#routes
routes:
  - from: https://verify.<base_url>
    to: http://verify:8000
    policy:
      - allow:
          or:
            - email:
                is: <approved_user>@gmail.com
    pass_identity_headers: true

What did you see in the logs?

Two notable logs:

pomerium_1  | {"level":"warn","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"identity_manager","error":"auth0: could not list roles: request failed: Get \"https://<auth0_url>/api/v2/roles?include_totals=true&page=0&per_page=50\": oauth2: cannot fetch token: 403 Forbidden\nResponse: {\"error\":\"access_denied\",\"error_description\":\"Client is not authorized to access \\\"https://<auth0_url>/api/v2/\\\". You need to create a \\\"client-grant\\\" associated to this API. See: https://auth0.com/docs/api/v2#!/Client_Grants/post_client_grants\"}","time":"2022-10-16T02:17:21Z","message":"failed to refresh directory users and groups"}

pomerium_1  | {"level":"info","service":"envoy","upstream-cluster":"route-60bdd4beda1b36dd","method":"GET","authority":"verify","path":"/api/verify-info","user-agent":"<user_agent>","referer":"https://verify.<base_url>/","forwarded-for":"<pomerium_ip>","request-id":"3d53d7f9-cb06-498c-bf59-dc2e2bc01d08","duration":56.566525,"size":202,"response-code":200,"response-code-details":"via_upstream","time":"2022-10-16T02:21:03Z","message":"http-request"}

Additional context

N/A

you need define signing_key in your config, please see this guide: Getting the user's identity | Pomerium

Thank you that worked like a charm! Should that link be added to the tutorial I linked above? I don’t know how I would have found it without you. I can make a request on the GitHub if you think it would be relevant.