Pomerium 500 server error

Hi there! I’m having troubles with my installation and it would be great if someone could give me a hand.

What happened?

My Pomerium setup is giving me a response “500” if i try to access the only route in my config file “mything”, this is after asking for GitHub access and successfully granting it. The interesting thing is that accessing “/.pomerium” does show my GitHub info only without any session-id nor a expires at date.

Interestingly, it also shows that my cookies are 0b. Also this behaviour is on both Brave (Chrome-based) and Firefox, on two different computers and even with a private window.

What did you expect to happen?

I expected to see the page of the

How’d it happen?

  1. Went into “mydomain.com/mything
  2. Logged-in through GitHub
  3. Authorized my website through GitHub.
    4.After what felt like a long time for a request i got a 500 response.

What’s your environment like?

  • Version: 0.21.3
  • OS: Raspberry Pi OS 11 (bullseye)
  • Installed through comntainer
  • Raspberry Pi OS - compatible version

What’s your config.yaml?

authenticate_service_url: "https://mydomain.com"
autocert: true
autocert_email: myemail@someprovider.com
idp_client_id: ****
idp_client_secret: ****
idp_provider: github
log_level: debug
cookie_secret: "****"
routes:
  - from: "https://mydomain.com"
    path: /mything
    preserve_host_header: true
    allow_any_authenticated_user: true
    policy:
      - allow:
          or:
            - email:
                is: myemail@someprovider.com
    to: "http://127.0.0.1:5001"

What did you see in the logs?

Note: this is just a snippet with what I feel like it’s important full logs below.

{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"mydomain.com","path":"/.well-known/pomerium/hpke-public-key","user-agent":"Go-http-client/2.0","referer":"","forwarded-for":"'client_ip'","request-id":"7ba8bad5-8183-4e8b-ae28-2103791ec34d","duration":2.098223,"size":0,"response-code":304,"response-code-details":"via_upstream","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"mydomain.com","path":"/mything","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","referer":"","forwarded-for":"'client_ip'","request-id":"1afe4346-efc6-434a-872b-4f2dfcec6b64","duration":171.664448,"size":1429,"response-code":302,"response-code-details":"ext_authz_denied","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"mydomain.com","path":"/.pomerium/sign_in","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","referer":"","forwarded-for":"'client_ip'","request-id":"afeb2b5f-ccc8-46ac-8b71-aef14d7a3ea1","duration":9.907307,"size":793,"response-code":302,"response-code-details":"via_upstream","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","emails":[{"email":"sertorbe@gmail.com","verified":true,"primary":true,"visibility":"public"}],"time":"2023-05-01T07:41:55Z","message":"github: user emails"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","emails":[{"email":"sertorbe@gmail.com","verified":true,"primary":true,"visibility":"public"}],"time":"2023-05-01T07:41:56Z","message":"github: user emails"}
{"level":"error","time":"2023-05-01T07:41:56Z"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","duration":1293.06225,"size":474,"status":302,"method":"GET","host":"mydomain.com","path":"/oauth2/callback?code=f05a745082dd1ed0ccdb&state=N2hYUDMyeXRFMmdEZGpBY0xYZi9idTEwRk9pai90RC90QUlaeFR2SnVoa2RJbFU1aVFxNE5CaWRhRWdBdEkrZ3JtQWZGaXNsTnV0U2lNd3RGc2xFU1E9PXwxNjgyOTI2OTE0fNNiLsdr6zSVPHlhIph_4yj6oRhdRoTjDAPaGcToPhp1wI8PtXvy_hQqLJosgfJXWGwe2F5SfF1H0lSNukIBVB71DF5wjyiOZSSr4gePml4zk83obX9bz7os9MoRz6PQseXFcPyZYUIZYFolnjIN3rZ5OFFgnFhByCE4aHw7RtJmvgLXuPkuPc6s-lFsYbin5htYVDdTLZK4f_bb5mJzZ-9iQVzhpsyQA9-3Eo-MqqBEmUSf6WmBtDTUwAGlCZH_USscEf0g8ylfD6FzRzPGezmMCZwpLFEHgNpfw-LMckc-gGnQ8jesaX4541VR4PgH98RBB9Q93YswCMh_i9hYZVurvfaD-CL1o6f59iBQVfAwUbOZDID_rCTWVHF1KzN5KpbM2NFOgbYP6mH_OgOe6mSFgY8SLT0kTuKLKku6KCUWU8H0FWwv3HVAStjcekKwef3QRwdEi5Xq0Ll4VPHjDJvFLsPLcfhwes7JYMQMoi7wEyIe5lsAo_PISpTei_99T4MaNaxVH33Wg30DdPKjpagPYGoa5PCzzqgiCajYxCwffM8yOXC_OpN_6in54WikZ31IvkeO06HBBK2niMYMz3Wk1pN1oRTXnLr08vEJHKBESc__LGbnU-3HmPtVUMB25CttiMkRtjc%3D","time":"2023-05-01T07:41:56Z","message":"http-request"}

Full logs: Pomerium log · GitHub

Additional context

This happened on both Brave (Chrome-based) and Firefox on different computers, with cookies cleared and even on a private window. Also, note that I’m using my own docker container that’s compatible with Raspberry Pi OS.

  1. Please leave either policy or allow_any_authenticated_user, I do not think this is valid combo.
  2. the ext_authz_denied means that your request was denied.
  3. However I do not see the log message from authorize service that would spell out the decision. Please see Audit Logs | Pomerium

Your comment gave me the idea of trying allow_public_unauthenticated_access: true and this let me use the service, so the problem is definetely on the authorize service.
The thing is after analyzing allow-why-false I see "user-unauthenticated" which is weird considering that my GitHub user info is shown in the logs and even in “/.pomerium”, any idea of why this is happening? Could be related to my 0b cookie?

yes if your _pomerium cookie is empty then your browser do not have a session to pass to Pomerium.

I understand you built Pomerium yourself - I’d recommended rolling back to earlier releases that are compatible with RPi and you make it working there.

It took me some time but I’ve been able to test this. I’ve tried two things. first, I installed on an Ubuntu image for raspberry (and thus using the official image) and got the same problem as with my custom image.

Then, I tried what you suggested and used an old version pre-envoy (0.8.4, as it’s the last one). At least I got as far as getting a 404 but it seems it is not detectung the path (and not giving it one results in 404 either way), here’s the config and the log: https://gist.github.com/sheosi/2eb14cec7da0a84ed73246df5aaa7fac

Another thing I’ve tried is using Google instead of GitHub (with the latest version), same result, only that something is clear now, when accessing “/” Google asks me once for my username and everything is fine, when going to a subpath Google asks me twice, it really seems that my email is not detected as it should, somehow. Any ideas?

I cannot comment about version 0.8 as it’s too old.

What I meant is to find latest version with envoy that would work on RPI.

Also please note the security advisory: [security] Pomerium v0.22.2, v0.21.4, v0.20.1, v0.19.2, v0.18.1, and v0.17.4 release