Pomerium 500 server error

Hi there! I’m having troubles with my installation and it would be great if someone could give me a hand.

What happened?

My Pomerium setup is giving me a response “500” if i try to access the only route in my config file “mything”, this is after asking for GitHub access and successfully granting it. The interesting thing is that accessing “/.pomerium” does show my GitHub info only without any session-id nor a expires at date.

Interestingly, it also shows that my cookies are 0b. Also this behaviour is on both Brave (Chrome-based) and Firefox, on two different computers and even with a private window.

What did you expect to happen?

I expected to see the page of the

How’d it happen?

  1. Went into “mydomain.com/mything
  2. Logged-in through GitHub
  3. Authorized my website through GitHub.
    4.After what felt like a long time for a request i got a 500 response.

What’s your environment like?

  • Version: 0.21.3
  • OS: Raspberry Pi OS 11 (bullseye)
  • Installed through comntainer
  • Raspberry Pi OS - compatible version

What’s your config.yaml?

authenticate_service_url: "https://mydomain.com"
autocert: true
autocert_email: myemail@someprovider.com
idp_client_id: ****
idp_client_secret: ****
idp_provider: github
log_level: debug
cookie_secret: "****"
routes:
  - from: "https://mydomain.com"
    path: /mything
    preserve_host_header: true
    allow_any_authenticated_user: true
    policy:
      - allow:
          or:
            - email:
                is: myemail@someprovider.com
    to: "http://127.0.0.1:5001"

What did you see in the logs?

Note: this is just a snippet with what I feel like it’s important full logs below.

{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"mydomain.com","path":"/.well-known/pomerium/hpke-public-key","user-agent":"Go-http-client/2.0","referer":"","forwarded-for":"'client_ip'","request-id":"7ba8bad5-8183-4e8b-ae28-2103791ec34d","duration":2.098223,"size":0,"response-code":304,"response-code-details":"via_upstream","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"mydomain.com","path":"/mything","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","referer":"","forwarded-for":"'client_ip'","request-id":"1afe4346-efc6-434a-872b-4f2dfcec6b64","duration":171.664448,"size":1429,"response-code":302,"response-code-details":"ext_authz_denied","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"mydomain.com","path":"/.pomerium/sign_in","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","referer":"","forwarded-for":"'client_ip'","request-id":"afeb2b5f-ccc8-46ac-8b71-aef14d7a3ea1","duration":9.907307,"size":793,"response-code":302,"response-code-details":"via_upstream","time":"2023-05-01T07:41:55Z","message":"http-request"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","emails":[{"email":"sertorbe@gmail.com","verified":true,"primary":true,"visibility":"public"}],"time":"2023-05-01T07:41:55Z","message":"github: user emails"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","emails":[{"email":"sertorbe@gmail.com","verified":true,"primary":true,"visibility":"public"}],"time":"2023-05-01T07:41:56Z","message":"github: user emails"}
{"level":"error","time":"2023-05-01T07:41:56Z"}
{"level":"debug","ip":"127.0.0.1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0","request-id":"6d0b5817-4e7b-4b48-84ad-b0ba100bdd37","duration":1293.06225,"size":474,"status":302,"method":"GET","host":"mydomain.com","path":"/oauth2/callback?code=f05a745082dd1ed0ccdb&state=N2hYUDMyeXRFMmdEZGpBY0xYZi9idTEwRk9pai90RC90QUlaeFR2SnVoa2RJbFU1aVFxNE5CaWRhRWdBdEkrZ3JtQWZGaXNsTnV0U2lNd3RGc2xFU1E9PXwxNjgyOTI2OTE0fNNiLsdr6zSVPHlhIph_4yj6oRhdRoTjDAPaGcToPhp1wI8PtXvy_hQqLJosgfJXWGwe2F5SfF1H0lSNukIBVB71DF5wjyiOZSSr4gePml4zk83obX9bz7os9MoRz6PQseXFcPyZYUIZYFolnjIN3rZ5OFFgnFhByCE4aHw7RtJmvgLXuPkuPc6s-lFsYbin5htYVDdTLZK4f_bb5mJzZ-9iQVzhpsyQA9-3Eo-MqqBEmUSf6WmBtDTUwAGlCZH_USscEf0g8ylfD6FzRzPGezmMCZwpLFEHgNpfw-LMckc-gGnQ8jesaX4541VR4PgH98RBB9Q93YswCMh_i9hYZVurvfaD-CL1o6f59iBQVfAwUbOZDID_rCTWVHF1KzN5KpbM2NFOgbYP6mH_OgOe6mSFgY8SLT0kTuKLKku6KCUWU8H0FWwv3HVAStjcekKwef3QRwdEi5Xq0Ll4VPHjDJvFLsPLcfhwes7JYMQMoi7wEyIe5lsAo_PISpTei_99T4MaNaxVH33Wg30DdPKjpagPYGoa5PCzzqgiCajYxCwffM8yOXC_OpN_6in54WikZ31IvkeO06HBBK2niMYMz3Wk1pN1oRTXnLr08vEJHKBESc__LGbnU-3HmPtVUMB25CttiMkRtjc%3D","time":"2023-05-01T07:41:56Z","message":"http-request"}

Full logs: Pomerium log · GitHub

Additional context

This happened on both Brave (Chrome-based) and Firefox on different computers, with cookies cleared and even on a private window. Also, note that I’m using my own docker container that’s compatible with Raspberry Pi OS.

  1. Please leave either policy or allow_any_authenticated_user, I do not think this is valid combo.
  2. the ext_authz_denied means that your request was denied.
  3. However I do not see the log message from authorize service that would spell out the decision. Please see Audit Logs | Pomerium

Your comment gave me the idea of trying allow_public_unauthenticated_access: true and this let me use the service, so the problem is definetely on the authorize service.
The thing is after analyzing allow-why-false I see "user-unauthenticated" which is weird considering that my GitHub user info is shown in the logs and even in “/.pomerium”, any idea of why this is happening? Could be related to my 0b cookie?

yes if your _pomerium cookie is empty then your browser do not have a session to pass to Pomerium.

I understand you built Pomerium yourself - I’d recommended rolling back to earlier releases that are compatible with RPi and you make it working there.

It took me some time but I’ve been able to test this. I’ve tried two things. first, I installed on an Ubuntu image for raspberry (and thus using the official image) and got the same problem as with my custom image.

Then, I tried what you suggested and used an old version pre-envoy (0.8.4, as it’s the last one). At least I got as far as getting a 404 but it seems it is not detectung the path (and not giving it one results in 404 either way), here’s the config and the log: https://gist.github.com/sheosi/2eb14cec7da0a84ed73246df5aaa7fac

Another thing I’ve tried is using Google instead of GitHub (with the latest version), same result, only that something is clear now, when accessing “/” Google asks me once for my username and everything is fine, when going to a subpath Google asks me twice, it really seems that my email is not detected as it should, somehow. Any ideas?

I cannot comment about version 0.8 as it’s too old.

What I meant is to find latest version with envoy that would work on RPI.

Also please note the security advisory: [security] Pomerium v0.22.2, v0.21.4, v0.20.1, v0.19.2, v0.18.1, and v0.17.4 release

Any version of Envoy pre-tcmalloc would be ancient (I can’t exactly pinpoint which one but definetely one before february of 2020, I saw it being called new on an isue, making it older than pomerium 0.8).

At this point I’m sure this is not related to Rpi anymore, the issue with broken Envoy was with Rpi OS only and was reported to work on other OSs, with my testing having the same result on Ubuntu and appeareance of the weird empty error, this looks much more like a Pomerium bug to me.

Hey, and thanks for the heads up!!

Let’s keep one thread per problem please, otherwise it’s very hard to follow.

I think there’s nothing to be done about RPi, Envoy would not be working there anytime soon. Envoy 1.17-1.23 unable to start on Raspberry · Issue #23339 · envoyproxy/envoy · GitHub

If you experience an issue on any other platform (i.e. Ubuntu / amd64) please do the following:

  1. Use most recent version (v0.22.2)
  2. Try the quickstart exactly as it’s written, without alternations Run Pomerium Core With Docker | Pomerium - if it doesn’t work for you, please open a new thread.

So, it seems that thia has been solved. It turns out that my config was causing a problem, Pomerium doesn’t like when a path is the same as the authenticate URL, it makes it loop on authentication resulting in the unnamed error you can see on the logs, this is definetely a bug.

Another problen was that I was using the path variable when what I meant was the prefix one.

So far, with that change it seems to work (there are some services I’m still having trouble with, but at least one of them is working), and what’s more, is that what is working is my raspios version.

Clarifification: I know that vanilla envoy doesnt work on RaspiOS, however, I made a version of the pomerium docker with an envoy version that actually does work there, just in case you are interested.