I’m sure we’re close here. I disabled redis for the time being to rule that out of the equation - now I’m getting a 403 forbidden when I navigate to verify.dev.sw.io and upstream connect error or disconnect/reset before headers. reset reason: connection failure when I go to authenticate.dev.sw.io . I ruled out certificate errors by using a prod let’s encrypt issuer.
Logs:
pomerium-authorize-d87bfd878-dvfxq pomerium {"level":"info","service":"authorize","request-id":"f1644a25-0aca-4b0b-a677-a96c11edd46b","check-request-id":"c64ea404-3cd8-4437-a8a7-5b1704482312","method":"GET","path":"/","host":"verify.tools.dev.sw.io","query":"","allow":false,"allow-why-false":["non-pomerium-route"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"","email":"","databroker_server_version":2427717062922184798,"databroker_record_version":18,"time":"2022-03-22T15:01:00Z","message":"authorize check"}
pomerium-authorize-d87bfd878-dvfxq pomerium {"level":"info","service":"authorize","request-id":"e0cb356c-27a1-4426-bb3a-de5b62e97c6f","check-request-id":"5835ff2a-7e70-4641-8d40-9f890209b87b","method":"GET","path":"/","host":"verify.tools.dev.sw.io","query":"","allow":false,"allow-why-false":["non-pomerium-route"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"","email":"","databroker_server_version":2427717062922184798,"databroker_record_version":18,"time":"2022-03-22T15:01:13Z","message":"authorize check"}
pomerium-authorize-d87bfd878-dvfxq pomerium {"level":"info","service":"authorize","request-id":"33851a3f-8404-4f3b-9468-ff297aeabd9f","check-request-id":"4070ccee-ae16-457c-a004-946c0867ed92","method":"GET","path":"/","host":"verify.tools.dev.sw.io","query":"","allow":false,"allow-why-false":["non-pomerium-route"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"","email":"","databroker_server_version":2427717062922184798,"databroker_record_version":18,"time":"2022-03-22T15:03:56Z","message":"authorize check"}
pomerium-proxy-77448fd8b6-wngpl pomerium {"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"verify.tools.dev.sw.io","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36","referer":"","forwarded-for":"10.0.25.163","request-id":"999cb853-d268-4e3b-8fa6-ae442f432976","duration":0.717767,"size":302,"response-code":302,"response-code-details":"via_upstream","time":"2022-03-22T15:04:10Z","message":"http-request"}
pomerium-proxy-77448fd8b6-wngpl pomerium {"level":"info","service":"envoy","upstream-cluster":"pomerium-pomerium-authenticate-authenticate-tools-dev-sw-io-eb4679af2e931dfc","method":"GET","authority":"authenticate.tools.dev.sw.io","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36","referer":"","forwarded-for":"10.0.47.110","request-id":"6a3824de-e66d-4db9-b817-4863820d407c","duration":0.450253,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-22T15:04:10Z","message":"http-request"}
Update: I got further by turning off TLS in the backend, so it seems that there is some issue with the backend certificates. Specifically, I set config.insecure to true temporarily and I got my redirect to Google Auth. Much further than I’ve gotten before!
Update 2: I clarified that the 403 forbidden when navigating to https://verify.dev.sw.io is happening regardless of whether insecure is set to true, so there is still something else going on. The authorize check is failing and not redirecting to my authentication service.
pomerium-authorize-658d577748-qbzxt pomerium {"level":"info","service":"authorize","request-id":"446c751e-4fc2-4394-81dd-4f4c6a6c4635","check-request-id":"99386579-d746-4559-b5f1-e8b37b63a349","method":"GET","path":"/","host":"verify.tools.dev.sw.io","query":"","allow":false,"allow-why-false":["non-pomerium-route"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"","email":"","databroker_server_version":9387271978945274956,"databroker_record_version":50,"time":"2022-03-22T19:54:09Z","message":"authorize check"}
pomerium-proxy-b74d99bfb-9745j pomerium {"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"verify.tools.dev.sw.io","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36","referer":"","forwarded-for":"10.0.2.171","request-id":"99386579-d746-4559-b5f1-e8b37b63a349","duration":3.854942,"size":11849,"response-code":403,"response-code-details":"ext_authz_denied","time":"2022-03-22T19:54:09Z","message":"http-request"}
