What happened?
The logs are full of authorize check entries from the upstream load balancer making /ping requests.
There are so many of these log entries that they overwhelm the number of non-ping authorize check logs and make the logs completely unusable without filtering them out using grep.
What did you expect to happen?
I expected /ping requests to be handled without emitting any log entries.
What’s your environment like?
Official pomerium docker image from docker hub running, on a Google Cloud managed instance group. Multiple all-in-one instances with postgres storage.
Pomerium is behind an Google Cloud HTTP(s) load balancer so that we can have its WAF and cloud shield functionality in front of Pomerium, but we would have the same volume of ping requests even if we used a layer 4 load balancer.
The exact container/version is:
"org.opencontainers.image.created": "2024-11-11T19:58:47Z",
"org.opencontainers.image.name": "pomerium",
"org.opencontainers.image.revision": "3e51b4f90521792cc151bfcf01ead86affdb06f7",
"org.opencontainers.image.source": "https://github.com/pomerium/pomerium",
"org.opencontainers.image.version": "0.28.0",
What’s your config.yaml?
"address": ":443"
"authenticate_service_url": "https://REDACTED"
"routes":
- OMITTED
CERTIFICATE_FILE=/run/secrets/cert.pem
CERTIFICATE_KEY_FILE=/run/secrets/key.pem
COOKIE_SECRET=REDACTED
DATABROKER_STORAGE_CONNECTION_STRING=REDACTED
DATABROKER_STORAGE_TYPE=postgres
IDP_CLIENT_ID=REDACTED
IDP_CLIENT_SECRET=REDACTED
IDP_PROVIDER=REDACTED
IDP_PROVIDER_URL=REDACTED
SHARED_SECRET=REDACTED
SIGNING_KEY=REDACTED
What did you see in the logs?
{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"authorize","request-id":"c860930e-aed0-4cb8-9ce9-63e75333db59","check-request-id":"c860930e-aed0-4cb8-9ce9-63e75333db59","method":"GET","path":"/ping","host":"pomerium","ip":"35.191.206.18","user":"","email":"","allow":true,"allow-why-true":["pomerium-route"],"deny":false,"deny-why-false":[],"time":"2025-02-20T17:26:05Z","message":"authorize check"}
{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"authorize","request-id":"c7023ccc-2fa1-417e-8b42-a0cf99349f12","check-request-id":"c7023ccc-2fa1-417e-8b42-a0cf99349f12","method":"GET","path":"/ping","host":"pomerium","ip":"35.191.206.19","user":"","email":"","allow":true,"allow-why-true":["pomerium-route"],"deny":false,"deny-why-false":[],"time":"2025-02-20T17:26:05Z","message":"authorize check"}
{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"authorize","request-id":"0c168ee0-343e-439a-be43-3c456d1d3fad","check-request-id":"0c168ee0-343e-439a-be43-3c456d1d3fad","method":"GET","path":"/ping","host":"pomerium","ip":"35.191.206.16","user":"","email":"","allow":true,"allow-why-true":["pomerium-route"],"deny":false,"deny-why-false":[],"time":"2025-02-20T17:26:06Z","message":"authorize check"}
Additional context
This is running inside google’s Container-Optimized OS.
It would be a high maintenance burden to take control over the config of the built-in fluentbit that forwards docker container logs to Google Cloud Logging, so in order to avoid having our Google Cloud Logging ingest all these useless log entries, we need to prevent the authorize service from emitting them in the first place.
I don’t want to turn off authorize check logging entirely, I still want a log of every request authorization except for /ping to the pomerium hostname from the load balancer and managed instance group health check.
