Proxy Protocol Causing Issues with Non-HTTPS Requests

Support
1

When enabling the proxy protocol on both the AWS NLB and the Pomerium Ingress Controller, all non-HTTPS requests (handled by the redirect server) return a 400 Bad Request error. This behavior is preventing ACME HTTP-01 challenges from functioning correctly.

Has anyone encountered this issue before or found a workaround to make ACME HTTP-01 challenges work in this setup?

2

I have not attempted to do this. Is the AWS NLB also terminating port 80 traffic? And is it also using the proxy protocol for this traffic to Pomerium (not just the HTTPS traffic but also the HTTP traffic)?

I believe the Pomerium redirect server does not support the proxy protocol. This might explain the issue. I suppose this would be a bug and if use_proxy_protocol is set it should also apply to the redirect server. I can create an issue if this sounds like what’s wrong.

Possible workarounds:

  1. Could you disable the proxy protocol just for port 80 traffic?
  2. Could you implement the redirect in a different way? Does the NLB support doing HTTP to HTTPS redirects?
3

Thank you for your reply.

Yes, the AWS NLB is terminating traffic on port 80 and using the proxy protocol for HTTP traffic.

Regarding the workarounds:

  1. As far as I know, there is currently no way to enable or disable the proxy protocol for a specific port (or target group) when using a Kubernetes Service of type LoadBalancer.
  2. Unfortunately, it does not seem possible.
4

Thanks for the update. I created this issue http_redirect_addr incompatible use_proxy_protocol · Issue #5403 · pomerium/pomerium · GitHub.

I agree that the redirect server should also be using the proxy protocol. We will need to update the code.

5

Hi @renatomjr ,

A fix for this was merged to main. It should make the http redirect server understand the proxy protocol. We will be doing a release in the new year that will include these changes.

While working on this I discovered that autocert is not compatible with the proxy protocol. The library we use to provision certificates doesn’t have the flexibility we need to get it to work, so fixing that problem is a bigger project. If autocert is needed, a workaround is to use Pomerium Zero custom domains, which can provision certificates via DNS records.