500 error while Running pomerium-helm on eks behind NLB

Support
1

What happened?

After creating a tunnel using

$ pomerium-cli tcp  worker1-iap.orgdomain.io:22
listening on 127.0.0.1:34523
error serving local connection: invalid http response code: 500

from a second terminal, while sshing its throwing 500

$ssh suser@localhost -p 34523

This is the corresponding log from pomerium-proxy pod

{"level":"info","service":"envoy","upstream-cluster":"","method":"CONNECT","authority":"worker1-iap.orgdomain.io:22","path":"","user-agent":"Go-http-client/1.1","referer":"","forwarded-for":"10.0.0.198","request-id":"ff1f4271-7603-457e-b70f-458dd749d2a3","duration":0.352889,"size":0,"response-code":500,"response-code-details":"ext_authz_error","time":"2022-04-20T16:39:12Z","message":"http-request"}

What did you expect to happen?

it to open auth page on browser and redirect after login

How’d it happen?

I am running pomerium-helm without TLS on eks and exposing it via NLB.
TLS termination happens at NLB and it forwards to http port of service.

What’s your environment like?

  • Pomerium version (retrieve with pomerium --version):
    image: pomerium/pomerium:v0.17.1
  • Server Operating System/Architecture/Cloud:
    helm.sh/chart: pomerium-31.1.3

What’s your config.yaml?

config:
  rootDomain: Orgdomain.io
  address: :80
  insecure: true
  grpc_address: :80
  insecure_server: true
  insecureProxy: true
  ingressController: false
  shared_secret: XXXXXXXXXXXXXXXXXXXXXX=
  cookie_secret: XXXXXXXXXXXXXXXXXXXX=
  routes:
    - from: tcp+https://worker1-iap.orgdomain.io:22
      to: tcp://10.0.105.253:22
      allowed_domains:
        - orgdomain.io
    - from: tcp+https://worker2-iap.orgdomain.io:22
      to: tcp://10.0.105.252:22
      allowed_domains:
        - orgdomain.io

What did you see in the logs?

{"level":"info","service":"envoy","upstream-cluster":"","method":"CONNECT","authority":"worker1-iap.orgdomain.io:22","path":"","user-agent":"Go-http-client/1.1","referer":"","forwarded-for":"10.0.0.198","request-id":"ff1f4271-7603-457e-b70f-458dd749d2a3","duration":0.352889,"size":0,"response-code":500,"response-code-details":"ext_authz_error","time":"2022-04-20T16:39:12Z","message":"http-request"}

Could someone help me to understand what is happening here?

1 Like
2
  • do you have IdP set up?
  • do any of the HTTP-only routes work ?
  • there must be an additional info from authorize service elaborating the reason for rejection
3

Yes I’ve configured google IdP,
authenticate endpoint is tested to work by exposing as ingress endpoint

not getting any log on authorize side while this errors out on proxy side

4

do any of the HTTP-only routes work?

5

yes, the exposed authenticate endpoint worked fine

6
  1. there must be a corresponding error in authorize pod with further details
  2. Normally you should have your browser open, note there is a browser-cmd option if there’s an issue with your default browser opening command.
  3. is your certificate chain trusted on the machine you’re running pomerium-cli? try disable-tls-verification option.