Getting connection_failure errors

Support
1

What happened?

I’m unable to get past authenticate portion. When I try to connect to the service, I get redirected to https://authenticate.<my-domain.com>/.pomerium/sign_in… with the following error:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

I tried it with Kubernetes API integration as well as accessing an application behind the proxy.

What’s your environment like?

  • Pomerium version (retrieve with pomerium --version): 30.0.1 deployed with Helm chart 0.16.4
  • Server Operating System/Architecture/Cloud: GCP

What’s your config.yaml?

authenticate:
  existingTLSSecret: pomerium-tls
  idp:
    provider: auth0
    url: https://<something>.us.auth0.com
    clientID: <...>
    clientSecret: <...>
    serviceAccount: <...>
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-staging
    tls:
      secretName: authenticate.pomerium-tls
  proxied: false

proxy:
  existingTLSSecret: pomerium-tls

databroker:
  existingTLSSecret: pomerium-tls
  storage:
    connectionString: rediss://pomerium-redis-master.pomerium.svc.cluster.local
    type: redis
    clientTLS:
      existingSecretName: pomerium-tls
      existingCASecretKey: ca.crt

authorize:
  existingTLSSecret: pomerium-tls

redis:
  enabled: true
  auth:
    enabled: false
  usePassword: false
  generateTLS: true
  tls:
    certificateSecret: pomerium-redis-tls

ingressController:
  enabled: true

config:
  sharedSecret: <...>
  rootDomain: my-domain.com
  existingCASecret: pomerium-tls
  generateTLS: false
  routes:
    - from: https://k8s.<my-domain.com>
      to: https://kubernetes.default.svc.cluster.local
      allow_spdy: true
      tls_skip_verify: false
      kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      policy:
        - allow:
            or:
              - domain:
                  is: <my-domain.com>
    - from: https://authenticate.<my-domain.com>
      to: https://pomerium-authenticate.pomerium.svc.cluster.local
      preserve_host_header: true
      allow_public_unauthenticated_access: false
      policy:
        - allow:
            or:
              - domain:
                  is: <my-domain.com>

{"level":"warn","time":"2022-03-07T04:42:06Z","msg":"stapling OCSP","service":"autocert","error":"no OCSP stapling for [pomerium-proxy.pomerium.svc.cluster.local pomerium-authorize.pomerium.svc.cluster.local pomerium-databroker.pomerium.svc.cluster.local pomerium-authenticate.pomerium.svc.cluster.local]: no OCSP server specified in certificate"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"k8s.<my-domain.com>","path":"/.pomerium/api/v1/login","user-agent":"Go-http-client/2.0","referer":"","forwarded-for":"10.128.0.3","request-id":"adfc8e7a-73d9-4ba1-beae-134c6b41617b","duration":2.987136,"size":417,"response-code":200,"response-code-details":"via_upstream","time":"2022-03-07T04:43:54Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"route-e0da971097cc8326","method":"GET","authority":"authenticate.<my-domain.com>","path":"/.pomerium/sign_in","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36","referer":"","forwarded-for":"10.128.0.4","request-id":"41aa730e-6943-46a5-9120-4278248d4501","duration":4.689853,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T04:43:57Z","message":"http-request}
1 Like
2

Hi, same problem here.

In our case the error is shown every other request: by continuing to reload the page the requests reach their intended targets. Pre 0.16 charts work fine.

    extraTLSSecrets:
      - star-muni-prod
    authenticate:
      existingTLSSecret: pomerium-tls
      idp:
        provider: oidc
        url: "https://keycloak.municipia.eng.it/auth/realms/municipia-int-services"
        clientID: "pomerium"
        clientSecret: "xxx"
      ingress:
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-prod
        tls:
          secretName: authenticate.skyfall.municipia.eng.it
    redis:
      enabled: true
      generateTLS: false
      tls:
        certificateSecret: pomerium-redis-tls
    
    databroker:
      existingTLSSecret: pomerium-tls
      storage:
        clientTLS:
          existingSecretName: pomerium-redis-tls
          existingCASecretKey: ca.crt

    authorize:
      existingTLSSecret: pomerium-tls

    proxy:
      existingTLSSecret: pomerium-tls
      service:
        annotations:
         service.beta.kubernetes.io/aws-load-balancer-type: nlb
         service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Prodotto=trasv"
         service.beta.kubernetes.io/aws-load-balancer-name: pomerium_ingress
         service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    ingressController:
      enabled: true

    config:
      rootDomain: skyfall.municipia.eng.it
      existingCASecret: pomerium-tls    
      sharedSecret: xxx
      cookieSecret: xxx
      generateTLS: false
      generateSigningKey: true
      forceGenerateSigningKey: true
      idp_scopes: profile,email,groups,openid

      routes: .....
3

I have also tried with pomerium 0.17.0, same behaviour.

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"aafc35b8-d97a-4f61-a37f-cc277b2523ed","duration":0.868792,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:25Z","message":"http-request"}

63

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/favicon.ico","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"https://authenticate.skyfall.municipia.eng.it/","forwarded-for":"100.64.0.128","request-id":"6cfedbb6-7926-4608-b40b-d03f6dcdd7a4","duration":7.181327,"size":19,"response-code":404,"response-code-details":"via_upstream","time":"2022-03-07T11:59:26Z","message":"http-request"}

62

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"6650d574-303f-48d4-bc2f-9ca5cd81984e","duration":0.749549,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:27Z","message":"http-request"}

61

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"62b6a57c-7efe-4afc-94d4-ff0ad1c3d8a9","duration":2.990168,"size":34,"response-code":302,"response-code-details":"via_upstream","time":"2022-03-07T11:59:31Z","message":"http-request"}

60

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"97517852-4ce6-497b-813b-a34a11541ed9","duration":0.873362,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:31Z","message":"http-request"}

59

{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"grafana.municipia.eng.it","path":"/api/live/ws","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36","referer":"","forwarded-for":"100.64.0.128","request-id":"d6d3d88a-3aa6-4c14-b876-a9f831481de6","duration":0.268787,"size":0,"response-code":403,"response-code-details":"upgrade_failed","time":"2022-03-07T11:59:32Z","message":"http-request"}

58

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"8129d51d-7357-4d7e-9eaf-79cab39c9b92","duration":45.141876,"size":421,"response-code":302,"response-code-details":"via_upstream","time":"2022-03-07T11:59:33Z","message":"http-request"}

57

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/oauth2/callback","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"fa3f40d7-a962-4ae6-9525-e661d5a4baed","duration":0.862663,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:34Z","message":"http-request"}

56

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/oauth2/callback","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"8933c49d-80f9-46ca-bf69-f5cd71622889","duration":417.554781,"size":79,"response-code":302,"response-code-details":"via_upstream","time":"2022-03-07T11:59:35Z","message":"http-request"}

55

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"4ed03cf6-808c-4e4a-ab4e-4dcd6997e4b6","duration":1.14443,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:35Z","message":"http-request"}

54

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"","forwarded-for":"100.64.0.128","request-id":"747c6465-f5dd-4149-93b6-3530426386f3","duration":32.322123,"size":6717,"response-code":200,"response-code-details":"via_upstream","time":"2022-03-07T11:59:37Z","message":"http-request"}

53

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/index.css","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"https://authenticate.skyfall.municipia.eng.it/.pomerium/","forwarded-for":"100.64.0.128","request-id":"eaa43ad7-073c-4eb4-89bb-d72a9a6ce909","duration":0.742,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:37Z","message":"http-request"}

52

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/index.js","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"https://authenticate.skyfall.municipia.eng.it/.pomerium/","forwarded-for":"100.64.0.128","request-id":"e1f5a9d4-c7ae-41ef-8cae-6455f0306e31","duration":880.65625,"size":739173,"response-code":200,"response-code-details":"via_upstream","time":"2022-03-07T11:59:38Z","message":"http-request"}

51

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/favicon.ico","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"https://authenticate.skyfall.municipia.eng.it/.pomerium/","forwarded-for":"100.64.0.128","request-id":"fb90a0a6-8cbd-4ede-9ee7-8a5ff627c919","duration":0.709498,"size":91,"response-code":503,"response-code-details":"upstream_reset_before_response_started{connection_failure}","time":"2022-03-07T11:59:38Z","message":"http-request"}

50

{"level":"info","service":"envoy","upstream-cluster":"pomerium-ingress-pomerium-authenticate-authenticate-skyfall-municipia-eng-it-c96466237ae869","method":"GET","authority":"authenticate.skyfall.municipia.eng.it","path":"/.pomerium/favicon-32x32.png","user-agent":"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30","referer":"https://authenticate.skyfall.municipia.eng.it/.pomerium/","forwarded-for":"100.64.0.128","request-id":"c27af79f-6b07-4819-aaa7-4ecb757808b5","duration":6.732066,"size":970,"response-code":200,"response-code-details":"via_upstream","time":"2022-03-07T11:59:39Z","message":"http-request"}

49

4

Hi,
i suppose I have found the culprit: by forcing to use ingress controller v 0.15.3 the behaviour is fine again.

5

Hi everyone. I think this is a bug. Opened an issue here to track.

6

Hi,

Thanks for the quick response. I looked at the Github issue and tried adding ingress.pomerium.io/service_proxy_upstream: "true" but I’m still seeing the same error. I also tried to downgrade to v0.15.3 as suggested by raicio and still getting the same error. I’m starting to suspect I might be having a different issue. Also, I’m using Auth0 as IdP and when I visit a page (I use kuard demo example) I don’t get redirected to Auth0 portal, I get upstream connect error or disconnect/reset before headers. reset reason: connection failure right away.

1 Like
7

I think I misunderstood the follow ups as being the same as your issue. Sorry about that @HighWatersDev.

I believe your issue is due to the letsencrypt-staging cert. Since you’re getting a cert successfully, would you mind trying Let’s Encrypt production? There’s some unintuitive certificate behavior on the authenticate backend and the fix hasn’t made its way into the helm chart yet. The issue shouldn’t occur when using production Let’s Encrypt.

8

I did switch to letsencrypt-prod but was still experiencing issues. I eventually found out that switching JWT signing algorithm to RS256 actually solved the problem. I submitted github issue for it here to maybe add that piece to Auth0 documentation.

9

Hi,
we are having some weird behaviour still: tcp routes to RDP connections do not work consinstently. They mainly do not work, but sometimes do.

The same configuration on pomerium deployed with a 0.25.x helm chart works like a charm.

15:21:30

Disconnected from Pomerium: invalid http response code: 503

15:21:20

Connecting to Pomerium…

15:21:19

Authentication required, web browser was open

15:21:19

Connecting to Pomerium…

15:21:16

Disconnected from Pomerium: invalid http response code: 503

15:21:06

Connecting to Pomerium…

15:21:05

Authentication required, web browser was open

15:21:05

Connecting to Pomerium…

15:21:01

Disconnected from Pomerium: invalid http response code: 503

15:20:51

Connecting to Pomerium…

15:20:51

Disconnected from Pomerium: read tcp [::1]:45982->[::1]:59444: wsarecv: An existing connection was forcibly closed by the remote host.

15:20:50

Connected to Pomerium

15:20:50

Connecting to Pomerium…

15:20:49

Authentication required, web browser was open

15:20:49

Connecting to Pomerium…

15:20:46

Disconnected from Pomerium: invalid http response code: 503

15:20:36

Connecting to Pomerium…

15:20:18

Authentication required, web browser was open

15:20:18

Connecting to Pomerium…

15:19:52

Listening for new connections

10

@raicio Please open a new post for different issues. Thanks.

@HighWatersDev [pomerium] use authenticate_internal_service_url by travisgroth · Pull Request #275 · pomerium/pomerium-helm · GitHub should fix the first issue you ran into.

11

@travisgroth ok, will do, thanx.

By the way, I have updated to helm chart 0.31.0 and removed the pinning to ingress controller v0.15.3.
As soon as the ingress controller deployed, i got back the error

“upstream connect error or disconnect/reset before headers. reset reason: connection failure”

but this time on every request. Chart 0.31.0 with ingress controller v0.15.3 works fine.

12

Hi @travisgroth, I’m sorry to report that also chart 31.0.0 does not work: we have a constant
upstream connect error or disconnect/reset before headers. reset reason: connection failure error on the authenticate url.

The only working config so far is: chart 30.1.10 with “v0.15.3” ingress controller.

What else can we do to troubleshoot?

Thank you.

13

@raicio I was able to work around this upstream connect error by setting ingress.enabled to false. Please see Ingress enabled set to true breaks authenticate service · Issue #282 · pomerium/pomerium-helm · GitHub for details. Hope it helps!

2 Likes