Upstream connect error with Pomerium Ingress Controller

Hi everyone,

I’m having trouble migrating from Helm chart 24.0 to 25.0 with the new Ingress Controller. After being authenticated with my IdP and after ~5sec of loading, I get upstream connect error or disconnect/reset before headers. reset reason: connection failure. Worked well with Traefik without Pomerium or with Pomerium as forwardauth (chart version 24) but not as an Ingress Controller.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns
    ingress.pomerium.io/allowed_domains: '["mydomain.tld"]'
    ingress.pomerium.io/pass_identity_headers: "true"
    ingress.pomerium.io/preserve_host_header: "true"
  name: app
spec:
  ingressClassName: pomerium-external
  rules:
  - host: app.mydomain.tld
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 8080
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - app.mydomain.tld
    secretName: app.mydomain.tld
---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  clusterIP: 10.100.195.194
  clusterIPs:
  - 10.100.195.194
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: app
  type: ClusterIP

The tricky point is that it works with others upstream apps so the problem might come from my app (Python/Flask/Gunicorn application with pretty every, by default, parameters, unfortunately, I can’t provide the Dockerfile). But so far, I have been unable to find what does not make it work…

Thx for all your help !

Here are steps to troubleshoot the connection between pomerium proxy and your service:

A proxy would make an HTTP (or HTTPS, in case secure_upstream annotation is set) to service.namespace.svc.cluster.local:port

Run an interactive debug container in the same namespace you’re running your pomerium proxy:

kubectl run -i --tty busybox --image=busybox --restart=Never -- sh

From there, use curl to make sure your service responds.

curl -v http://app.default.svc.cluster.local:8080/

Note that you requested pass_identity_headers: true which would make pomerium proxy set Host HTTP header to app.mydomain.tld - make sure your service is configured to in fact respond to that.

Hey @rguichard, did you ever find the solve for this issue?