Upstream connect error with Pomerium Ingress Controller

Support
1

Hi everyone,

I’m having trouble migrating from Helm chart 24.0 to 25.0 with the new Ingress Controller. After being authenticated with my IdP and after ~5sec of loading, I get upstream connect error or disconnect/reset before headers. reset reason: connection failure. Worked well with Traefik without Pomerium or with Pomerium as forwardauth (chart version 24) but not as an Ingress Controller.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns
    ingress.pomerium.io/allowed_domains: '["mydomain.tld"]'
    ingress.pomerium.io/pass_identity_headers: "true"
    ingress.pomerium.io/preserve_host_header: "true"
  name: app
spec:
  ingressClassName: pomerium-external
  rules:
  - host: app.mydomain.tld
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 8080
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - app.mydomain.tld
    secretName: app.mydomain.tld

---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  clusterIP: 10.100.195.194
  clusterIPs:
  - 10.100.195.194
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: app
  type: ClusterIP

The tricky point is that it works with others upstream apps so the problem might come from my app (Python/Flask/Gunicorn application with pretty every, by default, parameters, unfortunately, I can’t provide the Dockerfile). But so far, I have been unable to find what does not make it work…

Thx for all your help !

2

Here are steps to troubleshoot the connection between pomerium proxy and your service:

A proxy would make an HTTP (or HTTPS, in case secure_upstream annotation is set) to service.namespace.svc.cluster.local:port

Run an interactive debug container in the same namespace you’re running your pomerium proxy:

kubectl run -i --tty busybox --image=busybox --restart=Never -- sh

From there, use curl to make sure your service responds.

curl -v http://app.default.svc.cluster.local:8080/

Note that you requested pass_identity_headers: true which would make pomerium proxy set Host HTTP header to app.mydomain.tld - make sure your service is configured to in fact respond to that.

3

Hey @rguichard, did you ever find the solve for this issue?