Kubernetes | Ingress Controller | v0.18.0 | Ingress controller is processing non-pomerium ingresses

ulfox · August 24, 2022, 8:48pm

What happened?

Hi team,

In the ingress documentation, there is this warning block

Only one Ingress Controller instance is supported per Pomerium cluster.
Does this mean, 1 Pomerium ingress controller or 1 ingress controller in the whole cluster?

The reason I am asking is because after we upgraded Pomerium to latest helm chart v32.0.5 (Pomerium v0.18.0), we noticed that all our non-pomerium ALB (e.g. prometheus, thanos) ingresses in the cluster were processed by the Pomerium ingress controller (which broke the connectivity)

Is there a way to configure the ingress controller to only process ingress objects that have explicitly set the Pomerium as an ingress class?

What did you expect to happen?

Pomerium processing only ingresses with a Pomerium ingressClassName or annotation

How’d it happen?

Upgraded to Pomerium v0.18.0, helm chart 32.0.5

What’s your environment like?

Pomerium version (retrieve with pomerium --version): v0.18.0
Server Operating System/Architecture/Cloud: EKS v1.21.0

What’s your config.yaml?

authenticate:
  existingTLSSecret: pomerium-tls
  idp:
    clientID: ...
    clientSecret: ...
    provider: google
    serviceAccount: ...
  ingress:
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
    tls:
      secretName: authenticate.prod.aws.metrika.co-tls
  proxied: false
authorize:
  existingTLSSecret: pomerium-tls
config:
  cookieSecret: ...
  existingCASecret: pomerium-tls
  generateSigningKey: false
  generateTLS: false
  rootDomain: prod.aws.metrika.co
  sharedSecret: ...
  signingKey: ...
databroker:
  existingTLSSecret: pomerium-tls
  storage:
    clientTLS:
      existingCASecretKey: ca.crt
      existingSecretName: pomerium-tls
    connectionString: rediss://pomerium-redis-master.pomerium.svc.cluster.local
    type: redis
ingress:
  enabled: false
ingressController:
  enabled: true
proxy:
  existingTLSSecret: pomerium-tls
redis:
  auth:
    enabled: false
  enabled: true
  generateTLS: false
  tls:
    certificateSecret: pomerium-redis-tls
  usePassword: false

denis · August 25, 2022, 9:58pm

The warning means that when you have a Pomerium installation, it may only be controlled by a single ingress controller.

Pomerium is expected to coexist with other ingress controllers, as it should only process ingresses that belong to the pomerium class.

You should have the following automatically created IngressClass

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: pomerium
spec:
  controller: pomerium.io/ingress-controller

The only situation when Pomerium ingress controller would process other Ingresses is when the following two conditions are met:

the Ingress object does not refer to any ingressClass in the spec,
the Pomerium IngressClass is marked as a default controller, via ingressclass.kubernetes.io/is-default-class: "true" annotation

When Pomerium processes the Ingresses for reconciliation, for all non-managed ingresses, it should emit the following warning, that you should be able to observe via logs of the pomerium-ingress-controller pod.

{
  "level": "info",
  "ts": 1661463838.4125936,
  "logger": "controller.ingress",
  "msg": "deleted from pomerium",
  "reconciler group": "networking.k8s.io",
  "reconciler kind": "Ingress",
  "name": "httpbin",
  "namespace": "default",
  "reason": "not marked to be managed by this controller"
}

I tried to reproduce your situation by installing both NGINX and Pomerium ingress controllers, and I do not seem to be able to reproduce your issue, when different ingresses refer to different ingressClass.

ulfox · August 25, 2022, 10:31pm

Hi @denis, thanks for the reply.

I am sorry for not mentioning the ingress controller name, we are using aws-load-balancer-controller

Github Repo
Chart version: v2.3.1

Some of the affected ingresses, are normal ALB ingresses, e.g. Prometheus & Thanos ingresses. They all have the same structure as below

NAMESPACE    NAME                          CLASS      HOSTS ...
monitoring       thanos-query              <none>     FQDN   ...

Example manifest structure from Thanos

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
    alb.ingress.kubernetes.io/group.name: internal-default
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/load-balancer-name: ...
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: thanos
    meta.helm.sh/release-namespace: monitoring
  name: thanos-query
  namespace: monitoring
spec:
  rules:
  - host: FQDN
    http:
      paths:
      - backend:
          service:
            name: thanos-query
            port:
              name: http
        path: /
        pathType: Prefix

Pomerium installation before the issue

Values (see original post)
Namespace: pomerium
Helm Version: pomerium-31.2.6
Application Version: 0.17.3

The issue appeard by doing the following steps:

Upgraded Pomerium to → v0.18.0, helm chart 32.0.5
Ingresses with annotation alb changed the loadbalancer from AWS Controller LB to Pomerium’s Controller LB
External DNS updated the records and then we became aware of the issue

Pomerium Ingress controller logs for Thanos before the upgrade

"log":"{\"level\":\"info\",\"ts\":1661310404.1244335,\"logger\":\"controller.ingress\",\"msg\":\"deleted from pomerium\",\"reconciler group\":\"networking.k8s.io\",\"reconciler kind\":\"Ingress\",\"name\":\"thanos-query\",\"namespace\":\"monitoring\",\"reason\":\"not marked to be managed by this controller\"}\n","stream":"stderr","time":"2022-08-24T03:06:44.124920593Z"}

As you can see above Pomerium is logging that this ingress is not makred to be manged by this controller

Pomerium Ingress controller logs for Thanos after the upgrade

"{\"level\":\"info\",\"ts\":1661365180.0907464,\"logger\":\"controller.ingress\",\"msg\":\"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead\",\"reconciler group\":\"networking.k8s.io\",\"reconciler kind\":\"Ingress\",\"name\":\"thanos-query\",\"namespace\":\"monitoring\"}\n"

Above the messange now changed and we see that the controller is now being managed by Pomerium. There are no not marked to be managed by this controller entries after the upgrade (up to the rollback)

Note: I will attempt to do an upgrade on a testing cluster to see if the Pomerium ingress class get’s marked as a default ingress as you explained on your post

ulfox · August 26, 2022, 2:07pm

Additional logs

2022-08-24 21:23:56
{"log":"{"level":"info","ts":1661365436.7898235,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T18:23:56.789927475Z"}
        
2022-08-24 21:19:40
{"log":"{"level":"info","ts":1661365180.4547675,"logger":"controller.ingress","msg":"deleted from pomerium","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring","reason":"not marked to be managed by this controller"}n","stream":"stderr","time":"2022-08-24T18:19:40.455286222Z"}
        
2022-08-24 21:19:40     
{"log":"{"level":"info","ts":1661365180.0907464,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T18:19:40.090809901Z"}
        
2022-08-24 21:19:38     
{"log":"{"level":"info","ts":1661365178.7319083,"msg":"watch","deps":[{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"loki-scalable-loki-simple-scalable-gateway"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"pomerium","Name":"app-private-staging-aws-metrika-co"},{"Namespace":"pomerium","Name":"timescale-staging-aws-metrika-co"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"thanos-query"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-prometheus-0"},{"Namespace":"pomerium","Name":"app-staging-aws-metrika-co"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-prometheus-1"},{"Namespace":"pomerium","Name":"pomerium-authenticate"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"-removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"ray","Name":"ray-cluster-main-dashboard"},{"Namespace":"airflow","Name":"airflow-web"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager-0"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager-1"}],"ingressClass":"pomerium"}n","stream":"stderr","time":"2022-08-24T18:19:38.732093631Z"}
        
2022-08-24 20:18:05
{"log":"{"level":"info","ts":1661361485.0692291,"msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","ingress":{"name":"thanos-query","namespace":"monitoring"},"namespace":"monitoring","name":"thanos-query","reconcileID":"668f3b65-a4d8-4ea9-bdd8-d68de27a5cc8"}n","stream":"stderr","time":"2022-08-24T17:18:05.06935442Z"}
        
2022-08-24 20:06:02
{"log":"{"level":"info","ts":1661360762.2934139,"logger":"controller.ingress","msg":"deleted from pomerium","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring","reason":"not marked to be managed by this controller"}n","stream":"stderr","time":"2022-08-24T17:06:02.293559843Z"}
        
2022-08-24 20:06:02
{"log":"{"level":"info","ts":1661360762.0573654,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T17:06:02.057491797Z"}

So after 20:00 and before 21:19 we had the incident

ulfox · August 31, 2022, 9:30pm

This has been resolved, thanks a lot @denis

For other users, to find additional information about the resolution, please check this Pomerium slack thread

TL;DR resolution, use the following tag for the ingressController image ingressController.image.tag="sha-5294279"

Topic		Replies	Views
Upstream connect error with Pomerium Ingress Controller Support k8s	2	368	December 21, 2021
Kubernetes - Ingress - Multiple IDP? Support idp , k8s	3	999	September 23, 2022
[security] Pomerium v0.22.2, v0.21.4, v0.20.1, v0.19.2, v0.18.1, and v0.17.4 release Security	2	430	May 26, 2023
Enabling whitelisting to the Public facing Ingress Support idp , k8s	2	177	February 15, 2024
Kubernetes Deployment Across Clusters with mTLS Support k8s	2	347	December 6, 2021