# Kubernetes | Ingress Controller | v0.18.0 | Ingress controller is processing non-pomerium ingresses

**URL:** <https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148>\
**Category:** Support\
**Tags:** k8s\
**Created:** [August 24, 2022, 8:48pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148 "2022-08-24T20:48:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ulfox](https://avatars.discourse-cdn.com/v4/letter/u/ea5d25/32.png) [@ulfox](https://discuss.pomerium.com/u/ulfox)\
**Post date:** [August 24, 2022, 8:48pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148/1 "2022-08-24T20:48:14Z")

</div>

## What happened?

Hi team,

In the [ingress documentation](https://www.pomerium.com/docs/k8s/ingress), there is this warning block

Only one Ingress Controller instance is supported per Pomerium cluster.  
`Does this mean, 1 Pomerium ingress controller or 1 ingress controller in the whole cluster?`

The reason I am asking is because after we upgraded Pomerium to latest `helm chart v32.0.5` (`Pomerium v0.18.0`), we noticed that all our `non-pomerium ALB` (e.g. prometheus, thanos) ingresses in the cluster were processed by the Pomerium ingress controller (which broke the connectivity)

Is there a way to configure the ingress controller to only process ingress objects that have explicitly set the Pomerium as an ingress class?

## What did you expect to happen?

Pomerium processing only ingresses with a Pomerium ingressClassName or annotation

## How’d it happen?

1. Upgraded to Pomerium v0.18.0, helm chart 32.0.5

## What’s your environment like?

- Pomerium version (retrieve with `pomerium --version`): v0.18.0
- Server Operating System/Architecture/Cloud: EKS v1.21.0

## What’s your config.yaml?

```auto
authenticate:
  existingTLSSecret: pomerium-tls
  idp:
    clientID: ...
    clientSecret: ...
    provider: google
    serviceAccount: ...
  ingress:
    annotations:
      cert-manager.io/issuer: letsencrypt-prod
    tls:
      secretName: authenticate.prod.aws.metrika.co-tls
  proxied: false
authorize:
  existingTLSSecret: pomerium-tls
config:
  cookieSecret: ...
  existingCASecret: pomerium-tls
  generateSigningKey: false
  generateTLS: false
  rootDomain: prod.aws.metrika.co
  sharedSecret: ...
  signingKey: ...
databroker:
  existingTLSSecret: pomerium-tls
  storage:
    clientTLS:
      existingCASecretKey: ca.crt
      existingSecretName: pomerium-tls
    connectionString: rediss://pomerium-redis-master.pomerium.svc.cluster.local
    type: redis
ingress:
  enabled: false
ingressController:
  enabled: true
proxy:
  existingTLSSecret: pomerium-tls
redis:
  auth:
    enabled: false
  enabled: true
  generateTLS: false
  tls:
    certificateSecret: pomerium-redis-tls
  usePassword: false

```

---

<div class="post-metadata">

**Author:** ![denis](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@denis](https://discuss.pomerium.com/u/denis)\
**Post date:** [August 25, 2022, 9:58pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148/2 "2022-08-25T21:58:14Z")

</div>

> [@ulfox](#):
>
> Only one Ingress Controller instance is supported per Pomerium cluster.  
> `Does this mean, 1 Pomerium ingress controller or 1 ingress controller in the whole cluster?`

The warning means that when you have a Pomerium installation, it may only be controlled by a single ingress controller.

Pomerium is expected to coexist with other ingress controllers, as it should only process ingresses that belong to the `pomerium` class.

You should have the following automatically created `IngressClass`

```auto
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: pomerium
spec:
  controller: pomerium.io/ingress-controller

```

The only situation when Pomerium ingress controller would process other Ingresses is when the following two conditions are met:

- the `Ingress` object does not refer to any `ingressClass` in the spec,
- the Pomerium `IngressClass` is marked as a default controller, via `ingressclass.kubernetes.io/is-default-class: "true"` annotation

When Pomerium processes the Ingresses for reconciliation, for all non-managed ingresses, it should emit the following warning, that you should be able to observe via logs of the pomerium-ingress-controller pod.

```auto
{
  "level": "info",
  "ts": 1661463838.4125936,
  "logger": "controller.ingress",
  "msg": "deleted from pomerium",
  "reconciler group": "networking.k8s.io",
  "reconciler kind": "Ingress",
  "name": "httpbin",
  "namespace": "default",
  "reason": "not marked to be managed by this controller"
}

```

I tried to reproduce your situation by installing both NGINX and Pomerium ingress controllers, and I do not seem to be able to reproduce your issue, when different ingresses refer to different `ingressClass`.

---

<div class="post-metadata">

**Author:** ![ulfox](https://avatars.discourse-cdn.com/v4/letter/u/ea5d25/32.png) [@ulfox](https://discuss.pomerium.com/u/ulfox)\
**Post date:** [August 25, 2022, 10:31pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148/3 "2022-08-25T22:31:46Z")

</div>

Hi @denis, thanks for the reply.

I am sorry for not mentioning the ingress controller name, we are using aws-load-balancer-controller

- [Github Repo](https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller)
- Chart version: **v2.3.1**

Some of the affected ingresses, are normal ALB ingresses, e.g. Prometheus & Thanos ingresses. They all have the same structure as below

```auto
NAMESPACE NAME CLASS HOSTS ...
monitoring thanos-query <none> FQDN ...

```

Example manifest structure from Thanos

```auto
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
    alb.ingress.kubernetes.io/group.name: internal-default
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/load-balancer-name: ...
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: thanos
    meta.helm.sh/release-namespace: monitoring
  name: thanos-query
  namespace: monitoring
spec:
  rules:
  - host: FQDN
    http:
      paths:
      - backend:
          service:
            name: thanos-query
            port:
              name: http
        path: /
        pathType: Prefix

```

Pomerium installation before the issue

- Values (see original post)
- Namespace: pomerium
- Helm Version: pomerium-31.2.6
- Application Version: 0.17.3

The issue appeard by doing the following steps:

- Upgraded Pomerium to → v0.18.0, helm chart 32.0.5
- Ingresses with annotation alb changed the loadbalancer from AWS Controller LB to Pomerium’s Controller LB
- External DNS updated the records and then we became aware of the issue

Pomerium Ingress controller logs for **Thanos before the upgrade**

```auto
"log":"{\"level\":\"info\",\"ts\":1661310404.1244335,\"logger\":\"controller.ingress\",\"msg\":\"deleted from pomerium\",\"reconciler group\":\"networking.k8s.io\",\"reconciler kind\":\"Ingress\",\"name\":\"thanos-query\",\"namespace\":\"monitoring\",\"reason\":\"not marked to be managed by this controller\"}\n","stream":"stderr","time":"2022-08-24T03:06:44.124920593Z"}

```

As you can see above Pomerium is logging that **this ingress is not makred to be manged by this controller**

Pomerium Ingress controller logs for **Thanos after the upgrade**

```auto
"{\"level\":\"info\",\"ts\":1661365180.0907464,\"logger\":\"controller.ingress\",\"msg\":\"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead\",\"reconciler group\":\"networking.k8s.io\",\"reconciler kind\":\"Ingress\",\"name\":\"thanos-query\",\"namespace\":\"monitoring\"}\n"

```

Above the messange now changed and we see that the controller is now being managed by Pomerium. There are no **not marked to be managed by this controller** entries after the upgrade (up to the rollback)

Note: I will attempt to do an upgrade on a testing cluster to see if the Pomerium ingress class get’s marked as a default ingress as you explained on your post

---

<div class="post-metadata">

**Author:** ![ulfox](https://avatars.discourse-cdn.com/v4/letter/u/ea5d25/32.png) [@ulfox](https://discuss.pomerium.com/u/ulfox)\
**Post date:** [August 26, 2022, 2:07pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148/4 "2022-08-26T14:07:27Z")

</div>

Additional logs

```auto
2022-08-24 21:23:56
{"log":"{"level":"info","ts":1661365436.7898235,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T18:23:56.789927475Z"}
        
2022-08-24 21:19:40
{"log":"{"level":"info","ts":1661365180.4547675,"logger":"controller.ingress","msg":"deleted from pomerium","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring","reason":"not marked to be managed by this controller"}n","stream":"stderr","time":"2022-08-24T18:19:40.455286222Z"}
        
2022-08-24 21:19:40     
{"log":"{"level":"info","ts":1661365180.0907464,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T18:19:40.090809901Z"}
        
2022-08-24 21:19:38     
{"log":"{"level":"info","ts":1661365178.7319083,"msg":"watch","deps":[{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"loki-scalable-loki-simple-scalable-gateway"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"pomerium","Name":"app-private-staging-aws-metrika-co"},{"Namespace":"pomerium","Name":"timescale-staging-aws-metrika-co"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"thanos-query"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-prometheus-0"},{"Namespace":"pomerium","Name":"app-staging-aws-metrika-co"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-prometheus-1"},{"Namespace":"pomerium","Name":"pomerium-authenticate"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"-removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"--removed--","Name":"--removed--"},{"Namespace":"ray","Name":"ray-cluster-main-dashboard"},{"Namespace":"airflow","Name":"airflow-web"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager-0"},{"Namespace":"monitoring","Name":"kube-prometheus-stack-alertmanager-1"}],"ingressClass":"pomerium"}n","stream":"stderr","time":"2022-08-24T18:19:38.732093631Z"}
        
2022-08-24 20:18:05
{"log":"{"level":"info","ts":1661361485.0692291,"msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","ingress":{"name":"thanos-query","namespace":"monitoring"},"namespace":"monitoring","name":"thanos-query","reconcileID":"668f3b65-a4d8-4ea9-bdd8-d68de27a5cc8"}n","stream":"stderr","time":"2022-08-24T17:18:05.06935442Z"}
        
2022-08-24 20:06:02
{"log":"{"level":"info","ts":1661360762.2934139,"logger":"controller.ingress","msg":"deleted from pomerium","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring","reason":"not marked to be managed by this controller"}n","stream":"stderr","time":"2022-08-24T17:06:02.293559843Z"}
        
2022-08-24 20:06:02
{"log":"{"level":"info","ts":1661360762.0573654,"logger":"controller.ingress","msg":"use of deprecated annotation kubernetes.io/ingress.class, please use spec.ingressClassName instead","reconciler group":"networking.k8s.io","reconciler kind":"Ingress","name":"thanos-query","namespace":"monitoring"}n","stream":"stderr","time":"2022-08-24T17:06:02.057491797Z"}

```

So after 20:00 and before 21:19 we had the incident

---

<div class="post-metadata">

**Author:** ![ulfox](https://avatars.discourse-cdn.com/v4/letter/u/ea5d25/32.png) [@ulfox](https://discuss.pomerium.com/u/ulfox)\
**Post date:** [August 31, 2022, 9:30pm UTC](https://discuss.pomerium.com/t/kubernetes-ingress-controller-v0-18-0-ingress-controller-is-processing-non-pomerium-ingresses/148/5 "2022-08-31T21:30:29Z")

</div>

This has been resolved, thanks a lot @denis

For other users, to find additional information about the resolution, please check this [Pomerium slack thread](https://pomerium-io.slack.com/archives/CK92MUAES/p1661373583075729)

TL;DR resolution, use the following tag for the ingressController image `ingressController.image.tag="sha-5294279"`