# Massive RAM usage while using K8s CRD **URL:** \ **Category:** Support\ **Tags:** k8s\ **Created:** [January 20, 2023, 9:11am UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229 "2023-01-20T09:11:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![raicio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/raicio/32/58_2.png) [@raicio](https://discuss.pomerium.com/u/raicio)\ **Post date:** [January 20, 2023, 9:11am UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/1 "2023-01-20T09:11:29Z")
Hi everyone! ## What happened? We have switched to pomerium all-in-one with kubernetes CRD for both HTTPS and TCP rules. Since then, RAM usage has skyrocketed and pods take a long time to come online. Persistence is configured with postgres. If the TCP rules are removed from the CRD, RAM usage is fine and startup times are also fine. ## What’s your environment like? - Pomerium version (retrieve with `pomerium --version`): - Server Operating System/Architecture/Cloud: ## What’s your config.yaml? using CRD ## What did you see in the logs? During runtime, this error message is printed in several occurences: > “Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option \‘envoy.type.matcher.v3.RegexMatcher.google\_re2\’ from file regex.proto. This configuration will be removed from Envoy soon. Please see [Version history — envoy 1.37.0-dev-5742b1 documentation](https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history) for details. If continued use of this field is absolutely necessary, see [Runtime — envoy 1.37.0-dev-5742b1 documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features) for how to apply a temporary and highly discouraged override.” no other errors are printed. ## Additional context Please find attached the monitoring screenshots. ![pomerium CPU](https://canada1.discourse-cdn.com/flex031/uploads/pomerium/original/1X/592e28bd8df00466acf833ce17170ee52105be04.png) ![pomerium net](https://canada1.discourse-cdn.com/flex031/uploads/pomerium/original/1X/eb6ed66192a22537eed2d83031520cd08a26d765.png) ![pomerium ram](https://canada1.discourse-cdn.com/flex031/uploads/pomerium/original/1X/4498b1d3f7a0adabcfabd6abd89767a5ecc81e2c.png) ---
**Author:** ![raicio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/raicio/32/58_2.png) [@raicio](https://discuss.pomerium.com/u/raicio)\ **Post date:** [January 20, 2023, 10:00am UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/2 "2023-01-20T10:00:30Z")
Here is the log after a pod crash: > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 10 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 9 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 8 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 7 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 6 > > ```auto > {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error receiving sync record: rpc error: code = Unavailable desc = error reading from server: EOF","time":"2023-01-20T09:52:00Z","message":"sync"} > > ``` > > 5 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 4 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 3 > > ```auto > {"level":"error","error":"rpc error: code = Canceled desc = context canceled","time":"2023-01-20T09:52:00Z","message":"access log stream error, disconnecting"} > > ``` > > 2 > > ```auto > {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:36961: connect: connection refused\"","time":"2023-01-20T09:52:01Z","message":"sync"} > > ``` > > 1 > > ```auto > {"level":"fatal","pid":19,"time":"2023-01-20T09:52:01Z","message":"envoy: subprocess exited"} > > ``` This happens sometimes on startup and sometimes when adding or editing a rule via CRD. ---
**Author:** ![denis](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@denis](https://discuss.pomerium.com/u/denis)\ **Post date:** [January 20, 2023, 3:27pm UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/3 "2023-01-20T15:27:45Z")
Hi, Could you please elaborate which release did you upgraded from and to? Could you please give us some estimate wrt amount of your Ingress resources and how many of them use TCP? I assume you mean Ingress objects annotated with `tcp_upstream: true` [Ingress Configuration | Pomerium](https://www.pomerium.com/docs/deploying/k8s/ingress#tcp-services) ? ---
**Author:** ![denis](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@denis](https://discuss.pomerium.com/u/denis)\ **Post date:** [January 20, 2023, 3:44pm UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/4 "2023-01-20T15:44:56Z")
> [@raicio](#): > > This happens sometimes on startup and sometimes when adding or editing a rule via CRD. 1. this is not a crash but rather various Pomerium internal services winding down (note the `context cancelled`). there must be some other errors earlier in the log that actually caused the issue. 2. When you say `CRD` do you mean [`Ingress`](https://www.pomerium.com/docs/deploying/k8s/ingress) resource or [`Pomerium`](https://www.pomerium.com/docs/deploying/k8s/reference) global settings? I’m a bit confused here. There’s a lot happening here, would you join our [Slack channel](https://slack.pomerium.io/) to try troubleshoot it synchronously? ---
**Author:** ![raicio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/raicio/32/58_2.png) [@raicio](https://discuss.pomerium.com/u/raicio)\ **Post date:** [January 20, 2023, 3:58pm UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/5 "2023-01-20T15:58:28Z")
Hi @denis , we were running 0.19.1 via helm chart, and swiched to latest version via all in one deployment. Sure thing, i already am in the slack channel. Thank you ---
**Author:** ![raicio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/raicio/32/58_2.png) [@raicio](https://discuss.pomerium.com/u/raicio)\ **Post date:** [January 20, 2023, 3:59pm UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/6 "2023-01-20T15:59:52Z")
I confirm that we are using ingress TCP resources: 447 ingresses, of which 341 of them are TCP. thank you ---
**Author:** ![raicio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discuss.pomerium.com/raicio/32/58_2.png) [@raicio](https://discuss.pomerium.com/u/raicio)\ **Post date:** [January 20, 2023, 4:11pm UTC](https://discuss.pomerium.com/t/massive-ram-usage-while-using-k8s-crd/229/7 "2023-01-20T16:11:28Z")
Here is the config: pomerium.yaml > apiVersion: [ingress.pomerium.io/v1](http://ingress.pomerium.io/v1) > kind: Pomerium > metadata: > name: global > spec: > secrets: pomerium/bootstrap > authenticate: > url: [https://authenticate](https://authenticate)._redacted_ > identityProvider: > provider: oidc > url: [https://keycloak](https://keycloak)._redacted_ > secret: pomerium/idp > certificates: > - pomerium/pomerium-wildcard-tls > storage: > postgres: > secret: pomerium/dbsecret > jwtClaimHeaders: > additionalProperties: email, groups, user, preferred\_username example tcp ingress > ## apiVersion: v1 kind: Service metadata: name: _redacted_-ssh-service-tcp spec: type: ExternalName externalName: ‘_redacted_’ ports: - protocol: TCP name: ssh port: 22 > > apiVersion: [networking.k8s.io/v1](http://networking.k8s.io/v1) > kind: Ingress > metadata: > name: jente-demo-ssh-ssh-ingress-tcp > namespace: pomerium > annotations: > [ingress.pomerium.io/tcp\_upstream:](http://ingress.pomerium.io/tcp_upstream:) ‘true’ > [ingress.pomerium.io/allowed\_idp\_claims:](http://ingress.pomerium.io/allowed_idp_claims:) | > groups: > - _redacted_ > preferred\_username: > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > - _redacted_ > spec: > ingressClassName: pomerium > tls: > - hosts: > - _redacted_ > secretName: pomerium-wildcard-tls > rules: > - host: _redacted_ > http: > paths: > - pathType: ImplementationSpecific > backend: > service: > name: _redacted_ > port: > name: ssh