Pomerium loops re-reading configuration from CloudRun mounted Secret

Support
1

Dear team,

I’m observing Pomerium (using latest image from pomerium/pomerium:latest, should be v0.25.2) is looping in the fileutil/watcher permanently rebuilding/re-reading its config.yaml:

{
  "jsonPayload": {
    "level": "info",
    "message": "fileutil/watcher: file notification event",
    "op": "WRITE",
    "name": "/pomerium/config.yaml",
    "config_file_source": "/pomerium/config.yaml"
  }
}

I’ve deployed it as GCP CloudRun and mounted config as a file, provided from Secret Manager using the following config:

        volumeMounts:
        - mountPath: /pomerium
          name: pomerium
      volumes:
      - name: pomerium
        secret:
          items:
          - key: '2'
            path: config.yaml
          secretName: pomerium-config

I’ve examized how GCP mounts secret files and found out that the timestamp of the file is always fresh:

root@localhost:/pomerium$ ls -l
total 2
-r--r--r--    1 root     root          1475 Apr 19 21:12 config.yaml

It looks like used by Pomerium config file change notification mechanism (inode polling?) does not work correctly for this configuration. Pomerium is tricked to think the config.yaml is changing all the time.

I searched the code to find the option to disable the config watcher, but I didn’t find anything.

The constant configuration rebuilds does not looks correct, pollute logs and overloads the CPU.

I think I’ll work around by supplying config included in the docker image, but it would be generally cool to have an option to disable config file watching, in case it works incorrectly.

Would you mind to add such option? Or did I oversee it? Maybe over suggestions on how to mount secrets in CloudRun environment elegantly avoiding false config change notifications?

Kind regards,
Anton

2

Hi @0anton, thanks for reporting this issue. I’ve copied it over into a GitHub issue here: Pomerium loops re-reading configuration from CloudRun mounted Secret · Issue #5079 · pomerium/pomerium · GitHub.

We don’t currently have an option to disable config file watching, but it seems reasonable to add one for cases such as this.

Meanwhile, building a config file into the Docker image itself sounds like a good workaround.

Thanks,
Ken

1 Like