503 "cluster unavailable" when using tls_custom_ca_file

Support
1

What happened?

I have this route set:

  - from: https://service.publicdomain.com
    to: https://service.internaldomain.com
    host_rewrite: service.publicdomain.com
    tls_skip_verify: true  # TODO use CA certificate
    policy:
      #... our policy

It works fine, the internal service.internaldomain.com is signed with our internal CA, that I want to add as a custom CA to remove the tls_skip_verify, so I follow the cert chain all the way to the root, and get a root ca.

  - from: https://service.publicdomain.com
    to: https://service.internaldomain.com
    host_rewrite: service.publicdomain.com
    tls_custom_ca_file: /pomerium/root-ca.crt
    policy:
      #... our policy

And after restart I’m getting these 500: cluster unavailable errors.

What did you expect to happen?

It should validate TLS.

What’s your environment like?

  • Pomerium version 28.0
  • Linux / Docker Swarm
  • Internal CA is Windows

What did you see in the logs?

{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"envoy","upstream-cluster":"","method":"GET","authority":"service.numat-test.com","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0","referer":"","forwarded-for":"50.171.165.98,10.0.0.2","request-id":"61099152-b6b6-45d0-b933-d60892b1163a","duration":7.487834,"size":1118,"response-code":503,"response-code-details":"cluster_not_found","time":"2025-02-26T09:56:01-06:00","message":"http-request"}

Additional context

Is it a problem with the CA cert encoding? Should I have added the lower-level signing cert instead of the root CA cert?

2

Hi,

Are there any other errors in the logs? Perhaps you could enable debug logging to see?

The custom CA file should contain one or more pem-encoded certificates.

3

Thanks @calebdoxsey

I enabled debug mode and sieved through the logs, but couldn’t find anything interesting?
There is a: [Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] unknown cluster \\'route-308d22196e8f72fd\\'

Maybe it has to do with this?

{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"router","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] unknown cluster \\'route-308d22196e8f72fd\\'"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] Preparing local reply with details cluster_not_found"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] Executing sending local reply."}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"lua","time":"2025-02-27T17:24:06-06:00","message":"coroutine finished"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"lua","time":"2025-02-27T17:24:06-06:00","message":"coroutine finished"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"lua","time":"2025-02-27T17:24:06-06:00","message":"coroutine finished"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"lua","time":"2025-02-27T17:24:06-06:00","message":"coroutine finished"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"lua","time":"2025-02-27T17:24:06-06:00","message":"coroutine finished"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] encoding headers via codec (end_stream=false):\\n\\':status\\', \\'503\\'\\n\\'x-frame-options\\', \\'SAMEORIGIN\\'\\n\\'x-xss-protection\\', \\'1; mode=block\\'\\n\\'content-length\\', \\'1118\\'\\n\\'content-type\\', \\'text/html; charset=UTF-8\\'\\n\\'date\\', \\'Thu, 27 Feb 2025 23:24:06 GMT\\'\\n\\'server\\', \\'envoy\\'\\n\\'x-request-id\\', \\'f65c9f7e-a4b1-4deb-97bb-07cc89cb7356\\'\\n"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\",\\\"StreamId\\\":\\\"9489178197335497607\\\"] Codec completed encoding stream."}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http2","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\"] stream 3 closed: 0"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http2","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"25\\\"] Recouping 0 bytes of flow control window for stream 3."}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http2","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"18\\\"] Http2Visitor::OnEndStream(5)"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http2","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"18\\\"] stream 5 closed: 0"}
{"level":"debug","config_file_source":"/pomerium/config.yaml","bootstrap":true,"service":"envoy","name":"http2","time":"2025-02-27T17:24:06-06:00","message":"[Tags: \\\"ConnectionId\\\":\\\"18\\\"] Recouping 0 bytes of flow control window for stream 5."}
{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"envoy","upstream-cluster":"","method":"GET","authority":"service.publicdomain.com","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0","referer":"","forwarded-for":"2601:243:d01:8890::38ed,10.0.0.2","request-id":"0d36feee-3ab8-4917-ac68-7542c0e08acc","duration":18.117944,"size":1118,"response-code":503,"response-code-details":"cluster_not_found","time":"2025-02-27T17:24:06-06:00","message":"http-request"}
{"level":"info","config_file_source":"/pomerium/config.yaml","bootstrap":true,"server_name":"all","service":"envoy","upstream-cluster":"","method":"GET","authority":"service.publicdomain.com","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0","referer":"","forwarded-for":"2601:243:d01:8890::38ed,10.0.0.2","request-id":"f65c9f7e-a4b1-4deb-97bb-07cc89cb7356","duration":7.308622,"size":1118,"response-code":503,"response-code-details":"cluster_not_found","time":"2025-02-27T17:24:06-06:00","message":"http-request"}
4

My certificate was in x509 format, I re-encoded it on base64 PEM, but it didn’t do much, I still got the same error.