Serving with the wrong certificate

What happened?

A handful of our routes got net::ERR_CERT_COMMON_NAME_INVALID errors from the browsers (not all of them).
If I click on the security details on the browser, the wrong certificate is being presented. The certificate has a DN for the authenticate url.

  • our real domain is obscured by mydomain.com

What did you expect to happen?

That the internal.mydomain.com gets presented with the proper internal.mydomain.com certificate

What’s your environment like?

  • Pomerium Version v0.27.1
  • Server Operating System/Architecture/Cloud: Ubuntu / Docker Swarm

What’s your config.yaml?

# Main configuration flags : https://www.pomerium.com/docs/reference/
authenticate_service_url: https://authenticate.mydomain.com
# authorize service url will default to localhost in all-in-one mode, otherwise

# Certificates can be loaded as files or base64 encoded bytes.
# Autocert automatically generates subdomain certificates using LetsEncrypt
autocert: true

cookie_expire: "168h"  # 1 week, https://github.com/golang/go/issues/11473

jwt_claims_headers:
- name
- email
# ...

routes:
  - from: https://verify.mydomain.com
    to: http://pomerium_verify:8000
    pass_identity_headers: true
    policy: &allow-domain
      allow:
        and:
        - domain:
            is: mydomain.com

  # Internal Ngnix web server
  - from: https://internal.mydomain.com
    to: http://internal_internal  # The docker service 
    pass_identity_headers: true
    policy:
      <<: *allow-domain

  - from: https://launchpad.mydomain.com  # This service works fine - gets served the right certificate
    to: http://launchpad_launchpad:3000
    pass_identity_headers: true
    policy:
      <<: *allow-domain

### ... other routes

Our docker file is:

services:
  pomerium:
    image: localhost:5000/numat/pomerium
    build:
      context: .
      dockerfile: Dockerfile
    deploy:
      mode: replicated
      replicas: 1
    volumes:
      - /media/pomerium/data:/data:rw
    ports:
      - 80:80
      - 443:443
    networks:
      - swarm
    environment:
      - TZ=America/Chicago
      - SHARED_SECRET_FILE=...
      - COOKIE_SECRET_FILE=...
      - SIGNING_KEY_FILE=...
      - DATA_BROKER_STORE_TYPE=postgres
      - DATA_BROKER_STORE_CONNECTION_STRING=...
      - IDP_PROVIDER=azure
      - IDP_CLIENT_SECRET_FILE=...
      - IDP_CLIENT_ID=...
      - IDP_PROVIDER_URL=https://login.microsoftonline.com/...
    secrets:
      - POMERIUM_SHARED_SECRET
      - POMERIUM_COOKIE_SECRET
      - POMERIUM_IDP_CLIENT_SECRET
      - POMERIUM_SIGNING_KEY

What did you see in the logs?

Might not be related, but I’m seeing these:

	
{"level":"error","time":"2024-10-04T10:47:18-05:00","logger":"renew","msg":"will retry","service":"autocert","error":"[internal.mydomain.com] Renew: [internal.mydomain.com] solving challenge: internal.numat.com: [internal.mydomain.com] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":8,"retrying_in":1200,"elapsed":2434.484458426,"max_duration":2592000}

Additional context

We got autocert enabled. It has worked fine until now. The certificates are still valid, I just don’t understand why would it serve the wrong one.

Hi Julian,

Do you have a proxy in front of pomerium? Or is it accessed directly?

I ask because autocert uses a special protocol on top of TLS to provision the certificate via ALPN. A proxy (like Cloudflare) could break this.

We can also support an HTTP challenge, but only if the http redirect address is set. In previous versions of pomerium this would end up always enabled, which was a bug. You could try adding the redirect address: HTTP Redirect Address | Pomerium, and then maybe autocert could use the HTTP challenge to provision the certificate.

I’m not sure why you’d be seeing the authenticate certificate when accessing internal. I would expect it to serve a self-signed certificate when none of the certificates match.

1 Like

Thanks @calebdoxsey , Yes it seems we’re behind a Cloudflare proxy. From outside our company we get content served with a *.mydomain.com wildcard certificate, but when we’re onsite we get Pomerium-requested Letsencript certificates.
I was digging to the autocert folders, and indeed the affected services have expired certificates:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:3b:51:ec:73:58:e2:a7:96:64:6c:42:3b:a4:6b:f5:08:c0
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, O = Let's Encrypt, CN = E5
        Validity
            Not Before: Jul  6 03:52:13 2024 GMT
            Not After : Oct  4 03:52:12 2024 GMT
        Subject: CN = internal.mydomain.com

What you say:

In previous versions of pomerium this would end up always enabled, which was a bug

Is consistent with what I’m seeing, we had an old version of Pomerium (v0.19) and we upgraded by the end of August, which is consistent with the issue date of the latest valid certificates. From then I think we stopped renewing any certificates at all.

I’ll try to enable HTTP Redirect Address to use the HTTP challenge.

Edit Worked like a charm, Thanks @calebdoxsey

I added the environment to my docker yml file:

pomerium:
    # ...
    environment:
      - HTTP_REDIRECT_ADDR=":80"  # Needed for Autocert to use HTTP Challenge
      - TZ=America/Chicago
      # ...
1 Like