TCP proxy route_not_found

What happened?

./pomerium-cli tcp test.dev.redacted.com:4444 --disable-tls-verification --pomerium-url https://forwardauth.dev.redacted.com:5555 --listen :3333
2022/05/06 14:04:30 listening on [::]:3333
2022/05/06 14:04:37 connection closed
2022/05/06 14:04:37 error serving local connection: invalid http response code: 421```

curl localhost:3333 returns 421 error above.

## What did you expect to happen?
Being proxied to internal service

## What's your environment like?

- Pomerium version: latest debug docker image
- GKE
- Traefik ingress in GKE behind Google's TCP Load Balancer. 

# Traefik settings
  --entrypoints.metrics.address=:9100/tcp
  --entrypoints.pomerium.address=:5555/tcp
  --entrypoints.traefik.address=:9000/tcp
  --entrypoints.web.address=:8000/tcp
  --entrypoints.websecure.address=:8443/tcp
  --api.dashboard=true
  --ping=true
  --metrics.prometheus=true
  --metrics.prometheus.entrypoint=metrics
  --providers.kubernetescrd
  --providers.kubernetescrd.ingressClass=traefik-public
  --providers.kubernetescrd.allowCrossNamespace=true
  --providers.kubernetesingress
  --providers.kubernetesingress.allowEmptyServices=true
  --providers.kubernetesingress.ingressendpoint.publishedservice=traefik-public/traefik-public-cit1
  --providers.kubernetesingress.ingressClass=traefik-public
  --entrypoints.websecure.http.tls=true
  --accesslog=true
  --entryPoints.websecure.forwardedHeaders.insecure
  --log.level=DEBUG
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik-public
  name: pomerium-tcp
  namespace: pomerium-traefik
spec:
  entryPoints:
  - pomerium
  routes:
  - match: HostSNI(`*`)
    services:
    - name: pomerium-proxy
      namespace: pomerium-traefik
      port: 443
  tls:
    passthrough: true

What’s your config.yaml?

autocert: false
dns_lookup_family: V4_ONLY
address: :443
grpc_address: :443
certificate_authority_file: "/pomerium/ca/ca.crt"
certificates:
authenticate_service_url: https://authenticate.dev.redacted.com
authorize_service_url: https://pomerium-authorize.pomerium-traefik.svc.cluster.local
databroker_service_url: https://pomerium-databroker.pomerium-traefik.svc.cluster.local
idp_provider: okta
idp_scopes:
idp_provider_url: https://dev-REDACTED.okta.com
forward_auth_url: https://forwardauth.dev.redacted.com
idp_client_id: REDACTED
idp_client_secret: REDACTED
idp_service_account: REDACTED
databroker_storage_tls_skip_verify: false
routes:
  - from: tcp+https://test.dev.redacted.com:4444
    policy:
    - allow:
        and:
        - domain:
            is: redacted.com
    to: tcp://prometheus.monitoring.svc.cluster.local:9090
  - from: https://test.dev.onairent.live
    policy:
    - allow:
        and:
        - domain:
            is: redacted.com
    to: http://stub
  - from: https://authenticate.dev.redacted.com
    to: https://pomerium-authenticate.pomerium-traefik.svc.cluster.local
    allow_public_unauthenticated_access: true
  • pomerium-proxy ENV:
      INSECURE_SERVER:       false
      HTTP_REDIRECT_ADDR:    :80
      ADDRESS:               :443
      CERTIFICATE_FILE:      /pomerium/tls/tls.crt
      CERTIFICATE_KEY_FILE:  /pomerium/tls/tls.key
      SERVICES:              proxy
      LOG_LEVEL:             debug
      POMERIUM_DEBUG:        true
      PROXY_LOG_LEVEL:       debug

What did you see in the logs?

pomerium-proxy log

11:20AM DBG tls inspector: new connection accepted name=filter service=envoy

11:20AM DBG tls:onServerName(), requestedServerName: forwardauth.dev.redacted.com name=filter service=envoy

11:20AM DBG [C124537] new connection from 10.2.144.4:44000 name=conn_handler service=envoy

11:20AM DBG [C124537] new stream name=http service=envoy

11:20AM DBG [C124537][S3361548664698177273] request headers complete (end_stream=false):\n\':authority\', \'test.dev.redacted.com:4444\'\n\':method\', \'CONNECT\'\n\'user-agent\', \'Go-http-client/1.1\' name=http service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG [C124537][S3361548664698177273] no cluster match for URL \'\' name=router service=envoy

11:20AM DBG [C124537][S3361548664698177273] Sending local reply with details route_not_found name=http service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG coroutine finished name=lua service=envoy

11:20AM DBG [C124537][S3361548664698177273] encoding headers via codec (end_stream=true):\n\':status\', \'421\'\n\'strict-transport-security\', \'max-age=31536000; includeSubDomains; preload\'\n\'x-frame-options\', \'SAMEORIGIN\'\n\'x-xss-protection\', \'1; mode=block\'\n\'date\', \'Fri, 06 May 2022 11:20:19 GMT\'\n\'server\', \'envoy\'\n\'connection\', \'close\' name=http service=envoy

11:20AM DBG [C124537][S3361548664698177273] doEndStream() resetting stream name=http service=envoy

11:20AM DBG [C124537][S3361548664698177273] stream reset name=http service=envoy

11:20AM DBG [C124537] closing data_to_write=261 type=2 name=connection service=envoy

11:20AM DBG [C124537] setting delayed close timer with timeout 1000 ms name=connection service=envoy

11:20AM DBG [C124537] closing data_to_write=261 type=2 name=connection service=envoy

11:20AM DBG [C124537] write flush complete name=connection service=envoy

11:20AM DBG [C124537] remote early close name=connection service=envoy

11:20AM DBG [C124537] closing socket: 0 name=connection service=envoy

11:20AM DBG [C124537] SSL shutdown: rc=0 name=connection service=envoy

11:20AM DBG [C124537] adding to cleanup list name=conn_handler service=envoy

11:20AM INF http-request authority=test.dev.redacted.com:4444 duration=1.339214 forwarded-for=10.2.144.4 method=CONNECT path= referer= request-id=11dfc317-3c69-4997-b5f8-94ce0e5221a5 response-code=421 response-code-details=route_not_found service=envoy size=0 upstream-cluster= user-agent=Go-http-client/1.1

Additional context

Hi!
I’m struggling to make pomerium tcp proxy do its great job :slight_smile:

Pomerium’s ForwardAuth part with Traefik HTTP works just fine.
The problem is only with TCP proxying.

It works normally if I switch insecure: true within Helm chart, which makes Traefik work without TLS at all.
TCP doesn’t work properly only with enabled TLS settings.

CA, Certs and Keys for Pomerium are generated with installed into K8s cert-manager.

           apiVersion: cert-manager.io/v1
            kind: ClusterIssuer
            metadata:
              name: pomerium-traefik-issuer
            spec:
              selfSigned: {}

          pomerium-ca: |-
            apiVersion: cert-manager.io/v1
            kind: Certificate
            metadata:
              name: pomerium-traefik-ca
              namespace: pomerium-traefik
            spec:
              secretName: pomerium-traefik-ca
              duration: 175200h0m0s # 20y
              renewBefore: 166440h0m0s # 19y
              isCA: true
              issuerRef:
                name: pomerium-traefik-issuer
                kind: ClusterIssuer
              commonName: pomerium-traefik-ca

          # And this one really issues signed certificates
          pomerium-issuer: |-
            apiVersion: cert-manager.io/v1
            kind: Issuer
            metadata:
              name: pomerium-traefik-issuer
              namespace: pomerium-traefik
            spec:
              ca:
                secretName: pomerium-traefik-ca

          pomerium-tls: |-
            apiVersion: cert-manager.io/v1
            kind: Certificate
            metadata:
              name: pomerium-traefik-tls
              namespace: pomerium-traefik

            spec:
              secretName: pomerium-traefik-tls
              duration: 175200h0m0s # 20y
              renewBefore: 166440h0m0s # 19y
              dnsNames:
              - "*.pomerium-traefik.svc.cluster.local"
              - "*.pomerium-traefik"
              - "*.dev.redacted.com"
              - "*.dev.int.redacted.com"

              privateKey:
                algorithm: RSA
                encoding: PKCS1
                size: 2048

              usages:
              - server auth
              - client auth

              issuerRef:
                name: pomerium-traefik-issuer
                kind: Issuer

I also tried to remove Traefik from the chain, and create Google’s TCP Load Balancer and point it directly to Pomerium Proxy, but result was the same.

Also tried to dump envoy config, and at first glance it looks properly. It contains config parts for test.dev.redacted.com:4444 like

                   {
                    "upgrade_type": "CONNECT",
                    "enabled": true,
                    "connect_config": {}
                   }

Most recent Pomerium ingress controller supports creating TCP routes via Ingress resources. Please see tcp tunnel support by wasaga · Pull Request #196 · pomerium/ingress-controller · GitHub

Is there a particular reason you have to run Traefik in front of Pomerium, and not let Pomerium Ingress Controller manage Ingresses that require authentication and proxy their traffic directly?