What happened?
./pomerium-cli tcp test.dev.redacted.com:4444 --disable-tls-verification --pomerium-url https://forwardauth.dev.redacted.com:5555 --listen :3333
2022/05/06 14:04:30 listening on [::]:3333
2022/05/06 14:04:37 connection closed
2022/05/06 14:04:37 error serving local connection: invalid http response code: 421```
curl localhost:3333 returns 421 error above.
## What did you expect to happen?
Being proxied to internal service
## What's your environment like?
- Pomerium version: latest debug docker image
- GKE
- Traefik ingress in GKE behind Google's TCP Load Balancer.
# Traefik settings
--entrypoints.metrics.address=:9100/tcp
--entrypoints.pomerium.address=:5555/tcp
--entrypoints.traefik.address=:9000/tcp
--entrypoints.web.address=:8000/tcp
--entrypoints.websecure.address=:8443/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetescrd.ingressClass=traefik-public
--providers.kubernetescrd.allowCrossNamespace=true
--providers.kubernetesingress
--providers.kubernetesingress.allowEmptyServices=true
--providers.kubernetesingress.ingressendpoint.publishedservice=traefik-public/traefik-public-cit1
--providers.kubernetesingress.ingressClass=traefik-public
--entrypoints.websecure.http.tls=true
--accesslog=true
--entryPoints.websecure.forwardedHeaders.insecure
--log.level=DEBUG
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
annotations:
kubernetes.io/ingress.class: traefik-public
name: pomerium-tcp
namespace: pomerium-traefik
spec:
entryPoints:
- pomerium
routes:
- match: HostSNI(`*`)
services:
- name: pomerium-proxy
namespace: pomerium-traefik
port: 443
tls:
passthrough: true
What’s your config.yaml?
autocert: false
dns_lookup_family: V4_ONLY
address: :443
grpc_address: :443
certificate_authority_file: "/pomerium/ca/ca.crt"
certificates:
authenticate_service_url: https://authenticate.dev.redacted.com
authorize_service_url: https://pomerium-authorize.pomerium-traefik.svc.cluster.local
databroker_service_url: https://pomerium-databroker.pomerium-traefik.svc.cluster.local
idp_provider: okta
idp_scopes:
idp_provider_url: https://dev-REDACTED.okta.com
forward_auth_url: https://forwardauth.dev.redacted.com
idp_client_id: REDACTED
idp_client_secret: REDACTED
idp_service_account: REDACTED
databroker_storage_tls_skip_verify: false
routes:
- from: tcp+https://test.dev.redacted.com:4444
policy:
- allow:
and:
- domain:
is: redacted.com
to: tcp://prometheus.monitoring.svc.cluster.local:9090
- from: https://test.dev.onairent.live
policy:
- allow:
and:
- domain:
is: redacted.com
to: http://stub
- from: https://authenticate.dev.redacted.com
to: https://pomerium-authenticate.pomerium-traefik.svc.cluster.local
allow_public_unauthenticated_access: true
- pomerium-proxy ENV:
INSECURE_SERVER: false
HTTP_REDIRECT_ADDR: :80
ADDRESS: :443
CERTIFICATE_FILE: /pomerium/tls/tls.crt
CERTIFICATE_KEY_FILE: /pomerium/tls/tls.key
SERVICES: proxy
LOG_LEVEL: debug
POMERIUM_DEBUG: true
PROXY_LOG_LEVEL: debug
What did you see in the logs?
pomerium-proxy log
11:20AM DBG tls inspector: new connection accepted name=filter service=envoy
11:20AM DBG tls:onServerName(), requestedServerName: forwardauth.dev.redacted.com name=filter service=envoy
11:20AM DBG [C124537] new connection from 10.2.144.4:44000 name=conn_handler service=envoy
11:20AM DBG [C124537] new stream name=http service=envoy
11:20AM DBG [C124537][S3361548664698177273] request headers complete (end_stream=false):\n\':authority\', \'test.dev.redacted.com:4444\'\n\':method\', \'CONNECT\'\n\'user-agent\', \'Go-http-client/1.1\' name=http service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG [C124537][S3361548664698177273] no cluster match for URL \'\' name=router service=envoy
11:20AM DBG [C124537][S3361548664698177273] Sending local reply with details route_not_found name=http service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG coroutine finished name=lua service=envoy
11:20AM DBG [C124537][S3361548664698177273] encoding headers via codec (end_stream=true):\n\':status\', \'421\'\n\'strict-transport-security\', \'max-age=31536000; includeSubDomains; preload\'\n\'x-frame-options\', \'SAMEORIGIN\'\n\'x-xss-protection\', \'1; mode=block\'\n\'date\', \'Fri, 06 May 2022 11:20:19 GMT\'\n\'server\', \'envoy\'\n\'connection\', \'close\' name=http service=envoy
11:20AM DBG [C124537][S3361548664698177273] doEndStream() resetting stream name=http service=envoy
11:20AM DBG [C124537][S3361548664698177273] stream reset name=http service=envoy
11:20AM DBG [C124537] closing data_to_write=261 type=2 name=connection service=envoy
11:20AM DBG [C124537] setting delayed close timer with timeout 1000 ms name=connection service=envoy
11:20AM DBG [C124537] closing data_to_write=261 type=2 name=connection service=envoy
11:20AM DBG [C124537] write flush complete name=connection service=envoy
11:20AM DBG [C124537] remote early close name=connection service=envoy
11:20AM DBG [C124537] closing socket: 0 name=connection service=envoy
11:20AM DBG [C124537] SSL shutdown: rc=0 name=connection service=envoy
11:20AM DBG [C124537] adding to cleanup list name=conn_handler service=envoy
11:20AM INF http-request authority=test.dev.redacted.com:4444 duration=1.339214 forwarded-for=10.2.144.4 method=CONNECT path= referer= request-id=11dfc317-3c69-4997-b5f8-94ce0e5221a5 response-code=421 response-code-details=route_not_found service=envoy size=0 upstream-cluster= user-agent=Go-http-client/1.1
Additional context
Hi!
I’m struggling to make pomerium tcp proxy do its great job 
Pomerium’s ForwardAuth part with Traefik HTTP works just fine.
The problem is only with TCP proxying.
It works normally if I switch insecure: true within Helm chart, which makes Traefik work without TLS at all.
TCP doesn’t work properly only with enabled TLS settings.
CA, Certs and Keys for Pomerium are generated with installed into K8s cert-manager.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: pomerium-traefik-issuer
spec:
selfSigned: {}
pomerium-ca: |-
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pomerium-traefik-ca
namespace: pomerium-traefik
spec:
secretName: pomerium-traefik-ca
duration: 175200h0m0s # 20y
renewBefore: 166440h0m0s # 19y
isCA: true
issuerRef:
name: pomerium-traefik-issuer
kind: ClusterIssuer
commonName: pomerium-traefik-ca
# And this one really issues signed certificates
pomerium-issuer: |-
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: pomerium-traefik-issuer
namespace: pomerium-traefik
spec:
ca:
secretName: pomerium-traefik-ca
pomerium-tls: |-
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pomerium-traefik-tls
namespace: pomerium-traefik
spec:
secretName: pomerium-traefik-tls
duration: 175200h0m0s # 20y
renewBefore: 166440h0m0s # 19y
dnsNames:
- "*.pomerium-traefik.svc.cluster.local"
- "*.pomerium-traefik"
- "*.dev.redacted.com"
- "*.dev.int.redacted.com"
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
usages:
- server auth
- client auth
issuerRef:
name: pomerium-traefik-issuer
kind: Issuer
I also tried to remove Traefik from the chain, and create Google’s TCP Load Balancer and point it directly to Pomerium Proxy, but result was the same.
Also tried to dump envoy config, and at first glance it looks properly. It contains config parts for test.dev.redacted.com:4444 like
{
"upgrade_type": "CONNECT",
"enabled": true,
"connect_config": {}
}
