SSE (eventstream) not working through pomerium

[dwatrous]

Pomerium Forum Post: Server-Sent Events Not Streaming in Real-Time Through Ingress

What happened?

Backstage scaffolder task logs do not stream in real-time through Pomerium ingress. When users run a template/scaffolder task, the browser makes a request to /api/scaffolder/v2/tasks/{taskId}/eventstream using Server-Sent Events (SSE), but the logs only appear after the entire task completes, rather than streaming incrementally as each step executes.

What did you expect to happen?

Expected the event stream to deliver log events incrementally in real-time as the scaffolder task executes (similar to watching kubectl logs -f). The logs should appear in the browser UI as each step completes, not all at once at the end.

How’d it happen?

1. User navigates to Backstage at https://portal.idp.apps.com
2. User triggers a scaffolder template execution
3. Browser opens EventSource connection to /api/scaffolder/v2/tasks/{taskId}/eventstream
4. Logs do not appear in real-time - all logs appear only after the task completes
5. Verified that streaming works perfectly when accessed directly from within the cluster using a curl script that bypasses Pomerium

What’s your environment like?

• Pomerium version: Pomerium Ingress Controller on Kubernetes (latest from helm chart)
• Server Operating System/Architecture/Cloud: GKE (Google Kubernetes Engine) on Linux/amd64
• Backstage version: Latest (uses Server-Sent Events for task logs)
• Client: Chrome browser accessing through Pomerium proxy

What’s your config.yaml?

Pomerium Global CRD:

apiVersion: ingress.pomerium.io/v1
kind: Pomerium
metadata:
  name: global
  namespace: pomerium
spec:
  authenticate:
    url: https://authenticate.idp.apps.com
  certificates:
    - pomerium/apps-tls-secret
  identityProvider:
    provider: google
    secret: pomerium/pomerium-bootstrap-v2
  secrets: pomerium/pomerium-bootstrap-v2
  codecType: http2
  timeouts:
    idle: "0s"
    read: "3600s"
    write: "7200s"

Backstage Ingress Annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backstage
  namespace: backstage
  annotations:
    ingress.pomerium.io/allowed_domains: '["company.com"]'
    ingress.pomerium.io/pass_identity_headers: "true"
    ingress.pomerium.io/idle_timeout: "0s"
    ingress.pomerium.io/timeout: "0s"
    ingress.pomerium.io/allow_websockets: "true"
    ingress.pomerium.io/preserve_host_header: "false"
    ingress.pomerium.io/allow_spdy: "true"
    ingress.pomerium.io/set_response_headers: '{"X-Accel-Buffering": "no", "Cache-Control": "no-cache, no-store, must-revalidate", "X-Content-Type-Options": "nosniff"}'
spec:
  ingressClassName: pomerium
  rules:
    - host: portal.idp.apps.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backstage
                port:
                  number: 80

Backstage Configuration (app-config.production.yaml):

scaffolder:
  useLongPollingLogs: false  # Use SSE for streaming
  heartbeat:
    frequency: { seconds: 5 }
    timeout: { seconds: 30 }
  allowAnonymousAccess: true  # EventSource can't send Authorization headers

backend:
  server:
    keepAliveTimeout: { seconds: 650 }
    headersTimeout: { seconds: 660 }
  cors:
    origin: https://portal.idp.apps.com
    methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
    credentials: true

What did you see in the logs?

Initial attempt with invalid annotations resulted in reconciliation errors:

{"level":"error","msg":"Reconciler error","error":"parsing ingress: annotations: unknown ingress.pomerium.io/compression_enabled"}
{"level":"error","msg":"Reconciler error","error":"parsing ingress: annotations: unknown ingress.pomerium.io/streaming_enabled"}

After setting global timeouts with read=0s, write=0s (both zero):

{"level":"error","msg":"Reconciler error","error":"set config: settings: timeouts: read timeout (0s) must be less than write timeout (0s)"}

After fixing timeouts (read=3600s, write=7200s), Pomerium reconciles successfully:

{"level":"info","msg":"config updated","controller":"bootstrap pod/pomerium-xxx"}

Testing stream directly from within cluster (bypassing Pomerium) works perfectly:

kubectl run debug-stream --rm -i --restart=Never --image=curlimages/curl -- \
  curl -N http://backstage.backstage.svc.cluster.local/api/scaffolder/v2/tasks/{taskId}/eventstream

This shows real-time streaming output as each step executes, proving the Backstage backend is correctly implementing Server-Sent Events.

Pomerium Status shows successful reconciliation:

status:
  ingress:
    backstage/backstage:
      observedAt: "2026-04-02T22:18:32Z"
      observedGeneration: 1
      reconciled: true
  settingsStatus:
    observedAt: "2026-04-02T22:28:37Z"
    observedGeneration: 3
    reconciled: true

Additional context

What we’ve tried:

1. Set idle_timeout: "0s" and timeout: "0s" in ingress annotations
2. Added global timeouts to Pomerium CRD with idle: "0s", read: "3600s", write: "7200s"
3. Added X-Accel-Buffering: no response header
4. Enabled allow_websockets: true and allow_spdy: true
5. Set Cache-Control: no-cache, no-store, must-revalidate headers
6. Verified HTTP/2 is working (curl shows HTTP/2 302)
7. Forced codecType: http2 in Pomerium global config
8. Confirmed Backstage is using Server-Sent Events (useLongPollingLogs: false)
9. Verified all Pomerium configurations reconciled successfully

Key observations:

• Streaming works perfectly when accessed directly from within the cluster (bypassing Pomerium)
• Streaming does not work when accessed through Pomerium from the browser
• No errors in logs - responses eventually arrive, but all at once at the end rather than incrementally
• This suggests Pomerium/Envoy is buffering the response body despite our timeout and header configurations

Question:
Is there a way to disable response buffering in Pomerium/Envoy for Server-Sent Events endpoints? Or is there a specific configuration needed to support incremental response streaming for long-running HTTP requests?

We’ve exhausted the standard timeout and buffering header approaches. Any guidance on enabling true streaming through Pomerium would be greatly appreciated.

[dwatrous]

This wasn’t pomerium related. It was a security scanning intermediary buffering until the stream closed.