Infinite re-directs

Support
1

What happened?

put into redirect hell

What did you expect to happen?

to see some type of no access page

How’d it happen?

created a route to limit access based on email, signed in with user with different email and was placed in an infinite loop of redirects

What’s your environment like?

  • Pomerium version (latest):
  • Server Operating System/Architecture/Cloud: ubuntu 20.04

What’s your config.yaml?


# Main configuration flags : https://www.pomerium.com/docs/reference/


pomerium_debug: true
address: :80
cookie_secret:
shared_secret:

idp_provider: "google"
idp_provider_url: https://accounts.google.com
idp_client_id: 
idp_client_secret:
idp_service_account: 
insecure_server: true
forward_auth_url: http://fwdauth.domain.com
authenticate_service_url: https://authenticate.domain.com

signing_key: 
#signing_key_algorithm ES256

jwt_claims_headers:
  X-Email: user

routes:
  - from: https://diborane.domain.com
    to: https://verify
    allowed_groups::
      - it
    pass_identity_headers: true

What did you see in the logs?

nginx_1     | IP_Address - - [18/Dec/2021:04:04:33 +0000] "GET /state HTTP/2.0" 302 145 "https://diborane.domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
nginx_1     | IP_Address - - [18/Dec/2021:04:04:33 +0000] "GET /?uri=https://diborane.domaint.com/?pomerium_expiry=1639800573&pomerium_issued=1639800273&pomerium_redirect_uri=https%3A%2F%2Fdiborane.domain.com%2F&pomerium_session_encrypted=xog2dL9qidIC63AwM7kqKXUL08XEkrJc_KBV13rTDKvizhIdEUvfiv4ns7g2ZFJ8BBdG5hggGgVcctV9hTq1dLW5mKEpxSILbHFsKu2ww59KwexKaQDCauL7XmJbcYOIYRb7LISYsiaQGO4RowvbOSpBAyWwnMMuBC3Dbeu6j-nd0IXVMy8uL-K154pdyUyZwgHnmmYFE_t5lk0w_AMKPgXu1BV589GnX7M0VbhdipwnHNBXZdkcmtm-kuGG12wCm0Rr0AZye1XVMxEoJ8fJ-B7lQKXc0lA3Sj9Bb4vLXoo23tl6vdE63ng7KO8X2nN3woqmI8uT74MuPak8-dHIxcOvXeoAJt-fP1Euh-4lS0ie3toFPrlJM2fS7W-7_ZOsJILP_tKCQtlKf_Vflt3tX-DNcpuABu1oToKYEWb5pin6H9haDSf2oOstIp4QJNE4kMnyQEgZBYr0ZkSir0If1m5W9Kt0V86AJ0cL4D4sa9lN4-_KkfCnFfe11xhqJZ5UHOlf8NtqVd3PrzvyfeMtVqoDGRajP586euNXV1Hmm35jPBXYnTI_Vk6N1_bEBpv-b3Fm3VgqWc74sw__rD_ZrW9Ods7oKWULxEr7X85kz-KBzTqJe_kXLqRj9nQEELFdRGZUfanRzL6n6yAyRmvk1C3PiLmw2xSKH8_0TcGNREhc4GMiWG-DbIfw4lUh9LvfrhWPJmAZ-vpyd_P710Zsh5XCe1Q5fLABLYBtZL0%3D&pomerium_signature=ySIXRFs9W1bBYGtVjvekCJW10VLW255rQOGuy6lDJkw%3D HTTP/2.0" 302 55 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
nginx_1     | IP_Address- - [18/Dec/2021:04:04:33 +0000] "GET /?uri=https://diborane.domain.com/state HTTP/2.0" 302 397 "https://diborane.domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=5.786483 forwarded-for=IP_Address,172.18.0.5 method=GET path=/verify referer=https://diborane.domain.com/ request-id=e5f4a1af-4833-4b43-a6ae-5be0538ba819 response-code=401 response-code-details=ext_authz_denied service=envoy size=11222 upstream-cluster= user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=1.085572 forwarded-for=IP_Address,172.18.0.5 method=GET path=/ referer=https://diborane.domain.com/ request-id=7a4535fc-a21e-4198-86bf-2dcfd2f8d1c3 response-code=302 response-code-details=via_upstream service=envoy size=397 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=6.253214 forwarded-for=IP_Address,172.18.0.5 method=GET path=/verify referer=https://internal.domain.com/ request-id=dabf311d-59e6-4097-8344-6da48be6be44 response-code=401 response-code-details=ext_authz_denied service=envoy size=11222 upstream-cluster= user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=0.90446 forwarded-for=IP_Address,172.18.0.5 method=GET path=/ referer= request-id=d3ea0361-553f-406e-9134-caacd7338ba8 response-code=302 response-code-details=via_upstream service=envoy size=55 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=0.818854 forwarded-for=IP_Address,172.18.0.5 method=GET path=/ referer=https://diborane.domain.com/ request-id=1c0c3004-6d3b-4def-a01f-6ebe827d1669 response-code=302 response-code-details=via_upstream service=envoy size=397 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=6.786949 forwarded-for=IP_Address,172.18.0.5 method=GET path=/verify referer=https://diborane.domain.com/ request-id=067fca5b-375a-450a-9444-cb22a92004a8 response-code=401 response-code-details=ext_authz_denied service=envoy size=11222 upstream-cluster= user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=0.687645 forwarded-for=IP_Address,172.18.0.5 method=GET path=/ referer=https://internal.domain.com/ request-id=c256cc63-0fdd-4405-aba4-dd8b3316b501 response-code=302 response-code-details=via_upstream service=envoy size=387 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=0.939363 forwarded-for=IP_Address,172.18.0.5 method=GET path=/ referer=https://diborane.domain.com/ request-id=596baa91-a165-4337-84eb-3a7988aa31ee response-code=302 response-code-details=via_upstream service=envoy size=397 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=authenticate.domain.com duration=3.517033 forwarded-for=IP_Address,172.18.0.5 method=GET path=/.pomerium/sign_in referer=https://internal.domain.com/ request-id=ef74ed08-d996-4272-b340-c014cf5f63a2 response-code=302 response-code-details=via_upstream service=envoy size=1028 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=1.868924 forwarded-for=IP_Address,172.18.0.5 method=GET path=/verify referer= request-id=6a6fa417-b99f-4ce7-a0ae-a177ce5baf7b response-code=401 response-code-details=via_upstream service=envoy size=10915 upstream-cluster=pomerium-control-plane-http user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF http-request authority=fwdauth.domain.com duration=6.18971 forwarded-for=IP_Address,172.18.0.5 method=GET path=/verify referer=https://diborane.domain.com/ request-id=03961100-cdc2-4df0-bdeb-dafd2599bbdd response-code=401 response-code-details=ext_authz_denied service=envoy size=11222 upstream-cluster= user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
pomerium_1  | 4:04AM INF authorize check allow=false check-request-id=226c1e73-87c5-4464-9ac7-46745f19944b databroker_record_version=7 databroker_server_version=1197497410956795692 deny=null email=e.gomez@domain.com host=diborane.domain.com method=GET path=/ query= request-id=26764e2a-f95d-4710-b633-2c2a523d707e service=authorize session-id=a31274be-c580-44fb-8fb4-2411b18cd306 user=113747714814128887702
nginx_1     | IP_Address - - [18/Dec/2021:04:04:33 +0000] "GET / HTTP/2.0" 302 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
nginx_1     | IP_Address - - [18/Dec/2021:04:04:34 +0000] "GET /?uri=https://diborane.domain.com/ HTTP/2.0" 302 387 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
pomerium_1  | 4:04AM INF authorize check allow=false check-request-id=294d9172-bfa8-4405-abaf-8bab34f19025 databroker_record_version=7 databroker_server_version=1197497410956795692 deny=null email=e.gomez@domain.com host=diborane.domian.com method=GET path=/state query= request-id=93365419-d39b-4437-9839-a4cbc9b14c8b service=authorize session-id=a31274be-c580-44fb-8fb4-2411b18cd306 user=113747714814128887702
nginx_1     | IP_address - - [18/Dec/2021:04:04:34 +0000] "GET /state HTTP/2.0" 302 145 "https://diborane.domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
nginx_1     | IP_Address - - [18/Dec/2021:04:04:34 +0000] "GET /?uri=https://diborane.domain.com/state HTTP/2.0" 302 397 "https://diborane.domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
pomerium_1  | 4:04AM INF get id=a31274be-c580-44fb-8fb4-2411b18cd306 peer=127.0.0.1:43802 type=type.googleapis.com/session.Session
nginx_1     | IP_Address - - [18/Dec/2021:04:04:34 +0000] "GET /.pomerium/sign_in?pomerium_callback_uri=https%3A%2F%2Fdiborane.domain.com%2F&pomerium_expiry=1639800574&pomerium_forward_auth=fwdauth.domain.com&pomerium_issued=1639800274&pomerium_redirect_uri=https%3A%2F%2Fdiborane.domain.com%2F&pomerium_signature=KFk8Dw-fVJ_40Jg81C-rq2ow3P8kdbHZxDumDZIpbj8%3D HTTP/2.0" 302 1028 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"


nginx_1     | 73.208.199.78 - - [18/Dec/2021:04:04:34 +0000] "GET /?pomerium_expiry=1639800574&pomerium_issued=1639800274&pomerium_redirect_uri=https%3A%2F%2Fdiborane.domain.com%2F&pomerium_session_encrypted=EpEcXJ74SrDdhxVVYaB0C8UgSioPwUUuK8uWn5fq-zrEZfOWlAGj65gXMaDKawKfI6wu9pjxc7ecbupcdR73IuXgoGC-DTNz-z_iGLE65JBjgh_C42z3EukOV-ccDv4zUX8SjGIiDECbw9q2HfXzi3a3Zf1WZlRATVVsr-TEH_3JqXg0iENm9lUW32XhX5W7hWHFS_sZVSBT-OCyNrGN0xLpNLyuHwxP8t4YosISxrw9z5uY3ozh9KgftkUAQoxaPga_YE_LSyGIaooQz-DsHX0Te35pKZXlgQvDuC2tfTshl_e2e6tmKzHuP-EOWUUrDNzKUqtdMe6L0oJLafyY6_f1zJ9rFgKASe-26iRgvaEgQSQZkYZGtG3qAo-xw3_ym_IEWS9kPOtIXOEdqoXn_fYfLGQZGPLFS8lBR81M_5R_QofH8_hYU0r-gNS4vUNVZckEHjeaY5nvZwPs8rea0_po3mcEmT2zc2tSuAjWNakuOfm8R0GfdHC5yDNYxnh3U3NDj7rnO1QHwWc6Rdr_HfI4JjaKBjGJbNssicFAeCGCdSniipJJimjkgJUfq4W1oF-AJRcX7oAOWhqh5rSXNhEKC5Zk7reueyMD1CsWZDwVRX1vqgMsmzucFuRiK9OS90rI-txZDQNxQdFZBE98nPnplFY7VSF9Xn7i3OScC6IN-Qvl3tOcYhjeJsBwf1PQos1v69-vHhvs2DE3KG4FPqt1hz2NXLO1cD6oBOM%3D&pomerium_signature=6glafqqteyKNM_73BuSWMJvp_9i7dZ8Ajnl8f3bbdr4%3D HTTP/2.0" 302 145 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"
nginx_1     | IP_Address - - [18/Dec/2021:04:04:34 +0000] "GET /?uri=https://diborane.domain.com/?pomerium_expiry=1639800574&pomerium_issued=1639800274&pomerium_redirect_uri=https%3A%2F%2Fdiborane.domain.com%2F&pomerium_session_encrypted=EpEcXJ74SrDdhxVVYaB0C8UgSioPwUUuK8uWn5fq-zrEZfOWlAGj65gXMaDKawKfI6wu9pjxc7ecbupcdR73IuXgoGC-DTNz-z_iGLE65JBjgh_C42z3EukOV-ccDv4zUX8SjGIiDECbw9q2HfXzi3a3Zf1WZlRATVVsr-TEH_3JqXg0iENm9lUW32XhX5W7hWHFS_sZVSBT-OCyNrGN0xLpNLyuHwxP8t4YosISxrw9z5uY3ozh9KgftkUAQoxaPga_YE_LSyGIaooQz-DsHX0Te35pKZXlgQvDuC2tfTshl_e2e6tmKzHuP-EOWUUrDNzKUqtdMe6L0oJLafyY6_f1zJ9rFgKASe-26iRgvaEgQSQZkYZGtG3qAo-xw3_ym_IEWS9kPOtIXOEdqoXn_fYfLGQZGPLFS8lBR81M_5R_QofH8_hYU0r-gNS4vUNVZckEHjeaY5nvZwPs8rea0_po3mcEmT2zc2tSuAjWNakuOfm8R0GfdHC5yDNYxnh3U3NDj7rnO1QHwWc6Rdr_HfI4JjaKBjGJbNssicFAeCGCdSniipJJimjkgJUfq4W1oF-AJRcX7oAOWhqh5rSXNhEKC5Zk7reueyMD1CsWZDwVRX1vqgMsmzucFuRiK9OS90rI-txZDQNxQdFZBE98nPnplFY7VSF9Xn7i3OScC6IN-Qvl3tOcYhjeJsBwf1PQos1v69-vHhvs2DE3KG4FPqt1hz2NXLO1cD6oBOM%3D&pomerium_signature=6glafqqteyKNM_73BuSWMJvp_9i7dZ8Ajnl8f3bbdr4%3D HTTP/2.0" 302 55 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" "-"

nginx config

# Protected application
server {
  listen 80;
  listen 443 ssl http2;

# server_name verify.localhost.pomerium.io;
  server_name diborane.domain.com;
  ssl_certificate /etc/nginx/nginx.pem;
  ssl_certificate_key /etc/nginx/nginx-key.pem;


  location = /ext_authz {
    internal;

    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Forwarded-Proto "";

#   update domain to reflect numat-tech.com
    proxy_set_header Host fwdauth.domain.com;
#   proxy_set_header Host fwdauth.localhost.pomerium.io;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Original-Method $request_method;
    proxy_set_header X-Real-IP $remote_addr;

    proxy_set_header X-Forwarded-For $remote_addr;

    proxy_set_header X-Auth-Request-Redirect $request_uri;

    proxy_buffering off;

    proxy_buffer_size 4k;
    proxy_buffers 4 4k;
    proxy_request_buffering on;
    proxy_http_version 1.1;

    proxy_ssl_server_name on;
    proxy_pass_request_headers on;

    client_max_body_size 1m;

    # Pass the extracted client certificate to the auth provider

    set $target http://pomerium/verify?uri=$scheme://$http_host$request_uri;

    # uncomment to emulate nginx-ingress behavior
    # set $target http://pomerium/verify?uri=$scheme://$http_host$request_uri&rd=$pass_access_scheme://$http_host$escaped_request_uri;
    proxy_pass $target;
  }

  location @authredirect {
    internal;
    add_header Set-Cookie $auth_cookie;

    # uncomment to emulate nginx-ingress behavior
    # return 302 https://fwdauth.localhost.pomerium.io/?uri=$scheme://$host$request_uri&rd=$pass_access_scheme://$http_host$escaped_request_uri;

    return 302
#     https://fwdauth.localhost.pomerium.io/?uri=$scheme://$host$request_uri;
#     update domain to reflect numat-tech.com
      https://fwdauth.domian.com/?uri=$scheme://$host$request_uri;
  }

  location / {
   proxy_pass http://192.168.1.5;
#   root   /usr/share/nginx/html;
#   index  index.html index.htm;
#   root         html;
#   index    index.html index.htm index.php;
    include /etc/nginx/proxy.conf;
    # If we get a 401, respond with a named location
    error_page 401 = @authredirect;
    # this location requires authentication
    auth_request /ext_authz;
    auth_request_set $auth_cookie $upstream_http_set_cookie;
    add_header Set-Cookie $auth_cookie;
    # pass JWT assertion upstream
    #signing_key_algorithm LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUI5VDZEa3RBaDd6T1YzQnNOelZZV0p4Q1ppWjJxQklGRDdWc2lvOEM0bkhvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFY1pwc2hiMTBzRndWN3ZzOEdrc3B6NjA2Mmk2bi8rMDJLd0NiZzZCVFFXY2pGVXdTeXRoNgpmTkZPK05tOXJOYXdXSnh5REVUMTB6dUVteFBGQ1lKbGpnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=;
    auth_request_set $auth_resp_x_pomerium_jwt_assertion $upstream_http_x_pomerium_jwt_assertion;
    proxy_set_header x-pomerium-jwt-assertion $auth_resp_x_pomerium_jwt_assertion;
  }
}

Additional context

it seems like any policy that denies a user access put them into a redirect loop

2

Feel free to close this out, most likely user error.

Got fed up trying to diagnose the core issue and just rebuilt everything from scratch and pomerium is humming.

RCA: fat fingered the keyboard

1 Like
3

Hi @XxEnigmaticxX, I’m happy to leave this one be, but in the interest of helping others who might have the same issue I’ll point out that since this appears to be a forward-auth setup, the nginx config would also be helpful in diagnosing.

And again I’m compelled to point out that unless you have a specific scenario that needs it, forward auth is usually more headache than it’s worth. Pomerium can be pointed directly at any upstream service that hosts its own web service.

4

I can upload my nginx config here in a bit, I’d love some help in setting it up the way you describe just unsure how.

5

If you’re interested in a simpler solution, then forget the nginx config. If you’re just experimenting with Pomerium, try this guide I wrote: Securing Grafana with Pomerium. It passes the identity header directly to Grafana, which verifies it with the signing key from the authenticate service and associates the user based on their email in the claim.

6

well, i guess i wasnt full straight forward. while i am experimenting with pomerium, i am experimenting because i need to replace my organizations current auth system. so im making a case to do a cut over by essentially building out pomerium as a proof of concept so i can cut us over. so while it is an experiment, i plan to use this in prod when i get the ok

7

Neat!

While we usually prefer the forums so answers can help others, in this case I think it would behoove us to move to our Slack workspace, so we can talk more directly (and privately, if need be) about your organization’s needs and how to serve them.

Somehow there are two @Alex on Slack, but I’m the one with the matching profile picture :slight_smile: