Grafana auto_sign_up with Pomerium Identity

Integrations
1
2

My current configuration

Grafana.ini

[log]
level = debug
[auth.basic]
enabled = false
[analytics]
check_for_updates = true
[auth.jwt]
auto_sign_up = true
cache_ttl = 60m
email_claim = email
enabled = true
header_name = X-Pomerium-Jwt-Assertion
jwk_set_url = https://authenticate.ops.dev.sw.io/.well-known/pomerium/jwks.json
name_claim = name
username_claim = sub
[grafana_net]
url = https://grafana.net
[log]
mode = console
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[users]
allow_sign_up = false
auto_assign_org = true
auto_assign_org_role = Editor

Helm Values

authenticate:
  existingTLSSecret: pomerium-tls
  idp:
    clientID: ${client_id}
    clientSecret: ${client_secret}
    provider: google
    serviceAccount: ${service_account}
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-pomerium
      ingress.pomerium.io/service_proxy_upstream: "true"
    tls:
      secretName: authenticate-tools-tls
authorize:
  existingTLSSecret: pomerium-tls
config:
  authenticate_service_url: https://authenticate.ops.dev.sw.io
  autocert: false
  existingCASecret: pomerium-tls
  generateTLS: false
  insecure: false
  jwt_claims_headers: email,name
  rootDomain: ops.dev.sw.io
databroker:
  existingTLSSecret: pomerium-tls
  storage:
    clientTLS:
      existingCASecretKey: ca.crt
      existingSecretName: pomerium-redis-tls
    connectionString: rediss://pomerium-redis-master.tools.svc.cluster.local
    tlsSkipVerify: true
    type: redis
extraEnv:
  AUTOCERT: false
  JWT_CLAIMS_HEADERS: email,name
forwardAuth:
  enabled: false
ingress:
  enabled: false
ingressController:
  enabled: true
  image:
    tag: v0.16.1
  namespaces:
  - tools
  - monitoring
proxy:
  existingTLSSecret: pomerium-tls
redis:
  auth:
    enabled: false
  enabled: true
  generateTLS: false
  master:
    disableCommands: []
  replica:
    disableCommands: []
  tls:
    certificateSecret: pomerium-redis-tls
    enabled: true
  usePassword: false

Config.yaml

autocert: false
dns_lookup_family: V4_ONLY
address: :443
grpc_address: :443
certificate_authority_file: "/pomerium/ca/ca.crt"
certificates:
authenticate_service_url: https://authenticate.ops.dev.sw.io
authorize_service_url: https://pomerium-authorize.tools.svc.cluster.local
databroker_service_url: https://pomerium-databroker.tools.svc.cluster.local
idp_provider: google
idp_scopes:
idp_provider_url:
idp_client_id: ${client_id}
idp_client_secret: ${client_secret}
idp_service_account: ${service_account}
databroker_storage_tls_skip_verify: true
routes:

Grafana Logs:

t=2022-03-30T16:04:30+0000 lvl=dbug msg="Parsing JSON Web Token" logger=auth.jwt
t=2022-03-30T16:04:30+0000 lvl=dbug msg="Getting key set from endpoint" logger=auth.jwt url=https://authenticate.ops.dev.sw.io/.well-known/pomerium/jwks.json
t=2022-03-30T16:04:30+0000 lvl=dbug msg="Trying to verify JSON Web Token using a key" logger=auth.jwt
t=2022-03-30T16:04:30+0000 lvl=dbug msg="Validating JSON Web Token claims" logger=auth.jwt
t=2022-03-30T16:04:30+0000 lvl=dbug msg="Failed to find user using JWT claims" logger=context email_claim=sara@sw.com username_claim=<<numbers>>
t=2022-03-30T16:04:30+0000 lvl=eror msg="Invalid JWT" logger=context error="invalid username or password"
t=2022-03-30T16:04:30+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=401 remote_addr=10.0.8.126 time_ms=70 size=31 referer=
3

Resolved! This was a Dumb Thing™ - I was installing Grafana v8.3.6 when I needed Grafana > v8.4.0 . With the correct more recent version of Grafana installed, this works like a charm. Only thing I will note is that I had to set extraEnv.JWT_CLAIMS_HEADERS equal to email,name to get the Name field populated, the config.jwt_claims_headers: email,name alone didn’t seem to cut it. I’m happy to upvote a request to add this so that we can easily get a Name claim populated in the JWT request.

As a follow-up I will submit a PR for the docs so that anyone else installing this gets this lovely feature.

1 Like
4

PR up at feat(docs) Update Grafana docs to have auto_sign_up by sarasensible · Pull Request #3218 · pomerium/pomerium · GitHub

1 Like
5

So I’ve confirmed that auto_sign_up works with our JWT, and now I’m trying to figure out the best order of operations for the guide. I imagine the reader will want at least one user (themselves) to be an admin, but want the auto-assigned role for everyone else to be Editor or Viewer. Additionally, I’d like to rewrite the steps to not require direct access to the Grafana instance for initial setup (partially because I’m travelling and don’t have it myself at the moment, and would prefer not to punch a hole in my security straight to the container).

This post is mostly me rubber-ducking, but I’m thinking that the steps should perhaps not start with pass_identity_headers set on the Route, so that the reader can sign in with the default admin account first and set themselves up with admin access before auto_sign_in starts giving out lesser roles. This to me seems like a better option than changing the Grafana config around, since changes may recreate the container, when using docker-compose for example, and data persistence for Grafana is outside the scope of our guide.

I’m going to experiment more with this, but any and all input is welcome.

6

I just re-performed my setup as described above, where the steps are:

  1. Configure route without pass_identity_headers, and with preserve_host_header set.
  2. Login with the default admin/admin account
  3. Create an admin account for my IdP-provided user. Here it must be noted that when using Google the user ID will be a numeric string and not your email address.
  4. Remove preserve_host_header and add pass_identity_headers to the route
  5. Now I can sign in as myself directly from Pomerium as an admin, and new users will be created as editors.

I think this is probably the path to take. Again, all input welcome.

EDIT: P.S. preserve_host_header is required with the latest Grafana in order to sign in with the default admin account. Source: Origin not allowed messages after upgrade to 8.3.6 - InfluxDB - Grafana Labs Community Forums

7

I can’t speak for others but what is working in our use case is to restrict admin use to port-forwarding the service internally so that all ingress is bypassed for the less secure setup. I do like the idea of customizing one particular user to be admin, but for our use case we need to be able to share around the admin account internally so tying it to one user would defeat the purpose.

Ultimately the “right way” to do this would be go to Auth Proxy and use Grafana’s Enterprise features to associate groups with teams so that we could have a group of admins, but for a cost-conscious startup this is a good enough solution for now.

2 Likes
8

@saranicole thanks for the input. It would seem that there are several ways to peel this orange, and which is best is highly dependent on the use case. So I’m going to document the method that’s most reliant on Pomerium’s config, but also reference the other methods available.

Before I get to your PR though, I still need to test the jwt_claims_headers option. I’m also going to see if adjusting set_authorization_header will work to provide the entire data set from the IdP, and possibly let us set the username to something better for the case of Google.

1 Like
9

The Grafana Guide is updated! Thanks @saranicole for all your effort on this issue. I hope it will help others as well.

1 Like