Grafana oauth with Gitlab

Question

Environment:

• Grafana version: 8.2.1 Deployed on GKE using the kube-promethues-stack

What happened: I have one gke cluster where gitlab is running and exposed with two ingress: 1 with internal IP and 1 with external IP.

In a second cluster I have a grafana deployment that I would like to "connect" to gitlab in order to use gitlab as authentication provider. Since the internal url of gitlab is not reachable from grafana, I use the external one, but something doesn't seem to work: in the logs I see that a requests goes from grafana to the internal IP, and I can't understand why. This is the grafana config:

    auth.gitlab:
      enabled: true 
      allow_sign_up: true
      scopes: read_api
      auth_url: https://external-url/oauth/authorize
      token_url: https://external-url/oauth/token
      api_url: https://external-url/api/v4

and these are the logs from grafana:

t=2021-12-01T12:36:30+0000 lvl=dbug msg="OAuthLogin Got token" logger=oauth token="&{AccessToken:redacted TokenType:Bearer RefreshToken:redacted Expiry:0001-01-01 00:00:00 +0000 UTC raw:map[access_token:redacted created_at:1.637755993e+09 refresh_token:redacted scope:read_api token_type:Bearer]}"
t=2021-12-01T12:36:30+0000 lvl=dbug msg="HTTP GET https://external-url/api/v4/user: 200 OK {redacted information}
t=2021-12-01T12:36:30+0000 lvl=dbug msg="HTTP GET https://external-ip/api/v4/groups: 200 OK {redacted information}
t=2021-12-03T07:20:42+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.\"}"
t=2021-12-03T07:20:42+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/gitlab status=500 remote_addr=10.175.1.36 time_ms=58 size=1365 referer=https://EXTERNAL/

t=2021-12-01T12:38:42+0000 lvl=eror msg="Error getting groups from GitLab API" logger=oauth.gitlab err="Get \"https://INTERNAL-URL/api/v4/groups?order_by=name&owned=false&page=2&per_page=20&sort=asc&statistics=false&with_custom_attributes=false\": dial tcp internal-ip:443: connect: connection timed out"
t=2021-12-01T12:38:42+0000 lvl=dbug msg="OAuthLogin got user info" logger=oauth userInfo="&{redacted information}

Can anyone help me with this? Thanks in advance

Answer 1

Was the gitlab's oauth client secret got changed after the upgrade/any activity?
I use Keycloak's oauth, where the client id and client secret can be changed and if it is changed, I have to update the client secret value at Grafana's ENV GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET or config oauth_client_secret

This is the cause for my login.OAuthLogin(NewTransportWithCode) error.

Try and let me know.