Keycloak with ASP.NET: API returns 401 with authorize attribute

Question

I am trying to set up Keycloak in my ASP.NET application.

The intended flow is: Frontend (Vue.js) redirects to Keycloak login page, and only when authenticated allows the user to proceed. It then requests the data from the ASP.NET API.

The Vue.JS part seems to be working and the bearer token is manually placed in the request to the ASP.NET API. The network tab proves that the bearer token is included. Nonetheless, the API always returns 401.

My suspicion is that the backend configuration is not correct.

services
    .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddOpenIdConnect(options => SetOpenIdConnectOptions(options))
    .AddJwtBearer();

private void SetOpenIdConnectOptions(OpenIdConnectOptions options)
{
    options.SignInScheme = JwtBearerDefaults.AuthenticationScheme;
    options.Authority = Configuration.GetSection("Keycloak")["ServerRealm"];
    options.ClientId = Configuration.GetSection("Keycloak")["ClientId"];
    options.ClientSecret = Configuration.GetSection("Keycloak")["ClientSecret"];
    options.MetadataAddress = Configuration.GetSection("Keycloak")["Metadata"];
    options.RequireHttpsMetadata = false;
    options.GetClaimsFromUserInfoEndpoint = true;
    options.SaveTokens = true;
    options.ResponseType = OpenIdConnectResponseType.Token;
    options.Scope.Add("openid");
    options.Scope.Add("profile");
    options.TokenValidationParameters = new TokenValidationParameters()
    {
        ValidateAudience = true,
        ValidAudience = "test_audience",

        ValidateIssuer = true,
        ValidIssuers = new[] { options.Authority },

        ValidateIssuerSigningKey = false,
        //IssuerSigningKeys = openidconfig.SigningKeys,

        RequireExpirationTime = true,
        ValidateLifetime = true,
        RequireSignedTokens = false,
    };
}

What have I tried and important notes

I have tried to debug the authorization by creating events on the OpenIdConnectionOptions. None of the events are triggered. However, when I do the same thing on the .AddJwtBearer(), the events are triggered and complain about the "kid" (Signature validation failed. Unable to match key: kid:). I've understood that the .AddJwtBearer() call is required, but it seems a bit dodgy.

I have also tried this with similar settings, such as without the TokenvalidationParameters.