Using Refit with Microsoft Identity causes a constant loop of challenge exceptions

Question

I've recently added authentication and authorization to my API calls through Refit, using the standard MSAL implementation. The configuration is shown below:

services.AddAuthentication(options =>
            {
                options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
                options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
            })
            .AddMicrosoftIdentityWebApp(options =>
                {
                    options.Instance = Configuration.GetSection("AzureAd")["Instance"];
                    options.TenantId = Configuration.GetSection("AzureAd")["TenantId"];
                    options.ClientId = Configuration.GetSection("AzureAd")["ClientId"];
                    options.Domain = Configuration.GetSection("AzureAd")["Domain"];
                    options.ClientSecret = Configuration.GetSection("AzureAd")["ClientSecret"];
                    options.CallbackPath = Configuration.GetSection("AzureAd")["CallbackPath"];
                    options.SignedOutCallbackPath = Configuration.GetSection("AzureAd")["SignedOutCallbackPath"];
                },
                cookieAuthOptions =>
                {
                    cookieAuthOptions.ExpireTimeSpan = TimeSpan.FromMinutes(30);
                    cookieAuthOptions.SlidingExpiration = true;
                })
            .EnableTokenAcquisitionToCallDownstreamApi(new[] { Configuration["DocumentManager:Scopes"] })
            .AddDownstreamWebApi("DocumentManager", opts =>
            {
                opts.Scopes = Configuration["DocumentManager:Scopes"];
                opts.BaseUrl = "https://localhost:44350/";
            })
            .AddInMemoryTokenCaches();

Now I have added the DownstreamWebApi part to test if it would work using that in place of Refit and it works as expected. This leads me to believe that something is going wrong with my implementation of Refit. Here I add a delegating handler for obtaining and adding the token.

services.AddRefitClient<IClientData>()
            .ConfigureHttpClient(c =>
                c.BaseAddress = new Uri(Configuration.GetConnectionString("DocumentApiAddress")))
            .AddHttpMessageHandler<DmAuthHeaderHandler>();

The handler is seen below. It uses the built-in token acquisition interface:

public class DmAuthHeaderHandler : DelegatingHandler

{
    private readonly IConfiguration _configuration;
    private readonly ITokenAcquisition _tokenAcquisition;

    public DmAuthHeaderHandler(IConfiguration configuration, ITokenAcquisition tokenAcquisition) : base()
    {
        _configuration = configuration;
        _tokenAcquisition = tokenAcquisition;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var scopes = new[] { _configuration["DocumentManager:Scopes"] };

        var token = await _tokenAcquisition.GetAccessTokenForUserAsync(scopes);
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
        request.Headers.Add("X-Tenant-Id", _configuration["AzureAd:TenantId"]);

        return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
    }
}

And I call the refit interface method here:

protected override async Task OnInitializedAsync()
{
    try
    {
        _currentUser = (await AuthProvider.GetAuthenticationStateAsync()).User;
        var username = _currentUser.Identity.Name;
        var userAzureId = (await UserRepository.GetSingleUserWithSelectedPropertyAsync(x => x.Id, username)).Id;
        
        _client = await _clientDataAccess.GetClientAsync(ClientId, userAzureId);
        
    }
    catch (MicrosoftIdentityWebChallengeUserException ex)
    {
        _consentHandler.HandleException(ex);
    }
    catch (MsalUiRequiredException ex)
    {
        _consentHandler.HandleException(ex);
    }
}

It then issues a challenge, goes to login.microsoft.com and redirects back, after which it fetches the token and actually calls the API successfully. However, immediately after this it then challenges again and repeats the process.