Json web token scheme problem with webapi

Question

So I'm adding JWT authentication for a webapi. Everything works fine until I messed around with the scheme settings. Somehow the following settings dont work any more

builder.Services.AddAuthentication(options =>
{
    options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
    options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;

}).AddJwtBearer(options =>
{
    options.Events = new JwtBearerEvents
    {
        OnTokenValidated = context =>
        {
            //TODO
            var userMachine = context.HttpContext.RequestServices.GetRequiredService<UserManager<User>>();
            var user = userMachine.GetUserAsync(context.HttpContext.User);
            if (user == null)
                context.Fail("Unauthorized");
            return Task.CompletedTask;
        }
    };

    options.RequireHttpsMetadata = false;
    options.SaveToken = true;
    options.TokenValidationParameters = new TokenValidationParameters
    {
        ValidateIssuerSigningKey = true,
        IssuerSigningKey = new SymmetricSecurityKey(key),
        ValidateIssuer = false,
        ValidateAudience = false
    };

});

The problem is this scheme line of code

options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;

when I try to call a api method it somehow returns 404 instead of 401

But if I overwrite this scheme in the controller like this

   [Route("api/[controller]")]
    [ApiController]
    [Authorize(AuthenticationSchemes = AuthSchemes)]
    public class ValuesController : ControllerBase
    {
        private const string AuthSchemes = JwtBearerDefaults.AuthenticationScheme;

Wether I commented the previous scheme line of code or not the program runs. And if I call a api with out the JWT is returns 401 now and everything works.

Anybody knows why? Thank you!

PS. I didn't add an identity. Instead I added identity manually from scratch. I noticed that the problem happens after I create a new custom user inheriting from IdentityUser and add an migration. So basically it happens after I add these code below:

builder.Services.AddIdentity<ApplicationUser, IdentityRole>(options =>
{
    options.Password.RequireNonAlphanumeric = false;
}).AddEntityFrameworkStores<ApplicationContext>();

in program.cs and

 public class ApplicationContext : IdentityDbContext<ApplicationUser>
    {
        public ApplicationContext(DbContextOptions<ApplicationContext> options):base(options)   
        {

        }

    }

in Dbcontext.

Seems like for it to work I have to use the built-in identity. But the question is why it doesnt work if I do it manually?