SecurityTokenException when trying to authenticate ASPNET Core MVC app with Ws-Fed

Question

I have an ASPNet Core 2.2 app which I'm trying to configure to use WsFed Auth for authenticating with our STS. I've added the following to startup.

  services.AddAuthentication(sharedOptions =>
            {
                sharedOptions.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                sharedOptions.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
                sharedOptions.DefaultChallengeScheme = WsFederationDefaults.AuthenticationScheme;
            })
            .AddWsFederation(options =>
            {
                options.Configuration = new WsFederationConfiguration
                {
                    TokenEndpoint = Configuration["Sts:Issuer"],
                    Issuer = "FP",
                    KeyInfos = { new KeyInfo(GetStsIssuerCert(Configuration["Sts:Thumbprint"])) },

                };
                options.CallbackPath = new PathString("/Home/CallBack");
                options.ClaimsIssuer = Configuration["Sts:Issuer"];
                options.Wtrealm = Configuration["Sts:Realm"];
                options.TokenValidationParameters = new TokenValidationParameters
                {
                    ValidAudience = Configuration["Sts:Realm"]
                };
            })
            .AddCookie();

This works as expected when trying to hit main controller action which has "Authorize" attribute, however the User.Identity data is not populated. If I then add the following in startup.

   app.UseAuthentication();

I get the following error.

   SecurityTokenException: No token validator was found for the given token.  
Microsoft.AspNetCore.Authentication.WsFederation.WsFederationHandler.HandleRemoteAuthenticateAsync()

Do I need to use UseAuthentication()?

Any help on this would be greatly appreciated.