HttpClient 502 Bad Gateway Async only

Question

I am having trouble with calling a web service asynchronously. I have called it synchronously without error hundreds of times inside a loop, but when I call it asynchronously, I get a 502 Bad Gateway error, usually after about 250 asynchronous calls. As you can see, I've tried waiting 50ms with Thread.Sleep() and retrying up to 5 times per record. The 404 Not Found error is expected when I'm unable to validate a record. My framework is netstandard2.0.

class Program
    {
        static void Main(string[] args)
        {
            List<OriginalRecord> originalRecords = new List<OriginalRecord>();
            originalRecords.Add(new OriginalRecord()
            {
                Id = "IdOne",
                Foo = "fooOne"
            });
            originalRecords.Add(new OriginalRecord()
            {
                Id = "IdTwo",
                Foo = "fooTwo"
            });

            List<RecordValidation> unvalidatedRecords = new List<RecordValidation>();

            try
            {
                RecordValidator validator = new RecordValidator("myAccessToken", "myBarId");
                var task = validator.ValidateRecordsAsync(originalRecords.Select(r => r.Id).ToList());
                task.Wait();
                unvalidatedRecords = task.Result;
            }
            catch
            {
                throw;
            }
        }
    }
    public class RecordValidator
    {
        public RecordValidator(string AccessToken, string BarId)
        {
            httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.UserAgent.TryParseAdd("MessageService/3.1");

            httpClient.DefaultRequestHeaders.Authorization =
                new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", AccessToken);

            httpClient.DefaultRequestHeaders.Add("Accept", "application/hal+json");
            httpClient.DefaultRequestHeaders.ExpectContinue = false;

            barId = BarId;
        }

        private static HttpClient httpClient { get; set; }
        
        public string barId { get; set; }

        public async Task<List<RecordValidation>> ValidateRecordsAsync(List<string> Records)
        {
            List<Task<RecordValidation>> tasks = new List<Task<RecordValidation>>();

            foreach (var record in Records)
            {
                tasks.Add(Task.Run(() => ValidateRecordAsync(record)));
            }

            var results = await Task.WhenAll(tasks);
            return results.Where(r => !r.Validated).ToList();
        }

        private async Task<RecordValidation> ValidateRecordAsync(string Record)
        {
            string url = $"https://data.somewebapp.com/api/bar/{barId}/record/{Uri.EscapeDataString(Record)}";
            int retries = 0;
            bool success = false;
            HttpResponseMessage response = new HttpResponseMessage();
            try
            {

                do
                {
                    response = await httpClient.GetAsync(url);
                    if (response.IsSuccessStatusCode)
                    {
                        success = true;
                    }
                    else
                    {
                        Thread.Sleep(50);
                        retries++;
                    }
                } while (retries < 5 && !success);


                response.EnsureSuccessStatusCode();

                string responseString = await response.Content.ReadAsStringAsync();

                var output = JsonConvert.DeserializeObject<RecordValidation>(responseString);

                if (!string.IsNullOrEmpty(output.RecordId))
                    output.Validated = true;
                return output;
            }

            catch (HttpRequestException ex) when (ex.Message == "Response status code does not indicate success: 404 (Not Found).")
            {
                return new RecordValidation()
                {
                    RecordId = Record,
                    Validated = false
                };
            }

            catch
            {
                throw;
            }
        }
    }
    public class RecordValidation
    {
        public RecordValidation()
        {
            RecordId = string.Empty;
            Validated = false;
        }

        [JsonProperty("id")]
        public string RecordId { get; set; }

        public bool Validated { get; set; }
    }

    public class OriginalRecord
    {
        public string Id { get; set; }
        public string Foo { get; set; }
    }
}

Answer 1

250 parallel calls are way too much. It should be rather 10-30. Default max number of concurrent requests for the C# Http client is 10, so if you aren't overriding that value, it doesn't make sense to try to open more than 10 socket connections simultaneously. Also, keep in mind, 502 status is most likely returned by reverse proxy, which means you aren't really hitting the web API, proxy is telling you "slow down, I cannot handle that anymore". So in your case, it's not really HttpClient issue, otherwise, you would get some exception, like SocketException due to socket exhaustion, which you aren't probably getting due to HttpClient limitting concurrency to 10. Also, you have plenty of thread blocking code, like Thread.Sleep() or task.Wait(), change your main method signature to static async Task Main() and use await everywhere, otherwise, you will run into other types of issues, like thread exhaustion