Outbound TCP Connection issue cause be sending data to event hub and data lake from azure function

Question

I'm working on a Azure function with http POST trigger, once client call it and post a json data, I will send it to event hub and save to data lake. once it got hitted by the high traffic, 20k/hour, azure functino will generate high outbound TCP connection, which will exceed the limitation (1920) of the plan.

1. does high outbound TCP connection cause by writing to event hub, data lake, or both?
2. is there a way to reduce it so I don't have to pay more to upgrade our plan?
3. how to debug it to trouble shooting the problem?

here is the code of send data to event hub:

EventHubClient ehc = EventHubClient.CreateFromConnectionString(cn);

try
{
  log.LogInformation($"{CogniPointListener.LogPrefix}Sending {batch.Count} Events: {DateTime.UtcNow}");

  await ehc.SendAsync(batch);

  await ehc.CloseAsync();
}
catch (Exception exception)
{
  log.LogError($"{CogniPointListener.LogPrefix}SendingMessages: {DateTime.UtcNow} > Exception: {exception.Message}");
  throw;
}

here is the send data to data lake:

var creds = new ClientCredential(clientId, clientSecret);
var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();

// Create ADLS client object
AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);

try
{
    using (var stream = client.CreateFile(fileName, IfExists.Overwrite))
    {
        byte[] textByteArray = Encoding.UTF8.GetBytes(str);
        stream.Write(textByteArray, 0, textByteArray.Length);
    }

    // Debug
    log.LogInformation($"{CogniPointListener.LogPrefix}SaveDataLake saved ");
}
catch (System.Exception caught)
{
    string err = $"{caught.Message}Environment.NewLine{caught.StackTrace}Environment.NewLine";
log.LogError(err, $"{CogniPointListener.LogPrefix}SaveDataLake");
    throw;
}

Thanks,

Answer 1

TCP connections are limited in specific numbers depending on the plan you have your functions on (Consumption or a static plan in any level B/S/P). For high workloads I prefer to either

A: Use a queue with a separate function and limiting the concurrency by the function batch size and other settings

or

B: Use a SemaphoreSlim in order to control concurrency of outgoing traffic. (https://docs.microsoft.com/de-de/dotnet/api/system.threading.semaphoreslim?redirectedfrom=MSDN&view=netframework-4.7.2)

Answer 2

I just raised an issue with Azure SDK https://github.com/Azure/azure-sdk-for-net/issues/26884 reporting the problem of socket exhaustion when using ApplicationTokenProvider.LoginSilentAsync.

The current version 2.4.1 of Microsoft.Rest.ClientRuntime.Azure.Authentication uses the old version 4.3.0 of Microsoft.IdentityModel.Clients.ActiveDirectory that creates a new HttpClientHandler on every call.

Creating HttpClientHandler on every is bad. After HttpClientHandler is disposed, the underlaying socket connections are still active for significant time (in my experience 30+ seconds).

There's a thing called HttpClientFactory that ensures HttpClientHandler is not created frequently. Here's a guide from Microsoft explaining how to use HttpClient and HttpClientHandler properly - Use IHttpClientFactory to implement resilient HTTP requests. I wish they reviewed their SDKs to ensure they follow their own guidelines.

Possible workaround

Microsoft.IdentityModel.Clients.ActiveDirectory since version 5.0.1-preview supports passing a custom HttpClientFactory.

IHttpClientFactory myHttpClientFactory = new MyHttpClientFactory();

AuthenticationContext authenticationContext = new AuthenticationContext(
     authority: "https://login.microsoftonline.com/common",
     validateAuthority: true,
     tokenCache: <some token cache>,
     httpClientFactory: myHttpClientFactory);

So it should be possible to replicate what ApplicationTokenProvider.LoginSilentAsync does in your codebase to create AuthenticationContext passing your own instance of HttpClientFactory.

The things you might need to do:

• Ensure Microsoft.IdentityModel.Clients.ActiveDirectory with version of after 5.0.1-preview is added to the project
• Since the code is used in Azure functions, HttpClientFactory needs to be set up and injected. More info can be found in another StackOverflow answer
• Replace calls ApplicationTokenProvider.LoginSilentAsync(tenantId, creds) with something like that (this code is an inlined version of LoginSilentAsync that passes httpClientFactory to AuthenticationContext

var settings = ActiveDirectoryServiceSettings.Azure;
var audience = settings.TokenAudience.OriginalString;
var context = new AuthenticationContext(settings.AuthenticationEndpoint + domain,
    settings.ValidateAuthority,
    TokenCache.DefaultShared,
    httpClientFactory);

var authenticationProvider = new MemoryApplicationAuthenticationProvider(clientCredential);

var authResult = await authenticationProvider.AuthenticateAsync(clientCredential.ClientId, audience, context).ConfigureAwait(false);
var credentials = new TokenCredentials(
    new ApplicationTokenProvider(context, audience, clientCredential.ClientId, authenticationProvider, authResult),
    authResult.TenantId,
    authResult.UserInfo == null ? null : authResult.UserInfo.DisplayableId);

I really don't replicating the logic in the workaround, but I don't think there's any other option until it's fixed properly in Microsoft.Rest.ClientRuntime.Azure.Authentication

Good luck!