MongoDB HealthCheck Intermittently getting CancellationToken

Question

I am running into an intermittent Connection Closed issue resulting in a Cancellation Token.

We have an AKS Cluster with many pods. Each pod has a health check leveraging the AspNetCore Diagnostic HeathCheck pipeline.

We are using Mongo Atlas, hosted in Azure, connecting over a private endpoint.

For each pod's health check, we leverage the AspNetCore Diagnostic Health Check Package for MongoDB. This under the hood, connects to Mongo with a connection string pulled from IConfiguration and connects to the Database with the configuration settings provided by the connection string. It then does a simple query, asking the database to return a list of collections within the database. If this is successful it will return a Healthy Result back from the AspNetCore Diagnostics thus to the K8 probe.

For K8 probe here is the configuration:

livenessProbe:
   failureThreshold: 3
   httpGet:
     path: /Heartbeat
     port: 8080
     scheme: HTTP
   initialDelaySeconds: 3
   periodSeconds: 10
   successThreshold: 1
   timeoutSeconds: 15

Intermittently we are getting these errors wears the pod reports back an unhealthy connection, saying the connection has been closed. Below is the stack trace.

System.OperationCanceledException: The operation was canceled. at 
System.Threading.CancellationToken.ThrowOperationCanceledException() at
MongoDB.Driver.Core.Connections.BinaryConnection.ThrowIfCancelledOrDisposedOrNotOpen(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.SendMessagesAsync(IEnumerable`1 messages, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at
MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.HelloHelper.GetResultAsync(IConnection connection, CommandWireProtocol`1 helloProtocol, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.ConnectionInitializer.SendHelloAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.PooledConnection.OpenAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedInternalAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuseAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.ListCollectionsOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at
MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ExecuteReadOperationAsync[T](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ListCollectionNamesAsync(IClientSessionHandle session, ListCollectionNamesOptions options, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken) at
HealthChecks.MongoDb.MongoDbHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /_/src/HealthChecks.MongoDb/MongoDbHealthCheck.cs:line 57

I don't see any particular pattern or rhyme or reason why this is occurring especially because it's random. Also it happens when there is little to no traffic on the cluster either. So I have ruled out load.

Any ideas?