System.TimeoutException: A timeout occurred after 30000ms selecting a server using CompositeServerSelector

Question

my mongodb is running on an ec2 instance (T3.medium). I've used route53 to point to this instance with an 'A' record, so my connection string ends like this: mongodb://user:[email protected]/authSource=admin&readPreference=primary&directConnection=true

and I have the following (simplified) AWS lambdas:

public class Function
{
  public async Task FunctionA(ILambdaContext context)
  {
     try
     {   
        var until = DateTimeOffset.UtcNow;
        var collection = GetCollection();
        var token = GetCancellationToken(context);
        await collection.DeleteManyAsync(x => x.ValidUntilDate != null && x.ValidUntilDate <= until, token);
     }
     catch (Exception e)
     {
         context.Logger.LogLine($"Unknown error occurred. Error = {e}");
     }
  }

  public async Task FunctionB(ILambdaContext context)
  {
     try
     {  
         var until = DateTimeOffset.UtcNow;
         var collection = GetCollection();
         var token = GetCancellationToken(context);
         var update = Builders<TheCollection>.Update.Combine(
               Builders<TheCollection>.Update.PullFilter(
                   x => x.AnArray,
                   x => x.ValidUntilDate != null && x.ValidUntilDate <= until),
               Builders<TheCollection>.Update.Set(x => x.Visited, true));
         await _fingerprintHashTableCollection.UpdateManyAsync(
               x => !x.Visited && x.AnArray.Any(f => f.ValidUntilDate != null && f.ValidUntilDate <= until),
               update,
               cancellationToken: token);
     }
     catch (Exception e)
     {
         context.Logger.LogLine($"Unknown error occurred. Error = {e}");
     }
  }

  public async Task FunctionC(ILambdaContext context)
  {
     try
     {  
         var collection = GetCollection();
         var token = GetCancellationToken(context);
         await collection.DeleteManyAsync(x => x.AnArray.Count == 0, token);
     }
     catch (Exception e)
     {
         context.Logger.LogLine($"Unknown error occurred. Error = {e}");
     }
  }

  public IMongoDatabase GetDatabase() 
  {
    string connectionString = Environment.GetEnvironmentVariable("THE_CONNECTION_STRING");
    var dbClient = new MongoClient(connectionString);
    return dbClient.GetDatabase(dbName);
  }

  public IMongoCollection<TheCollection> GetCollection()
  {
     var db = GetDatabase();
     return db.GetCollection<TheCollection>("the_collection");
  }

  private static CancellationToken GetCancellationToken(ILambdaContext context)
  {
      double seconds = context.RemainingTime.TotalSeconds / 1.5;
      var ts = TimeSpan.FromSeconds(seconds);
      return new CancellationTokenSource(ts).Token;
  }
}

FunctionA Run each minute and has a timeout of 15min

FunctionB Run each 5 minutes and has a timeout of 15min

FunctionC Run each 10 minutes and has a timeout of 15min

FunctionA Completes without problems (it takes 2 - 3 seconds to complete)

FunctionB Completes without problems (it takes 2 seconds to complete)

The problem comes with FunctionC, it has a weird behaviour where 2 out 5 executions fail with one these two errors

First error:

Unknown error occurred. 
Error = System.TimeoutException: A timeout occurred after 30000ms selecting a server using CompositeServerSelector
{ 
Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector
{ 
AllowedLatencyRange = 00:00:00.0150000 
}, 
OperationsCountServerSelector 
}. 
Client view of cluster state is { ClusterId : "1", DirectConnection : "True", Type : "Standalone", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mymongodb.com:27017" }", EndPoint: "Unspecified/mymongodb.com:27017", ReasonChanged: "ServerInitialDescription", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", LastHeartbeatTimestamp: null, LastUpdateTimestamp: "2022-04-01T19:56:34.2250329Z" }] }
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAfterServerSelectionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.StartImplicitSessionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionBase`1.UpdateManyAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)

Second error (xx.yyy.xx.zz is the public ip address of my ec2 instance):

Unknown error occurred. Error = System.TimeoutException: A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", DirectConnection : "True", Type : "Standalone", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mymongodb.com:27017" }", EndPoint: "Unspecified/mymongodb.com:27017", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server
---> System.TimeoutException: Timed out connecting to xx.yyy.xx.zz:27017. Timeout was 00:00:30.
MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)
MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)
MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)", LastHeartbeatTimestamp: "2022-04-01T19:47:04.8863017Z", LastUpdateTimestamp: "2022-04-01T19:47:04.8863830Z" }] }.
MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
MongoDB.Driver.Core.Clusters.Cluster.SelectServerHelper.WaitingForDescriptionToChange()
MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
MongoDB.Driver.MongoClient.AreSessionsSupportedAfterServerSelectionAsync(CancellationToken cancellationToken)
MongoDB.Driver.MongoClient.AreSessionsSupportedAsync(CancellationToken cancellationToken)
MongoDB.Driver.MongoClient.StartImplicitSessionAsync(CancellationToken cancellationToken)
MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
MongoDB.Driver.MongoCollectionBase`1.UpdateManyAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)

and I'm not sure why this randomly happens. Would love some help, thanks!