Parallelize C# Graph API SDK methods

Question

I'm connecting to and fetching transitive groups data from MS Graph API via. following logic:

var queryOptions = new List<QueryOption>()
{
      new QueryOption("$count", "true")
};

var lstTemp = graphClient.Groups[$"{groupID}"].TransitiveMembers
                    .Request(queryOptions)
                    .Header("ConsistencyLevel", "eventual")
                    .Select("id,mail,onPremisesSecurityIdentifier").Top(999)
                    .GetAsync().GetAwaiter().GetResult();


var lstGroups = lstTemp.CurrentPage.Where(x => x.ODataType.Contains("group")).ToList();

while (lstTemp.NextPageRequest != null)
{
           lstTemp = lstTemp.NextPageRequest.GetAsync().GetAwaiter().GetResult();
           lstGroups.AddRange(lstTemp.CurrentPage.Where(x => x.ODataType.Contains("group")).ToList());
}

Although the following logic works fine, for larger data set where the result count could be around 10K records or more, I've noticed the time required to fetch all of the results is around 10-12 seconds.

I'm looking for a solution by which we can parallelize (or multi-threading/tasking) API calls are executed in such a way that the overall time to get completed results is further reduced. In C# we have Parallel.For etc. can I use it in this scenario to replace my regular While loop mentioned above?

Any suggestions?

Answer 1

Not really using the Parallel.For api, but you can execute a bunch of asynchronous tasks concurrently by throwing them into a List<Task<T>> and awaiting the whole list with Task.WhenAll. Your code may look something like this:

var queryOptions = new List<QueryOption>()
{
        new QueryOption("$count", "true")
};

// Creating the first request
var firstRequest = graphClient.Groups[$"{groupID}"].TransitiveMembers
                    .Request(queryOptions)
                    .Header("ConsistencyLevel", "eventual")
                    .Select("id,mail,onPremisesSecurityIdentifier").Top(999)
                    .GetAsync();

// Creating a list of all requests (starting with the first one)
var requests = new List<Task<IGroupTransitiveMembersCollectionWithReferencesPage>>() { firstRequest };

// Awaiting the first response
var firstResponse = await firstRequest;

// Getting the total count from the request
var count = (int) firstResponse.AdditionalData["@odata.count"];

// Setting offset to the amount of data you already pulled
var offset = 999;

            
while (offset < count)
{
    // Creating the next request
    var nextRequest = graphClient.Groups[$"{groupID}"].TransitiveMembers
        .Request() // Notice no $count=true (may potentially hurt performance and we don't need it anymore anyways)
        .Header("ConsistencyLevel", "eventual")
        .Select("id,mail,onPremisesSecurityIdentifier")
        .Skip(offset).Top(999) // Skipping the data you already pulled
        .GetAsync();

    // Adding it to the list
    requests.Add(nextRequest);

    // Increasing the offset
    offset += 999;
}

// Waiting for all the requests to finish
var allResponses = await Task.WhenAll(requests);

// This flattens the list while filtering as you did
allResponses
    .Select(x => x.CurrentPage)
    .SelectMany(x => x.Where(x => x.ODataType.Contains("group")));

Couldn't check if this code works without a Graph tenant, so you might need to modify a bit, but I hope you can see the general idea.

Also I allowed myself to refactor the code to use proper async/await since it's good and standard practice to do that, but it should work with .GetAwaiter().GetResult() if you can't use await in your context for some reason (please consider, though).