Firestore query never completes inside Google Cloud Event Trigger Function

Question

I have a Google Cloud Event Trigger Function that has been running for a few months. A short time ago the function started to fail and I've been looking into it. I've narrowed it down to a request for a specific Firestore collection never completing.

Here is the code I'm using to troubleshoot. I've removed everything except for a console log and a call to a function to help me calculate the size of the collection (both of which I added after the problem surfaced). I can run this fine in the Firebase Emulator using a current backup and get a total size of the collection equal to about 6.5MB. The collection has 3001 documents in it.

    let totalSize = 0;

    await admin
        .firestore()
        .collection("users")
        .get()
        .then(function(querySnapshot) {
          querySnapshot.forEach(function(doc) {
            console.log(doc.data().name);
            const bytes = sizeof(doc.data());
            totalSize += bytes;
          });
        });

    console.log("total size of the users collection: ", totalSize);
  },

When running from GCP, the function always errors out after 5 minutes with these function logs:

After the Function execution took 222ms line, it waits five minutes (which is the max time a GCF can run) and gives the Unhandled rejection error.

If I replace the users collection with any other collection it works fine.

Either I'm too tired (most likely), or my Google-Fu isn't what I thought it was, or I'm the first person to ever encounter this (least likely).

Why would a 6.5MB collection never return when running inside a Google Cloud Function?

UPDATE: added logs as requested (changed project/location/function names)

insertId,labels.execution_id,logName,receiveTimestamp,resource.labels.function_name,resource.labels.project_id,resource.labels.region,resource.type,severity,textPayload,timestamp,trace
000000-57823b5d-a3c3-49c8-bca6-617f8ba6fbc4,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,ERROR,"Error: Process exited with code 16
    at process.<anonymous> (/layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/invoker.js:96:22)
    at process.emit (events.js:400:28)
    at process.emit (domain.js:475:12)
    at process.exit (internal/process/per_thread.js:173:15)
    at sendCrashResponse (/layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/logger.js:38:9)
    at process.<anonymous> (/layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/invoker.js:92:44)
    at process.emit (events.js:400:28)
    at process.emit (domain.js:475:12)
    at processPromiseRejections (internal/process/promises.js:245:33)
    at processTicksAndRejections (internal/process/task_queues.js:96:32)",2022-01-26T14:19:56.798Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000009-b01ee32a-6f0e-4dc5-9d02-ddcbc955f8de,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at processTicksAndRejections (internal/process/task_queues.js:95:5)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000008-81c85f7d-e66e-4eb1-913c-f35578ea80fd,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:140:25",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000007-36da194f-7a44-4c98-a86b-54f8a85ee883,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at cloudFunction (/workspace/node_modules/firebase-functions/lib/cloud-functions.js:134:23)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000006-0a7895f7-3180-4b84-a8bc-4638108c2efd,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at /workspace/_competingHorseActions/index.js:15:21",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000005-966bd66e-0725-42a9-aafa-f4f3d587a01a,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at Object.createSearchesToDo (/workspace/_competingHorseActions/dataFunctions.js:16:10)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000004-9efc598d-50e0-4b76-b084-b738e1149460,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at CollectionReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:1438:21)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000003-72ff7fe8-9b67-4cca-96bc-293d7cf71528,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,"    at CollectionReference._get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:1449:23)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000002-6b781252-5a83-4deb-9189-e257db5a96b7,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,Caused by: Error,2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000001-54f3eb45-17da-46e9-9420-ec9541a1de22,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,ERROR,"Error: 9 FAILED_PRECONDITION: The requested snapshot version is too old.
    at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:331:49)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78
    at processTicksAndRejections (internal/process/task_queues.js:77:11)",2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000000-a62c012a-0e24-40cf-b18b-0e883577f022,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:20:05.306874125Z,myEventTriggerFunction,my-project,my-location,cloud_function,,Unhandled rejection,2022-01-26T14:19:55.541Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000000-475f9e2c-4cd8-4cc3-80d1-5ac7cf3dd12f,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:14:58.785021807Z,myEventTriggerFunction,my-project,my-location,cloud_function,DEBUG,"Function execution took 1272 ms, finished with status: 'ok'",2022-01-26T14:14:48.976106222Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000000-40548c05-e247-4de0-b876-5bec6b081452,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:14:58.785021807Z,myEventTriggerFunction,my-project,my-location,cloud_function,,createSearchesToDo Search,2022-01-26T14:14:48.966Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08
000000-ec65a497-72ac-4a87-9a00-4baba7b8c04e,ipijpyxnda0o,projects/my-project/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2022-01-26T14:14:58.785021807Z,myEventTriggerFunction,my-project,my-location,cloud_function,DEBUG,Function execution started,2022-01-26T14:14:47.705838675Z,projects/my-project/traces/94a63d9f2e241b7bf1309eda7adf8f08

UPDATE 2: I can pull all ~3000 documents to my local machine from the users collection using Firefoo in just a few seconds.

It seems the problem is isolated to the users collection itself, running inside the GCF.