Is there a workaround for the Firebase Query "IN" Limit to 10?

Question

I have a query for firebase that has an array of IDs that has a size > 10. Firebase has restrictions on the number of records to query in one session. Is there a way to query against more than 10 at a time?

[Unhandled promise rejection: FirebaseError: Invalid Query. 'in' filters support a maximum of 10 elements in the value array.]

https://cloud.google.com/firestore/docs/query-data/queries

  let query = config.db
    .collection(USER_COLLECTION_NAME)
    .where("id", "in", matchesIdArray);
  const users = await query.get();

(matchesIdArray.length needs to be unlimited)

Answer 1

Your only workaround is to make one query for each item in the array that you would normally use with a single "in" query. Or, batch the requests in the array.

  let query = config.db
    .collection(USER_COLLECTION_NAME)
    .where("id", "==", matchesIdArray[0]);
  const users = await query.get();

You'd have to use the above code in a loop over the matchesIdArray array, and merge the results after they are all done.

Answer 2

BEST METHOD

convert the list into a list that contains sublists of 10 items each. then for loop through that second list and query through firebase in each loop.

EXAMPLE:

List<String> phoneNumbers = ['+12313','+2323','1323','32323','32323','3232', '1232']; //CAN BE UPTO 100 or more

CONVERT THE PHONE NUMBERS TO A LIST OF SUBLIST OF 10 ITEMS EACH

List<List<String>> subList = [];
for (var i = 0; i < phoneNumbers.length; i += 10) {
    subList.add(
        phoneNumbers.sublist(i, i + 10> phoneNumbers.length ? phoneNumbers.length : i + 10));
}

NOW RUN FIREBASE QUERY

subList.forEach((element) {
      firestore
          .collection('Stories')
          .where('userPhone', whereIn: element)
      .get()
      .then((value) {
    
    value.docs.forEach((snapshot) {
      //handle the list
    });

  });

Answer 3

One common way to work around this limitation is to retrieve the items in batches, and then either process the results from each query either sequentially or in parallel.

Another common workaround is to model your data in a way that doesn't require for you to read dozens of documents to handle an individual request from your user. It's hard to say how you could reduce that number, but it often involves duplicating the data that you need from those separate documents into a single aggregated document.

An example of this: if you have a news site and need to show the latest 10 article headlines for each of 5 categories, you can do:

1. Do 50 separate reads, one for each document.
2. Create a document with the headlines for the latest 10 articles, and then only need to read 5 documents (one for each category).
3. Create a document with the latest 10 headlines for all 5 categories, and then only need to read that one document.

In these last two scenarios you're making the code that writes to the database more complex, as it needs to now write the aggregated documents too. But in return you have much less data to read, which reduces the cost, and improves the performance of your app. This type trade-off is very common when using NoSQL databases, which tend to be used in scenarios that have massively more reads than writes of their data.

For more data modeling advice, I recommend:

• NoSQL data modeling
• Getting to know Cloud Firestore

Answer 4

I faced the same problem and my solution using typescript was :

• for complete observables i used rxjs forkJoin

getPagesByIds(ids: string[]): Observable<Page[]> {
        ids = [...ids];
        if (ids.length) {
            let observables: Observable<Page[]>[] = [];
            while (ids.length) {
                let observable = this.afs.collection<Page>(PAGE, ref => ref.where('id', 'in', ids.splice(0, 10))).get().pipe(map(pages => pages.docs.map(page => page.data())))
                observables.push(observable)
            }
            return combineLatest(observables).pipe(map(pages => pages.flat(1)))
        }
        return of ([])
}

• for uncomplete observables i used rxjs combineLatest

getPagesByIds(ids: string[]): Observable<Page[]> {
        ids = [...ids];
        if (ids.length) {
            let observables: Observable<Page[]>[] = [];
            while (ids.length) {
                let observable = this.afs.collection<Page>(PAGE, ref => ref.where('id', 'in', ids.splice(0, 10))).get().pipe(map(pages => pages.docs.map(page => page.data())))
                observables.push(observable)
            }
            return combineLatest(observables).pipe(map(pages => pages.flat(1)))
        }
        return of ([])
}

Answer 5

As per the firebase documentation it support up to 10 ids only in the where field, For querying more than 10 elements either we need to query each document individually or split array to chunks of 10 ids.

For querying each items individually. Check the code below,

let usersPromise = [];

usersIds.map((id) => {
    usersPromise.push(firestore.collection("users").doc(id).get());
});
    
Promise.all(usersPromise).then((docs) => {
    const users = docs.map((doc) => doc.data());

    // do your operations with users list
});