Firestore doesn't retrieve more than 600 items

Question

I am trying to load some data from Firestore in a Flutter app. When I load upto 600 items , it works fine. but even if I try retrieving 700I don't get anything.

Here is the code:

Stream<List<Booking>> listenToWeekEntries()  {
    return FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .limitToLast(1000).snapshots().map((event) => List.from(event.docs.map((e) => Booking.fromSnapshot(e))));
          //

  }

I only have 1 collection 'bookings' and in it I have about 50000 documents(bookings) (growing at about 120/day). Each booking has about 13 fields. I am trying to create charts representing the counts on different days, months and years. What's the best way to do this?

I have tried looking around for a solution but have had not luck.

I have also tried the following :

late DocumentSnapshot _lastDocument;
List<Booking> entries = List.empty(growable: true);

Constructor(){
    getEntries();
}

void getEntries() async {
    var q = await FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .limitToLast(50).get();
    entries.addAll(q.docs.map((e) => Booking.fromSnapshot(e)));
    notifyListeners();
    print('${entries.length}');
    if(entries.length < 2000){
      _lastDocument = q.docs[q.docs.length - 1];
      getMoreEntries();
    }
  }

void getMoreEntries() async {
    print('WEEK_DEBUG | Getting more entries after $_lastDocument.');
    var q = await FirebaseFirestore.instance
        .collection('bookings')
        .orderBy('entryTimestamp')
        .where('entryTimestamp',isGreaterThanOrEqualTo: getWeekFilter()[0]/1000, isLessThanOrEqualTo: getWeekFilter()[1]/1000)
        .startAfterDocument(_lastDocument)
        .limitToLast(50).get();

    entries.addAll(q.docs.map((e) => Booking.fromSnapshot(e)));
    notifyListeners();
    _lastDocument = q.docs[q.docs.length - 1];
    if(entries.length < 2000){
      getMoreEntries();
    }
  }

 List getWeekFilter(){
    var now = DateTime.now();
    var firstDay = now.subtract(const Duration(days: 7));
    var lastDay = now.add(const Duration(days: 7));
    var from = DateTime(firstDay.year, firstDay.month, firstDay.day).millisecondsSinceEpoch;
    var to = DateTime(lastDay.year, lastDay.month, lastDay.day).millisecondsSinceEpoch;
    return [from, to];

  }

With this, getMoreEntries() does get called however the total length of entries remains 50.

Any help is greatly appreciated.

Answer 1

I couldn't tell you why you can't retrieve more than 600 docs per query. Yet I'm sure you can sense why doing such a huge query just to count docs is not a good idea.
I'd advise you to create a simple counter subcollection, of which each doc id could be the day they were counting bookings creation for. You just have to increase the day's counter by one after each booking, each doc summing the daily booking count (and incidentally, the id of each booking created that day if there are only a handful of those, and you'd need to retrieve them). After that you just have to query every day's counter and sum them up in the front-end. You could also start a new subcollection counter every year to cap the number of booking day docs to sum up to 365/6, and put the final sum of previous years count in a field of the root counter, so you can still access previous years count without querying always more docs.
To get a view of how this looks in practice, you should look into distributed counters in the docs.