How to refactor recursive method to use RxJs Observables for pagination

Question

I'm new to RxJs

I have a method that paginates through all pages available on a url and stores the json response in an array, using a recursive approach.

Is there a simple way to achieve the same outcome using RxJs observables without recursion?

    ...

    export interface Contestant {
      rank: number;
      finish_time: number;
      score: number;
    }

    export interface Contest {
      url: URL;
      contestNumber: number;
      lastPage: number;
      /**
       * Number of contestants.
       * Gotten as lastPage x 25.
       */
      totalContestants: number;
    }

    export interface Response {
      total_rank: Contestant[];
    }
    ...

  /**
   * Scrape all useful entries from a contest
   * and return an array of entries
   **/
  async scrapeContestData(contest: Contest): Promise<Contestant[]> {
    /**
     * This recursive approach assumes 500bytes per stack call
     * and a lastPage typically <= 1000
     * for a rough estimation of 500kb of stack space needed
     *
     * The default node stack size is 984kb so for a typical
     * contest this should be fine. If not, needs refactoring.
     */
    if (contest.lastPage === 0) return [];
    contest.url.searchParams.set('pagination', contest.lastPage.toString());

    const res = await firstValueFrom(
      this.httpService.get<Response>(contest.url.toString()),
    );

    contest.lastPage--;

    return res.data.total_rank.concat(await this.scrapeContestData(contest));
  }

Answer 1

I might try something like this:

Most recursion in RxJS can be nicely handled by expand. In this case, however, I've removed the recursion in favor of a simple range, since your algorithm isn't really making choices between recursive calls.

I haven't tested this, but with a bit of tinkering it should work.

scrapeContestData(contest: Contest): Observable<Contestant[]> {

  return range(0, contest.lastPage - 1).pipe(
    map(v => contest.lastPage - v),
    concatMap(page => {
      const url = contest.url;
      url.searchParams.set("pagination", page.toString());
      return this.httpService.get<Response>(url.toString()).pipe(take(1))
    ),
    toArray(),
    map((datum: Contestant[][]) => datum.flat())
  );

}

A quick aside:

Arrays and Observables are abstractly very similar things. One is a list across space (in memory) and the other is a list across time (in CPU cycles).

All that to say, the abstract collection-type functions that exist on arrays mostly all exist for Observables as well.

We could re-write this without Array#flat like this:

scrapeContestData(contest: Contest): Observable<Contestant[]> {

  return range(0, contest.lastPage - 1).pipe(
    map(v => contest.lastPage - v),
    concatMap(page => {
      const url = contest.url;
      url.searchParams.set("pagination", page.toString());
      return this.httpService.get<Response>(url.toString()).pipe(take(1))
    ),
    concatMap(v => v),
    toArray()
  );

}