Return observable while doing task - Firebase Functions

Question

I have a function in my Firebase Cloud Functions that needs to run a scraping task. While the function is scraping the public website I need to return to the consumer a progress feedback. (eg. "Now it's at 10%", "Now it's at 30%", etc...).

I tought using Observables but I get weird results because the scraping function is async but it should return immediately the observable.

This is the simplified code of my Firebase Functions Backend:

exports.scrapeFunction = functions.region('europe-west2').https.onCall((data) => {
 const res$ = new BehaviorSubject<Response>({
  completion: 0,
  completionMsg: 'One moment...',
  error: undefined,
  user: undefined,
 });
 scrape(data.id, res$);
 return res$.asObservable();
});

const scrape = async (id: number, res$: BehaviorSubject<Response>) => {

 const browser = await puppeteer.launch({ headless: true });
 const page = await browser.newPage();

 res$.next({
  completion: 10,
  completionMsg: 'I\'m loading the browser...'
 })

 await page.goto('https://example.com');

 res$.next({
  completion: 20,
  completionMsg: 'Page opened...'
 })

 /* Other scraping async functions... */

}

On my frontend, instead I've a code similar to this:

response$ = Observable<Response> | undefined;

ngOnInit(): void {
 const scrapeFunction= getFunctions(undefined, 'europe-west2');
 const verifyClient = httpsCallable(functions, 'scrapeFunction');
 const { data } = await scrapeFunction({ id: 0001 });
 console.log('Data is', data);
 this.response$ = (data as any).source as Observable<Response>;
 this.response$.subscribe((res) => console.log(res));
 return;
}

Logging in the console I get:

Data is 
 {source: {...}}

 source:
  closed: false
  currentObservers: null
  hasError: false
  isStopped: false
  observers: []
  thrownError: null
  _value: {completion: 0, completionMsg: 'One moment...'}
  [[Prototype]]: Object
 [[Prototype]]: Object

I also get the following error because the function is not returning an Observable as I know, but an object with the first key "source".

ERROR Error: Uncaught (in promise): TypeError: _this.response$.subscribe is not a function
TypeError: _this.response$.subscribe is not a function
    at dashboard.component.ts:35:29
    at Generator.next (<anonymous>)
    at asyncGeneratorStep (asyncToGenerator.js:3:1)
    at _next (asyncToGenerator.js:25:1)
    at ZoneDelegate.invoke (zone.js:372:1)
    at Object.onInvoke (core.js:28692:1)
    at ZoneDelegate.invoke (zone.js:371:1)
    at Zone.run (zone.js:134:1)
    at zone.js:1276:1
    at ZoneDelegate.invokeTask (zone.js:406:1)
    at resolvePromise (zone.js:1213:1)
    at zone.js:1120:1
    at asyncGeneratorStep (asyncToGenerator.js:6:1)
    at _next (asyncToGenerator.js:25:1)
    at ZoneDelegate.invoke (zone.js:372:1)
    at Object.onInvoke (core.js:28692:1)
    at ZoneDelegate.invoke (zone.js:371:1)
    at Zone.run (zone.js:134:1)
    at zone.js:1276:1
    at ZoneDelegate.invokeTask (zone.js:406:1)

Can someone understand this behavior?

[UPDATE 1]

I edited my index.ts file in Firebase Functions to be like this:

const res$ = new BehaviorSubject<Response>({
 completion: 0,
 completionMsg: 'One moment...',
});
exports.verifyClient = functions.https.onCall((data) => {
 scrapeCheck(data.id);
 return verifyClientRes$.asObservable();
});

const scrapeCheck = async (id: number) => {
 await scrape(id, res$);
 res$.complete();
 return;
};

/* The scrape function is actually on an external file */
const scrape = (/* ... */) => {
 /* ... */
}

** But i still got the same error on the console.

[UPDATE 2]

I created a StackBlitz project. You can find it here: https://stackblitz.com/edit/wherescrypto?file=functions/src/index.ts

The function that is causing this error is in "functions/src/index.ts" and it's called "verifyClient"

Answer 1

Reviewing some documentation that could help to fix the issue you have, I was wondering if you already considered trying it with a realtime database or firestore.

You'll have to push your updates somewhere, the client app can see them that way. This solution was mentioned in a comment in this stackoverflow question.

Answer 2

Putting scrape in setTimeout will do the trick here. so that scrape is invoked after observable is returned.

exports.scrapeFunction = functions.region('europe-west2').https.onCall((data) => {
 const res$ = new BehaviorSubject<Response>({
  completion: 0,
  completionMsg: 'One moment...',
  error: undefined,
  user: undefined,
 });
 setTimeout(() => scrape(data.id, res$));
 return res$.asObservable();
});