'Download file from internet and send it to S3 as stream

I am trying to download some PDFs from internet into my Amazon S3 bucket, so far i download the files on my server and then upload them from my server to S3 Bucket but i was curious if i can upload them while downloading them as stream.

private async download(url: string, path: string): Promise<void> {
    const response = await fetch(url);
    const fileStream = createWriteStream(path);
    await new Promise((resolve, reject) => {
      response.body.pipe(fileStream);
      response.body.on('error', reject);
      fileStream.on('finish', resolve);
    });
  }

and this is my upload file after i downloaded it

public async upload(path: string, name: string): Promise<string> {
    const url = 'documents/${name}.pdf';
    const params = {
      Body: createReadStream(path),
      Bucket: AWS_S3_BUCKET_NAME,
      Key: url
    }

    const data = await s3.putObject(params).promise().then(data => { console.log(data); return url; }, err => { console.log(err); return err; });

    return data;
  }

I am looking for a way to merge these 2 functions into one and return the S3 bucket reply after finished or throw an error if download or upload gave an error.

Also i wanted to ask if it is possible to call this function multiple times in parallel and if it's possible, how many times is safe to not break the server.

Thank you in advance, Daniel!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source