How to upload file from URL to S3 bucket (aws-sdk-js v3)?

Question

I've tried the below code with no luck, which showing this error message:

(node:88634) UnhandledPromiseRejectionWarning: NotImplemented: A header you provided implies functionality that is not implemented

import { fromIni } from '@aws-sdk/credential-provider-ini'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import https from 'https'
import { Readable } from 'stream'
import awsConfig from './aws-exports'

const s3Client = new S3Client({
  credentials: fromIni({ profile: 'default' }),
  region: 'ap-northeast-1',
})

async function getFileFromUrl(url: string): Promise<Readable> {
  return new Promise((resolve) => {
    https.get(url, (response) => {
      resolve(response)
    })
  })
}

async function upload(file: Readable) {
  const uploadCommand = new PutObjectCommand({
    Bucket: awsConfig.aws_user_files_s3_bucket,
    Key: 'test.jpg',
    Body: file,
    ACL: 'public-read',
  })

  await s3Client.send(uploadCommand)
}

async function migrate() {
  const file = await getFileFromUrl(
    'https://example.com/logo.png'
  )
  await upload(file)
  console.log('done')
}

migrate()

I could confirm that it's ok if I change the Body to a string... Does anyone know how to do this correctly? Thanks!

Answer 1

The problem here is that your getFileFromUrl function does not work, and AWS doesn't know how to handle the object you are handing it. You need to wait for the data event from https, like this:

async function getFileFromUrl (url) {
  return new Promise((resolve) => {
    https.get(url, (response) => {
      response.on('data', (d) => {
        resolve(d)
      })
    })
  })
}