nodejs Transfer uploaded file directly to S3 without double waiting time

Question

Hi i am developing REST api's for mobile application in which user uploads images to app server then server uploads images to amazon s3 and update the database.

The problem:

1.User wants to upload 10MB file    It takes 10 seconds to travel from user's device to my server
2.After server has received last bit, it starts uploading to S3. Another 10 seconds.
3.After 20 seconds in total, user is presented with hyperlink.

I think these 20 seconds can be reduced by half, to 12 or so. If Nodejs were smart enough to start uploading to S3 not after the last bit arrived, but after the first bit arrived from user - don't wait for all file to be uploaded, but start streaming on the fly.

Is this possible?

Answer 1

you should send pre-signed S3 URLs to the users so users can upload to S3 directly, not going through your node server at all.

See here: http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html

Answer 2

I am not sure why you need to upload your file first to EC2 and then to S3 when your API could upload the file directly to S3 instead.

Also, node.js is smart enough to allow you to start processing your data upon arrival (see on('file') event in busboy for example).

In S3, you could use the managed uploader to upload streams of unknown size using a configurable amount of concurrency to perform multipart uploads where possible.

The managed uploader allows for easy and efficient uploading of buffers, blobs, or streams, using a configurable amount of concurrency to perform multipart uploads where possible. This abstraction also enables uploading streams of unknown size due to the use of multipart uploads.