'What is the fastest way to upload files (i.e. least CPU cycles / file)?
What is the fastest way to upload files (i.e. least CPU cycles / file)? Say I have 1 million files on disk and I want to upload all of them to an HTTPS API with unlimited capacity. I want to fully saturate a fast cloud container network.
Code is running in a Docker container with 2-8 CPUs and 3 GB+ ram.
Node.js streams is pretty good but is CPU-bound w/ SSDs.
The root problem seems to that uploading a file means copying a file out of the kernel to userland and then back to the kernel and then to networking — which is CPU/memory heavy.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|