Recursively move files from SFTP to S3 preserving structure

Question

I'm trying to recursively move files from an SFTP server to S3, possibly using boto3. I want to preserve the folder/file structure as well. I was looking to do it this way:

import pysftp

private_key = "/mnt/results/sftpkey"

srv = pysftp.Connection(host="server.com", username="user1", private_key=private_key)

srv.get_r("/mnt/folder", "./output_folder")

Then take those files and upload them to S3 using boto3. However, the folders and files on the server are numerous with deep levels and also large in size. So my machine ends up running out of memory and disk space. I was thinking of a script where I could download single files and upload single files and then delete and repeat.

I know this would take a long time to finish, but I can run this as a job without running out of space and not keep my machine open the entire time. Has anyone done something similar? Any help is appreciated!

Answer 1

You have to do it file-by-file.

Start with the recursive download code here:
Python pysftp get_r from Linux works fine on Linux but not on Windows

After each sftp.get, do S3 upload and remove the file.

Actually you can even copy the file from SFTP to S# without storing the file locally:
Transfer file from SFTP to S3 using Paramiko