'Looping through download.file() in R causes timeout
I'm trying to create a pipeline for downloading multiple files, unzipping and merging them in the end but I get a timeout error at the step of the download using base R function download.file.
Below is my code snippet
files %>%
paste0(url, .) %>%
map(download.file(., destfile = gsub("^.*/", "", .)))
First comes the base url and the file name is extracted for later use. The above code returns the following
Warning messages:
1: In download.file(., destfile = gsub("^.*/", "", .)) :
URL 'https://example.com/files/file1.csv.gz': Timeout of 60 seconds was reached
2: In download.file(., destfile = gsub("^.*/", "", .)) :
URL 'https://example.com/files/file2.csv.gz': Timeout of 60 seconds was reached
3: In download.file(., destfile = gsub("^.*/", "", .)) :
some files were not downloaded
From what I understand the problem seems to be that a connection is established to download ALL the files at the same time. Beyond that, any help is appreciated.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
