'doSNOW and Foreach loop (R) on cluster?
I am using a cluster to run a foreach loop in parallel, using doSNOW. The loop works on my desktop, I receive this warning when running on the cluster
Execution halted
Error in unserialize(node$con) : error reading from connection
Calls: local ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
The loop is rather large, so I have just provided a very basic sample here (I do not believe the error is in the loop, as it works on the desktop).
library(sp)
library(raster)
library(fields)
library(tidyr)
library(dplyr)
library(sphereplot)
library(dismo)
library(doSNOW)
library(parallel)
cores <- (detectCores()/2)/2
print(cores)
cl <- makeCluster(cores, type = "SOCK", outfile = "")
registerDoSNOW(cl)
FossilClimCoV <- foreach(i = 0:5,.combine = "rbind",
.packages = paste(c("dplyr","dismo",
"sp","raster",
"fields",
"tidyr","sphereplot","doSNOW","parallel"
)))%dopar%{
print(i)
FossilTemp <- Fossils%>%dplyr::filter(Age == i)
if(nrow(FossilTemp)>0){
BULK removed for ease
return(FossilTemp1)
}
}
I'm not sure how to fix this error. I don't understand why it will not work on the cluster, but will on my desktop.
EDIT 1 I have now resolved this large error by changing from a doSNOW backend to doParallel.
library(doParallel)
registerDoParallel(cores=3)
*foreach loop*
However, I now have a new error:
Calls: %dopar% -> <Anonymous>
Execution halted
If I change the errorhandling to "remove" the foreach loop will always return an empty vector.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
