'Assigning different types of tasks to the same `foreach` loop
Many are familiar with foreach()
to assign a loop across many cores in parallel using %dopar%
. However, in R how do you send a single job request for a variety of different types of tasks for distribution across the cores?
This is my minimal working example/solution using eval()
and expression()
but am open to alternatives if this is sub-optimal.
# R 4.1.3 on Windows 11
library(doParallel)
ncores = 3
cl <- makePSOCKcluster(ncores)
registerDoParallel(cl)
stopifnot(ncores==getDoParWorkers()) # check worker number assigned
# use commandsFun() to list the code commands you want each core to do
commandsFun <- function(k) {
# the list y is stored in all cores but only the k^{th} element gets evaluated
y = list(expression(1+1),
expression(print("Yo!")),
expression(Sys.sleep(3)))
ans = eval(y[[k]])
return(ans)
}
ans = foreach(i = 1:3, .combine = rbind) %dopar%
commandsFun(k = i) # only the particular list element is run by each core
print(ans) # prints when all cores complete. ans is 2 rows, as Sys.sleep() returns NULL
stopCluster(cl)
Solution 1:[1]
(Disclaimer: I'm the author) This is where the future ecosystem shines:
library(future)
## Here, parallelize on local machine, but there
## are plenty of other backends to choose from
plan(multisession, workers = 3)
fs <- list()
fs[[1]] <- future(1+1)
fs[[2]] <- future(print("Yo!"))
fs[[3]] <- future(Sys.sleep(3))
vs <- value(fs)
#> [1] "Yo!"
str(vs)
#> List of 3
#> $ : num 2
#> $ : chr "Yo!"
#> $ : NULL
If you want it to look even more like plain R code, you can use the future assignment operator %<-%
with list environments;
vs <- listenv::listenv()
vs[[1]] %<-% { 1+1 }
vs[[2]] %<-% { print("Yo!") }
vs[[3]] %<-% { Sys.sleep(3) }
vs <- as.list(vs)
#> [1] "Yo!"
str(vs)
#> List of 3
#> $ : num 2
#> $ : chr "Yo!"
#> $ : NULL
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | HenrikB |