'Assigning different types of tasks to the same `foreach` loop

Many are familiar with foreach() to assign a loop across many cores in parallel using %dopar%. However, in R how do you send a single job request for a variety of different types of tasks for distribution across the cores?

This is my minimal working example/solution using eval() and expression() but am open to alternatives if this is sub-optimal.

# R 4.1.3 on Windows 11
library(doParallel)
ncores = 3
cl <- makePSOCKcluster(ncores)
registerDoParallel(cl)
stopifnot(ncores==getDoParWorkers()) # check worker number assigned

# use commandsFun() to list the code commands you want each core to do
commandsFun <- function(k) {
  # the list y is stored in all cores but only the k^{th} element gets evaluated
  y = list(expression(1+1),
           expression(print("Yo!")),
           expression(Sys.sleep(3)))
  ans = eval(y[[k]])
  return(ans)
}

ans = foreach(i = 1:3, .combine = rbind) %dopar%
  commandsFun(k = i) # only the particular list element is run by each core

print(ans) # prints when all cores complete. ans is 2 rows, as Sys.sleep() returns NULL
stopCluster(cl)


Solution 1:[1]

(Disclaimer: I'm the author) This is where the future ecosystem shines:

library(future)

## Here, parallelize on local machine, but there
## are plenty of other backends to choose from
plan(multisession, workers = 3)

fs <- list()
fs[[1]] <- future(1+1)
fs[[2]] <- future(print("Yo!"))
fs[[3]] <- future(Sys.sleep(3))

vs <- value(fs)
#> [1] "Yo!"
str(vs)
#> List of 3
#>  $ : num 2
#>  $ : chr "Yo!"
#>  $ : NULL

If you want it to look even more like plain R code, you can use the future assignment operator %<-% with list environments;

vs <- listenv::listenv()
vs[[1]] %<-% { 1+1 }
vs[[2]] %<-% { print("Yo!") }
vs[[3]] %<-% { Sys.sleep(3) }
vs <- as.list(vs)
#> [1] "Yo!"
str(vs)
#> List of 3
#>  $ : num 2
#>  $ : chr "Yo!"
#>  $ : NULL

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 HenrikB