'memory failure when largely iteratating with foreach %dopar%

registerDoParallel(128)

b <- matrix(10.0003, ncol = 10, nrow = 12251352)

res1 <- foreach(i = 1:nrow(b), .combine = "rbind") %dopar% (1:10)

When run the above code, I got the memory problem.

Error in mcfork() : unable to fork, possible reason: Cannot allocate memory

How to fully make use of the 128 cores without crashed memory?

Can one of the following be the effective way?

  1. add for as outer loop
  2. add for as inner loop
  3. add foreach %do% as outer loop
  4. add foreach %do% as inner loop
  5. add foreach %:% as outer loop
  6. add foreach %:% as inner loop

Another question: will foreach copy its output variable across all cores?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source