Parallel & Asynchronous Computing

mirai — minimalist, fast async evaluation (m <- mirai(slow(1)); m[]), daemons, and parallel maps; powers purrr::in_parallel() and crew (worker launchers for targets, including cloud/HPC).

37.1 parallel (base)

library(parallel)
detectCores()
results <- mclapply(1:8, function(i) slow(i), mc.cores = 4)    # forking (Unix/macOS only)

cl <- makeCluster(4)                     # socket cluster (works on Windows)
clusterExport(cl, c("slow"))
clusterEvalQ(cl, library(dplyr))
results <- parLapply(cl, 1:8, function(i) slow(i))
parSapply(cl, 1:8, sqrt)
stopCluster(cl)

37.2 future ecosystem (unified, backend-agnostic)

library(future); library(future.apply); library(furrr)
plan(multisession, workers = 4)          # sequential, multisession, multicore, cluster, future.batchtools
future_lapply(1:8, slow)
future_map_dbl(1:8, slow, .options = furrr_options(seed = TRUE))   # parallel purrr
f <- future({ slow(1) }); value(f)       # explicit futures
plan(sequential)

37.3 foreach

library(foreach); library(doFuture)
registerDoFuture(); plan(multisession)
res <- foreach(i = 1:8, .combine = c) %dopar% { slow(i) }

37.4 mirai & crew

mirai — minimalist, fast async evaluation (m <- mirai(slow(1)); m[]), daemons, and parallel maps; powers purrr::in_parallel() and crew (worker launchers for targets, including cloud/HPC).

37.5 Async in Shiny & promises

promises + future/mirai keep Shiny apps responsive during long computations (ExtendedTask).

37.6 Tips

Parallelism has overhead — use it for coarse-grained tasks. Set RNG streams for reproducibility (future.seed = TRUE, RNGkind("L'Ecuyer-CMRG")). Watch memory: each worker holds its own copy of data. Use multithreaded BLAS (OpenBLAS, MKL) for linear algebra speedups.