Parallel SEM Forests with the future Package

Andreas M. Brandmaier

2025-11-26

This vignette shows how to run SEM forests in parallel using the future package. The future package provides a standardized way of evaluating R expressions asynchronously using the computing resources available to the user – whether it is a single machine with multiple cores or a computing cluster.

We build on the latent growth curve model from the Getting Started vignette and use a future::plan() to distribute trees across the available CPU cores of a local machine.

Packages

library(semtree)
library(OpenMx)
library(future)

Simulate the latent growth curve data

We reuse the linear latent growth curve example: each individual has five repeated measures (X1 to X5) and one dichotomous predictor (P1) that influences the mean slope. Setting a seed keeps the simulation reproducible.

set.seed(23)
N <- 1000
M <- 5
icept <- rnorm(N, 10, sd = 4)
slope <- rnorm(N, 3, sd = 1.2)
p1 <- sample(c(0, 1), size = N, replace = TRUE)
loadings <- 0:4
x <-
  (slope + p1 * 5) %*% t(loadings) +
  matrix(rep(icept, each = M), byrow = TRUE, ncol = M) +
  rnorm(N * M, sd = .08)
growth.data <- data.frame(x, factor(p1), factor(sample(c(0,1),N,TRUE)))
names(growth.data) <- c(paste0("X", 1:M), "P1","Noise")

Specify and fit the SEM

The model is identical to the one used in the introductory vignette: a two-factor latent growth curve with fixed factor loadings for the intercept and slope.

manifests <- names(growth.data)[1:5]
growthCurveModel <- mxModel("Linear Growth Curve Model Path Specification",
    type="RAM",
       manifestVars=manifests,
    latentVars=c("intercept","slope"),
    mxData(growth.data, type="raw"),
    # residual variances
    mxPath(
        from=manifests,
        arrows=2,
        free=TRUE,
        values = c(.1, .1, .1, .1, .1),
        labels=c("residual","residual","residual","residual","residual")
    ),
    # latent variances and covariance
    mxPath(
        from=c("intercept","slope"),
        arrows=2,
        connect="unique.pairs",
        free=TRUE,
        values=c(2, 0, 1),
        labels=c("vari", "cov", "vars")
    ),
    # intercept loadings
    mxPath(
        from="intercept",
        to=manifests,
        arrows=1,
        free=FALSE,
        values=c(1, 1, 1, 1, 1)
    ),
    # slope loadings
    mxPath(
        from="slope",
        to=manifests,
        arrows=1,
        free=FALSE,
        values=c(0, 1, 2, 3, 4)
    ),
    # manifest means
    mxPath(
        from="one",
        to=manifests,
        arrows=1,
        free=FALSE,
        values=c(0, 0, 0, 0, 0)
    ),
    # latent means
    mxPath(
        from="one",
        to=c("intercept", "slope"),
        arrows=1,
        free=TRUE,
        values=c(1, 1),
        labels=c("meani", "means")
    )
) # close model

growthCurveModel <- mxRun(growthCurveModel)
#> Running Linear Growth Curve Model Path Specification with 6 parameters
#> Warning: In model 'Linear Growth Curve Model Path Specification' Optimizer
#> returned a non-zero status code 5. The Hessian at the solution does not appear
#> to be convex. See ?mxCheckIdentification for possible diagnosis (Mx status
#> RED).

Enable parallelism with future

semtree::semforest() uses the future.apply infrastructure internally, so registering a parallel plan automatically parallelizes tree construction. Here we choose a conservative number of workers to keep vignette builds light-weight, that is, a maximum of two workers (if available), and we restore the previous plan when we’re done. Please see the documentation of future for other plans, such as sequential for non-parallel execution or cluster for using a computing cluster.

old_plan <- future::plan()
on.exit(future::plan(old_plan), add = TRUE)

workers <- max(1, min(2, future::availableCores()))
future::plan(future::multisession, workers = workers)

Grow the SEM forest in parallel

With the plan in place, we can grow a forest. We keep the number of trees modest for the sake of a fast example; in practice, increase num.trees (e.g., 200–500) for stable variable importance estimates.

#> Beginning initial fit attemptFit attempt 0, fit=4841.4129502689, new current best! (was 4841.68119221572)Beginning fit attempt 1 of at maximum 10 extra tries                        Fit attempt 1, fit=4841.41295026589, new current best! (was 4841.4129502689)                                                                            Beginning initial fit attemptFit attempt 0, fit=2386.6443068647, new current best! (was 2390.91864907122)                                                                            Beginning initial fit attemptFit attempt 0, fit=933.414331788414, new current best! (was 1188.6255269588)                                                                            Beginning initial fit attemptFit attempt 0, fit=927.622724089175, new current best! (was 1198.01877989393)                                                                             Beginning initial fit attemptFit attempt 0, fit=2446.20581882165, new current best! (was 2450.49430121033)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=2446.20581882071, new current best! (was 2446.20581882165)                                                                             Beginning initial fit attemptFit attempt 0, fit=1118.82057976983, new current best! (was 1388.55496808656)                                                                             Beginning initial fit attemptFit attempt 0, fit=783.95193259808, new current best! (was 1057.65085074511)                                                                            ✔ Tree construction finished [took 3s].
#> Beginning initial fit attemptFit attempt 0, fit=4857.99683300402, new current best! (was 4859.21177661586)                                                                             Beginning initial fit attemptFit attempt 0, fit=2043.61820642061, new current best! (was 2554.34370851868)                                                                             Beginning initial fit attemptFit attempt 0, fit=935.892083913097, new current best! (was 937.32198379464)                                                                            Beginning initial fit attemptFit attempt 0, fit=1105.14584297666, new current best! (was 1106.29622260119)                                                                             Beginning initial fit attemptFit attempt 0, fit=1780.11329988171, new current best! (was 2303.65312449289)                                                                             Beginning initial fit attemptFit attempt 0, fit=970.348732415183, new current best! (was 973.859971187111)                                                                             Beginning initial fit attemptFit attempt 0, fit=801.538036221115, new current best! (was 806.253328697046)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4837.2179115598, new current best! (was 4840.61442884144)                                                                            Beginning initial fit attemptFit attempt 0, fit=1901.15484935501, new current best! (was 2434.93404806847)                                                                             Beginning initial fit attemptFit attempt 0, fit=1879.95371736256, new current best! (was 2402.28386356542)                                                                             ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4877.26068018816, new current best! (was 4878.8770102035)Beginning fit attempt 1 of at maximum 10 extra tries                        Fit attempt 1, fit=4877.26068017697, new current best! (was 4877.26068018816)                                                                             Beginning initial fit attemptFit attempt 0, fit=2455.74880690412, new current best! (was 2457.59307648514)                                                                             Beginning initial fit attemptFit attempt 0, fit=2417.87069004512, new current best! (was 2419.66760368406)                                                                             ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4797.07462072364, new current best! (was 4798.39076237284)                                                                             Beginning initial fit attemptFit attempt 0, fit=1944.82725881168, new current best! (was 2499.09633094297)                                                                             Beginning initial fit attemptFit attempt 0, fit=903.253141522375, new current best! (was 908.851229897821)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=903.253141519611, new current best! (was 903.253141522375)                                                                             Beginning initial fit attemptFit attempt 0, fit=1030.61537100073, new current best! (was 1035.97602885435)                                                                             Beginning initial fit attemptFit attempt 0, fit=1750.76043665217, new current best! (was 2297.97828985628)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4843.28555041772, new current best! (was 4846.13143455715)                                                                             Beginning initial fit attemptFit attempt 0, fit=2473.97954068927, new current best! (was 2477.95111253351)                                                                             Beginning initial fit attemptFit attempt 0, fit=907.265546877468, new current best! (was 1178.79043644132)                                                                             Beginning initial fit attemptFit attempt 0, fit=1021.28112300835, new current best! (was 1295.18910425902)                                                                             Beginning initial fit attemptFit attempt 0, fit=2360.83101030083, new current best! (was 2365.33443787792)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=2360.83101029861, new current best! (was 2360.83101030083)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4740.87356224327, new current best! (was 4742.94988525661)                                                                             Beginning initial fit attemptFit attempt 0, fit=1987.16384152455, new current best! (was 2501.00102936027)                                                                             Beginning initial fit attemptFit attempt 0, fit=972.33115894756, new current best! (was 975.117265064975)Beginning fit attempt 1 of at maximum 10 extra tries                        Fit attempt 1, fit=972.331158946444, new current best! (was 972.33115894756)                                                                            Beginning initial fit attemptFit attempt 0, fit=1009.09291990782, new current best! (was 1012.0465763473)                                                                            Beginning initial fit attemptFit attempt 0, fit=1723.7612009215, new current best! (was 2239.87253285012)                                                                            Beginning initial fit attemptFit attempt 0, fit=967.232366318714, new current best! (was 972.810893807774)                                                                             Beginning initial fit attemptFit attempt 0, fit=743.215563275571, new current best! (was 750.950307096374)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=743.215563273839, new current best! (was 743.215563275571)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4885.05897783269, new current best! (was 4886.26235257213)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=4885.05897782651, new current best! (was 4885.05897783269)                                                                             Beginning initial fit attemptFit attempt 0, fit=2464.32900562986, new current best! (was 2470.76759441273)                                                                             Beginning initial fit attemptFit attempt 0, fit=2407.20648366729, new current best! (was 2414.29138332742)                                                                             ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4848.28071768685, new current best! (was 4850.81501043059)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=4848.2807176816, new current best! (was 4848.28071768685)                                                                            Beginning initial fit attemptFit attempt 0, fit=2469.22523372227, new current best! (was 2476.615680638)                                                                           Beginning initial fit attemptFit attempt 0, fit=2363.81170620688, new current best! (was 2371.66503716345)                                                                             Beginning initial fit attemptFit attempt 0, fit=1013.07915547846, new current best! (was 1296.65721166409)                                                                             Beginning initial fit attemptFit attempt 0, fit=814.548108614579, new current best! (was 1067.15449453749)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4832.50378626754, new current best! (was 4834.60301704753)                                                                             Beginning initial fit attemptFit attempt 0, fit=2002.03422991225, new current best! (was 2550.55216666672)                                                                             Beginning initial fit attemptFit attempt 0, fit=953.157968486568, new current best! (was 955.205678494332)                                                                             Beginning initial fit attemptFit attempt 0, fit=1044.98518133996, new current best! (was 1046.82855138445)                                                                             Beginning initial fit attemptFit attempt 0, fit=1728.81097960556, new current best! (was 2281.95161964954)                                                                             Beginning initial fit attemptFit attempt 0, fit=922.111610046701, new current best! (was 924.687157207946)                                                                             Beginning initial fit attemptFit attempt 0, fit=800.942736507304, new current best! (was 804.123822402314)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4838.30329089626, new current best! (was 4839.69453713705)                                                                             Beginning initial fit attemptFit attempt 0, fit=1931.61992456887, new current best! (was 2443.97255945332)                                                                             Beginning initial fit attemptFit attempt 0, fit=982.174206169178, new current best! (was 986.126443669142)                                                                             Beginning initial fit attemptFit attempt 0, fit=941.167053924839, new current best! (was 945.493480917554)                                                                             Beginning initial fit attemptFit attempt 0, fit=1864.39482482846, new current best! (was 2394.33073141645)                                                                             Beginning initial fit attemptFit attempt 0, fit=1020.25924383151, new current best! (was 1024.06550387261)                                                                             Beginning initial fit attemptFit attempt 0, fit=835.944802179873, new current best! (was 840.329320950811)                                                                             ✔ Tree construction finished [took 3s].
#> Beginning initial fit attemptFit attempt 0, fit=4815.73245572566, new current best! (was 4816.10818458514)                                                                             Beginning initial fit attemptFit attempt 0, fit=2475.22199878845, new current best! (was 2479.99791831093)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=2475.22199878732, new current best! (was 2475.22199878845)                                                                             Beginning initial fit attemptFit attempt 0, fit=2330.2738869442, new current best! (was 2335.73453739355)                                                                            ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4766.80701495913, new current best! (was 4767.8832438517)                                                                            Beginning initial fit attemptFit attempt 0, fit=1917.1036870605, new current best! (was 2476.50135952985)                                                                            Beginning initial fit attemptFit attempt 0, fit=1762.44980486545, new current best! (was 2290.30565539497)                                                                             Beginning initial fit attemptFit attempt 0, fit=958.743305635649, new current best! (was 963.3258765412)                                                                           Beginning initial fit attemptFit attempt 0, fit=792.862686353358, new current best! (was 799.123928326935)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4825.57164723138, new current best! (was 4826.61394093424)                                                                             Beginning initial fit attemptFit attempt 0, fit=1945.72610223376, new current best! (was 2503.19585707468)                                                                             Beginning initial fit attemptFit attempt 0, fit=1794.17783811362, new current best! (was 2322.37579001318)                                                                             ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4759.79018090796, new current best! (was 4762.95192064227)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=4759.790180905, new current best! (was 4759.79018090796)                                                                           Beginning initial fit attemptFit attempt 0, fit=2021.83680962457, new current best! (was 2527.55274066178)                                                                             Beginning initial fit attemptFit attempt 0, fit=1706.77845806714, new current best! (was 2232.23744027947)                                                                             Beginning initial fit attemptFit attempt 0, fit=853.939987471505, new current best! (was 857.648547669673)                                                                             Beginning initial fit attemptFit attempt 0, fit=844.817586128653, new current best! (was 849.129910411521)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4792.56044371011, new current best! (was 4793.25942269636)                                                                             Beginning initial fit attemptFit attempt 0, fit=2477.12921152579, new current best! (was 2484.67129590925)                                                                             Beginning initial fit attemptFit attempt 0, fit=1038.04233893742, new current best! (was 1308.39511713537)                                                                             Beginning initial fit attemptFit attempt 0, fit=896.806686208458, new current best! (was 1168.73409439153)                                                                             Beginning initial fit attemptFit attempt 0, fit=2299.84121281263, new current best! (was 2307.8891477889)Beginning fit attempt 1 of at maximum 10 extra tries                        Fit attempt 1, fit=2299.84121281073, new current best! (was 2299.84121281263)                                                                             Beginning initial fit attemptFit attempt 0, fit=1018.91678628493, new current best! (was 1283.93181386932)                                                                             Beginning initial fit attemptFit attempt 0, fit=751.851984963089, new current best! (was 1015.90939896056)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4816.01457604909, new current best! (was 4818.703991084)Beginning fit attempt 1 of at maximum 10 extra tries                       Fit attempt 1, fit=4816.01457604293, new current best! (was 4816.01457604909)                                                                             Beginning initial fit attemptFit attempt 0, fit=1972.26018811376, new current best! (was 2491.30644425653)                                                                             Beginning initial fit attemptFit attempt 0, fit=974.964914632091, new current best! (was 979.261188550006)                                                                             Beginning initial fit attemptFit attempt 0, fit=987.986539656616, new current best! (was 992.998999553467)                                                                             Beginning initial fit attemptFit attempt 0, fit=1790.38190255829, new current best! (was 2324.70813180017)                                                                             Beginning initial fit attemptFit attempt 0, fit=906.139129240533, new current best! (was 910.044322605566)                                                                             Beginning initial fit attemptFit attempt 0, fit=875.877495264485, new current best! (was 880.337579975463)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4848.46694579256, new current best! (was 4849.91198197862)                                                                             Beginning initial fit attemptFit attempt 0, fit=2422.48994008734, new current best! (was 2427.58654115739)                                                                             Beginning initial fit attemptFit attempt 0, fit=949.60754342489, new current best! (was 1193.01693914372)                                                                            Beginning initial fit attemptFit attempt 0, fit=972.00381451372, new current best! (was 1229.47300089882)                                                                            Beginning initial fit attemptFit attempt 0, fit=2415.065507095, new current best! (was 2420.88040474833)Beginning fit attempt 1 of at maximum 10 extra tries                       Fit attempt 1, fit=2415.06550709467, new current best! (was 2415.065507095)                                                                           ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4751.03354812164, new current best! (was 4755.23806730752)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=4751.03354811434, new current best! (was 4751.03354812164)                                                                             Beginning initial fit attemptFit attempt 0, fit=2315.19763865933, new current best! (was 2318.30117448961)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=2315.19763865811, new current best! (was 2315.19763865933)                                                                             Beginning initial fit attemptFit attempt 0, fit=2429.61196434522, new current best! (was 2432.7323737125)                                                                            Beginning initial fit attemptFit attempt 0, fit=1072.55608862605, new current best! (was 1346.23068189987)                                                                             Beginning initial fit attemptFit attempt 0, fit=808.259466328533, new current best! (was 1083.38128246236)Beginning fit attempt 1 of at maximum 10 extra tries                         Fit attempt 1, fit=808.259466328151, new current best! (was 808.259466328533)Beginning fit attempt 2 of at maximum 10 extra tries                         Fit attempt 2, fit=808.259466327431, new current best! (was 808.259466328151)                                                                             ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4826.1893879509, new current best! (was 4829.00149100655)                                                                            Beginning initial fit attemptFit attempt 0, fit=2022.31943600189, new current best! (was 2524.97809631185)                                                                             Beginning initial fit attemptFit attempt 0, fit=1764.07469136654, new current best! (was 2301.2112917175)                                                                            ✔ Tree construction finished [took less than a second].
#> ✔ Forest completed [took 22s]

Inspect variable importance

Because the latent growth data only include the grouping predictor P1, importance results confirm that this variable drives heterogeneity in the model parameters. Parallel evaluation speeds up the permutation step when multiple predictors are present.

vim <- varimp(forest)
print(vim, sort.values = TRUE)
#> Variable Importance
#>       Noise          P1 
#>    8.063749 2873.661157
plot(vim)

After the forest is built, future::plan(old_plan) (triggered via on.exit) returns the session to its prior parallel configuration.