Multiprocessing and single-core processing

Use Multiprocessing ?

The TAD package uses the future API to enhance reproducibility, and uniformize the way multiprocessing is done across plateforms. The future package is not mandatory, but it is recommanded (we used the HenrikBengtsson/[email protected] to test this package). You can install it with:

## if you want to install the version 1.33.2
remotes::install_github("HenrikBengtsson/[email protected]")

## if you want to install any version
remotes::install_github("HenrikBengtsson/future")

## if you want to install the latest version
remotes::install_github("HenrikBengtsson/future@*release")

For small datasets (< 16 variables/columns), we recommand using the future::sequencial strategy/plan.

Otherwise, you can use the future::multisession strategy/plan to fasten some of the processing in this package.

Run with Single Core processing

Load the future library and define the future::plan to future::multisession.


library(future)
plan(multisession)
weights <- data.frame(
  sp1 = c(1, 0),
  sp2 = c(2, 8),
  sp3 = c(0, 2)
)
aggregation_factor <- data.frame(
  plots = c("plot1", "plot2")
)

time_before <- proc.time()[[1]]
## This will run in singleprocess mode
str(TAD::generate_random_matrix(
  weights = weights,
  aggregation_factor = aggregation_factor,
  randomization_number = 500
))
#> 'data.frame':    1002 obs. of  4 variables:
#>  $ number: int  0 0 1 1 2 2 3 3 4 4 ...
#>  $ index1: num  1 0 1 0 1 0 2 0 2 0 ...
#>  $ index2: num  2 8 2 8 2 2 1 8 1 8 ...
#>  $ index3: num  0 2 0 2 0 8 0 2 0 2 ...

Run with Multiprocessing

Load the future library and define the future::plan to future::multisession.


library(future)
plan(multisession)
weights <- data.frame(
  sp1 = c(1, 0),
  sp2 = c(2, 8),
  sp3 = c(0, 2)
)
aggregation_factor <- data.frame(
  plots = c("plot1", "plot2")
)

time_before <- proc.time()[[1]]
## This will run in multiprocessing mode
str(TAD::generate_random_matrix(
  weights = weights,
  aggregation_factor = aggregation_factor,
  randomization_number = 500
))
#> 'data.frame':    1002 obs. of  4 variables:
#>  $ number: int  0 0 1 1 2 2 3 3 4 4 ...
#>  $ index1: num  1 0 1 0 2 0 1 0 1 0 ...
#>  $ index2: num  2 8 2 2 1 8 2 8 2 2 ...
#>  $ index3: num  0 2 0 8 0 2 0 2 0 8 ...

When you have finished

Don’t forget to reset the plan to future::sequencial when you have finished your processing. It is necesary to cleanup the resources allocated with the underlying calls of the parallel package.

plan(sequential)

Running the TAD Analysis


with_parallelism <- function(x) {
  future::plan(future::multisession)
  on.exit(future::plan(future::sequential))
  force(x)
}

# weights <- TAD::AB[, c(5:102)]
weights <- TAD::AB[, c(5:30)]
with_parallelism(
  result <- TAD::launch_analysis_tad(
    weights = weights,
    weights_factor = TAD::AB[, c("Year", "Plot", "Treatment", "Bloc")],
  trait_data = log(TAD::trait[["SLA"]])[seq_len(ncol(weights))],
    aggregation_factor_name = c("Year", "Bloc"),
    statistics_factor_name = c("Treatment"),
    regenerate_abundance_df = TRUE,
    regenerate_weighted_moments_df = TRUE,
    regenerate_stat_per_obs_df = TRUE,
    regenerate_stat_per_rand_df = TRUE,
    randomization_number = 100,
    seed = 1312,
    significativity_threshold = c(0.05, 0.95)
  )
)

str(result)
#> List of 9
#>  $ raw_abundance_df          :'data.frame':  9696 obs. of  27 variables:
#>   ..$ number : int [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index1 : num [1:9696] 0 0.84 0 9.15 0 ...
#>   ..$ index2 : num [1:9696] 0 0.84 0 0 0 ...
#>   ..$ index3 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index4 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index5 : num [1:9696] 1 1.681 21.26 22.876 0.794 ...
#>   ..$ index6 : num [1:9696] 1 0 1.575 0 0.794 ...
#>   ..$ index7 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index8 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index9 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index10: num [1:9696] 0 0 0 0 0 ...
#>   ..$ index11: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index12: num [1:9696] 0 0.84 0 0 0 ...
#>   ..$ index13: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index14: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index15: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index16: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index17: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index18: num [1:9696] 0 0 0.787 0.654 0 ...
#>   ..$ index19: num [1:9696] 2 0 0 1.31 2.38 ...
#>   ..$ index20: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index21: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index22: num [1:9696] 0 0 2.36 0 0 ...
#>   ..$ index23: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index24: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index25: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index26: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>  $ filtered_weights          :'data.frame':  96 obs. of  25 variables:
#>   ..$ SP1 : num [1:96] 0 0.84 0 9.15 0 ...
#>   ..$ SP3 : num [1:96] 0 0.84 0 0 0 ...
#>   ..$ SP4 : int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP5 : num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP6 : num [1:96] 1 1.681 21.26 22.876 0.794 ...
#>   ..$ SP7 : num [1:96] 1 0 1.575 0 0.794 ...
#>   ..$ SP8 : int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP9 : num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP11: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP12: num [1:96] 0 0 0 0 0 ...
#>   ..$ SP13: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP14: num [1:96] 0 0.84 0 0 0 ...
#>   ..$ SP15: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP16: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP18: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP19: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP20: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP21: num [1:96] 0 0 0.787 0.654 0 ...
#>   ..$ SP22: num [1:96] 2 0 0 1.31 2.38 ...
#>   ..$ SP24: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP25: num [1:96] 0 0 2.36 0 0 ...
#>   ..$ SP26: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP28: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP30: int [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ SP31: num [1:96] 0 0 0 0 0 0 0 0 0 0 ...
#>  $ filtered_weights_factor   :'data.frame':  96 obs. of  4 variables:
#>   ..$ Year     : Factor w/ 12 levels "2010","2011",..: 1 1 1 1 2 2 2 2 3 3 ...
#>   ..$ Plot     : Factor w/ 8 levels "4","6","11","13",..: 2 4 6 8 2 4 6 8 2 4 ...
#>   ..$ Treatment: Factor w/ 2 levels "Mown_NPK","Mown_Unfertilized": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ Bloc     : Factor w/ 2 levels "1","2": 1 1 2 2 1 1 2 2 1 1 ...
#>  $ filtered_trait_data       : num [1:25] 2.73 3.47 3.48 3.63 3.2 ...
#>  $ abundance_df              :'data.frame':  9696 obs. of  26 variables:
#>   ..$ number : int [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index1 : num [1:9696] 0 0.84 0 9.15 0 ...
#>   ..$ index2 : num [1:9696] 0 0.84 0 0 0 ...
#>   ..$ index3 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index4 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index5 : num [1:9696] 1 1.681 21.26 22.876 0.794 ...
#>   ..$ index6 : num [1:9696] 1 0 1.575 0 0.794 ...
#>   ..$ index7 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index8 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index9 : num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index10: num [1:9696] 0 0 0 0 0 ...
#>   ..$ index11: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index12: num [1:9696] 0 0.84 0 0 0 ...
#>   ..$ index13: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index14: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index15: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index16: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index17: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index18: num [1:9696] 0 0 0.787 0.654 0 ...
#>   ..$ index19: num [1:9696] 2 0 0 1.31 2.38 ...
#>   ..$ index21: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index22: num [1:9696] 0 0 2.36 0 0 ...
#>   ..$ index23: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index24: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index25: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ index26: num [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>  $ weighted_moments          :'data.frame':  9696 obs. of  10 variables:
#>   ..$ Year        : Factor w/ 12 levels "2010","2011",..: 1 1 1 1 2 2 2 2 3 3 ...
#>   ..$ Plot        : Factor w/ 8 levels "4","6","11","13",..: 2 4 6 8 2 4 6 8 2 4 ...
#>   ..$ Treatment   : Factor w/ 2 levels "Mown_NPK","Mown_Unfertilized": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ Bloc        : Factor w/ 2 levels "1","2": 1 1 2 2 1 1 2 2 1 1 ...
#>   ..$ number      : int [1:9696] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ mean        : num [1:9696] 3.19 3.18 3.19 3.07 3.17 ...
#>   ..$ variance    : num [1:9696] 0.0212 0.0607 0.0108 0.0427 0.0193 ...
#>   ..$ skewness    : num [1:9696] 0.768 -0.891 -0.929 -0.979 1.103 ...
#>   ..$ kurtosis    : num [1:9696] 1.95 2.62 6.19 2.04 2.58 ...
#>   ..$ distance_law: num [1:9696] -0.503 -0.037 3.467 -0.78 -0.497 ...
#>  $ statistics_per_observation:'data.frame':  96 obs. of  20 variables:
#>   ..$ Year                             : Factor w/ 12 levels "2010","2011",..: 1 1 1 1 2 2 2 2 3 3 ...
#>   ..$ Plot                             : Factor w/ 8 levels "4","6","11","13",..: 2 4 6 8 2 4 6 8 2 4 ...
#>   ..$ Treatment                        : Factor w/ 2 levels "Mown_NPK","Mown_Unfertilized": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ Bloc                             : Factor w/ 2 levels "1","2": 1 1 2 2 1 1 2 2 1 1 ...
#>   ..$ standardized_observedmean        : num [1:96] -0.296 -0.611 0.201 -0.689 -0.26 ...
#>   ..$ standardized_min_quantilemean    : num [1:96] -1.75 -1.83 -1.68 -1.78 -1.74 ...
#>   ..$ standardized_max_quantilemean    : num [1:96] 1.28 1.51 1.37 1.43 1.31 ...
#>   ..$ significancemean                 : logi [1:96] FALSE FALSE FALSE FALSE FALSE FALSE ...
#>   ..$ standardized_observedvariance    : num [1:96] -0.441 0.536 -0.594 0.532 -0.613 ...
#>   ..$ standardized_min_quantilevariance: num [1:96] -1.006 -1.231 -1.03 -0.974 -1.19 ...
#>   ..$ standardized_max_quantilevariance: num [1:96] 1.89 1.55 2.23 2.16 1.74 ...
#>   ..$ significancevariance             : logi [1:96] FALSE FALSE FALSE FALSE FALSE FALSE ...
#>   ..$ standardized_observedskewness    : num [1:96] 1.647 -0.567 -0.309 -0.247 1.286 ...
#>   ..$ standardized_min_quantileskewness: num [1:96] -1.37 -1.42 -1.39 -1.8 -1.29 ...
#>   ..$ standardized_max_quantileskewness: num [1:96] 1.52 1.73 1.42 1.56 1.35 ...
#>   ..$ significanceskewness             : logi [1:96] TRUE FALSE FALSE FALSE FALSE FALSE ...
#>   ..$ standardized_observedkurtosis    : num [1:96] 0.292 0.755 -0.612 -0.702 0.344 ...
#>   ..$ standardized_min_quantilekurtosis: num [1:96] -1.814 -1.724 -0.973 -0.757 -1.619 ...
#>   ..$ standardized_max_quantilekurtosis: num [1:96] 1.16 1.54 1.74 1.94 1.24 ...
#>   ..$ significancekurtosis             : logi [1:96] FALSE FALSE FALSE FALSE FALSE FALSE ...
#>  $ stat_per_rand             :'data.frame':  202 obs. of  8 variables:
#>   ..$ number               : int [1:202] 0 0 1 1 2 2 3 3 4 4 ...
#>   ..$ Treatment            : Factor w/ 2 levels "Mown_NPK","Mown_Unfertilized": 1 2 1 2 1 2 1 2 1 2 ...
#>   ..$ slope                : num [1:202] 1.15 1.76 1.04 1.23 1.03 ...
#>   ..$ intercept            : num [1:202] 2.15 1.23 1.89 1.93 1.54 ...
#>   ..$ rsquare              : num [1:202] 0.811 0.865 0.94 0.864 0.988 ...
#>   ..$ tad_stab             : num [1:202] 1.512 0.674 1.149 0.91 0.568 ...
#>   ..$ distance_to_family   : num [1:202] 1.65 0.994 1.173 1.07 0.628 ...
#>   ..$ cv_distance_to_family: num [1:202] 206.5 258.4 198.9 332.9 94.6 ...
#>  $ ses_skr                   :'data.frame':  2 obs. of  13 variables:
#>   ..$ slope_ses       : num [1:2] -0.18 3.54
#>   ..$ slope_signi     : logi [1:2] FALSE TRUE
#>   ..$ intercept_ses   : num [1:2] 1.48 -2.32
#>   ..$ intercept_signi : logi [1:2] FALSE TRUE
#>   ..$ rsquare_ses     : num [1:2] -0.843 -0.147
#>   ..$ rsquare_signi   : logi [1:2] FALSE FALSE
#>   ..$ tad_stab_ses    : num [1:2] 0.858 -0.852
#>   ..$ tad_stab_signi  : logi [1:2] FALSE FALSE
#>   ..$ tad_eve_ses     : num [1:2] 0.917 -0.475
#>   ..$ tad_eve_signi   : logi [1:2] FALSE FALSE
#>   ..$ cv_tad_eve_ses  : num [1:2] -1.05 -0.455
#>   ..$ cv_tad_eve_signi: logi [1:2] FALSE FALSE
#>   ..$ Treatment       : chr [1:2] "Mown_NPK" "Mown_Unfertilized"