Below are step-by-step instructions on how to use the files in the OSF repository to replicate the analyses described in the manuscript.




Pre-processing

No. Description Input Scripts Output

1

For each [PANEL], a .xlsx workbook with the information about each risk measure and other relevant variables (e.g., age or year of birth) is compiled by adapting the information from the risk measure codebook and the panel’s codebooks/questionnaires. Separate tabs include information on the different classes of variables (e.g., measures, demographics, id). The merge_var_info.R script reads each sheet/tab from each .xlsx workbook, saves each one as a dataframe and merges the information from all the panels/sample into a list object (panel_var_info.rds). This file contains the information needed for the pre-processing of the raw data.

var_info/codebook_main.xlsx
var_info/indv_panel_var_info/PANEL_var_info.xlsx

var_info/code/merge_var_info.R

var_info/panel_variable_info.rds

2

For each [PANEL], an R script reads the panel_var_info.rds file and Raw Panel Data file(s) to select the relevant variables, and creates a .csv file (long format) with the relevant (and if needed, cleaned and re-coded) raw data, as well as a .csv file with the main information on the different risk preference measures that will be included in the analyses (e.g., measure category, domain). Simultaneously, a summary overview of the data is created for inspection. These pre-processed files are in the same format for all panels.

var_info/panel_variable_info.rds
local_drive/Raw panel data

pre_processing/code/PANEL_preproc.R

local_drive/PANEL_proc_data.csv
var_info/indv_panel_var_info/PANEL_risk_var_info.csv

3

The information on the final risk preference measures that will be included in the analyses (i.e., [PANEL]_risk_var_info.csv) for each sample are merged into a single R list object (panel_risk_info.rds) using the merge_risk_info.R script. This file is used for data processing.

var_info/indv_panel_var_info/PANEL_risk_var_info.csv

var_info/code/merge_risk_info.R

var_info/panel_risk_info.rds



Processing

Test-retest correlations

No. Description Input Scripts Output

1

For each [PANEL], an R script reads the [PANEL]_proc_data.csv and computes the test-retest correlations for different retest intervals for each measure using the function calc_retest.R. In addition, the relevant information for each risk preference measure is joined to the set of correlations. Altogether, this is saved as a .csv file. The [PANEL]_retest_data.csv file is in the same format for all panels.

local_drive/PANEL_proc_data.csv
var_info/panel_risk_info.rds

processing/code/temp_stability/calc_retest_function.R
processing/code/temp_stability/PANEL_retestcalc.R

processing/output/temp_stability/PANEL_retest_data.csv

2

Using merge_retest.R and the [PANEL]_retest.csv files, a complete_retest.csv file is created with the retest correlations of all the panels/samples.

processing/output/temp_stability/PANEL_retest.csv

processing/code/temp_stability/merge_retest.R

processing/output/temp_stability/complete_retest.csv

3

Using calc_agg_retest.R and the complete_retest.csv file, aggregated correlation coefficients are computed based on different criteria (e.g., minimum number of responses, age bins) and saves complete_agg_retest_yb[5/10/20].csv files (split by size of age bins).

processing/output/temp_stability/complete_retest.csv

processing/code/temp_stability/calc_agg_retest.R

processing/output/temp_stability/complete_agg_retest_ yb[5/10/20].csv


Inter-correlations

No. Description Input Scripts Output

1

For each panel, the R script [PANEL]_intercor.R reads the [PANEL]_proc_data.csv and computes the inter-correlations between different measures collected at the same data collection point using the function calc_intercor.R. In addition, the relevant information for each pair of risk preference measures is joined to the set of inter-correlations (from panel_risk_info.rds). Altogether, this is saved as a .csv file. The [PANEL]_intercor_data.csv file has the same format for all panels.

var_info/panel_risk_info.rds
local_drive/Raw panel data

processing/code/convergent_val/ calc_intercor_function.R
processing/code/convergent_val/PANEL_intercor.R

processing/output/convergent_val/PANEL_intercor_data.csv

2

Using the merge_intercor.R script and the [PANEL]_intercor_data.csv files, a complete_intercor.csv file is created with the inter-correlations of all the panels/samples.

processing/output/convergent_val/PANEL_intercor_data.csv

processing/code/convergent_val/merge_intercor.R

processing/output/convergent_val/complete_intercor.csv

3

Using calc_agg_intercor.R and the complete_intercor.csv file, aggregated correlation coefficients are computed based on different criteria (e.g., minimum number of responses) and saves complete_agg_intercor_yb[5/10/20].csv files (split by size of age bins).

processing/output/convergent_val/complete_intercor.csv

processing/code/convergent_val/calc_agg_intercor.R

processing/output/convergent_val/complete_agg_intercor_yb[5/10/20].csv



Analysis

Test-retest correlations

No. Description Input Scripts Output

1

Variance Decomposition: The var_decomposition.R script reads the complete_retest.csv and conducts variance decomposition using all the test-retest correlations (i.e., omnibus analysis) and separately for each measure category.

processing/output/temp_stability/complete_retest.csv

analysis/code/temp_stability/var_decomposition.R

analysis/output/temp_stability/shapley_values_[omni/pro/fre/beh]retest.csv
analysis/output/temp_stability/shapley_values[omni/pro/fre/beh]_retest_boot.csv

2

Fitting the MASC model: Using the corresponding R script (i.e., masc_[pro/fre/beh].R) for each measure category and the complete_agg_retest_yb10.csv file, the (meta-analytic) non-linear mixed-effects model (i.e.,MASC) is estimated. In addition, checks of model fit and convergence diagnostics are performed.

processing/output/temp_stability/complete_agg_retest_yb10.csv

analysis/code/temp_stability/fit_masc_[pro/fre/beh].R

analysis/output/temp_stability/masc_[pro/fre/beh].rds

3

Multiverse Analysis - Variance Decomposition: The var_decomposition_multiverse.R script reads the complete_retest.csv file and conducts variance decomposition for all the test-retest correlations (i.e., omnibus analysis) and separately for each measure category with the different subsets of retest correlations to assess the consistency of the variance explained by each predictor.

processing/output/temp_stability/complete_retest.csv

analysis/code/temp_stability/var_decomp_retest_multiverse.R

analysis/output/temp_stability/ shapley_values_multiv_[omni/pro/fre/beh].csv

4

Multiverse Analysis - Fitting the MASC model: Using the corresponding R script (i.e., fit_masc_[pro/fre/beh]_multiverse.R) for each measure category, the MASC model is estimated using the different subsets of aggregated correlation coefficients from the complete_agg_retest_yb[5/10/20].csv files. A summary of the output is saved to assess the consistency of the results.

processing/output/temp_stability/complete_agg_retest_yb[5/10/20].csv

analysis/code/temp_stability/masc_[pro/fre/beh]_multiverse.R

analysis/output/temp_stability/multiverse_fit_masc_[pro/fre/beh].rds


Inter-correlations

No. Description Input Scripts Output

1

Variance Decomposition: The var_decomposition.R script reads the complete_intercor.csv to conduct variance decomposition for the inter-correlations.

processing/output/convergent_val/complete_intercor.csv

analysis/code/convergent_val/var_decomposition.R

analysis/output/convergent_val/shapley_value_intercor.csv
analysis/output/convergent_val/shapley_values_intercor_boot.csv

2

Meta-Analysis and Meta-Regressions: The brms_ma.R script reads the complete_agg_intercor_yb10.csv file, estimates the different Bayesian random effects models (i.e., intercept only and with covariates), and saves the output.

processing/output/convergent_val/complete_agg_intercor_yb10.csv

analysis/code/brms_ma.R

analysis/output/convergent_val/fit_convergent_ma_[overall/measure/domain].rds

3

Multiverse Analysis - Variance Decomposition: The var_decomposition_multiverse.R script reads the complete_intercor.csv file and conducts variance decomposition for the inter-correlations with the different subsets of inter-correlations to assess the consistency of the variance explained by each predictor.

processing/output/convergent_val/complete_intercor.csv

analysis/code/var_decomposition_multiverse.R

analysis/output/ convergent_val/shapley_values_multiv_overall.csv

4

Multiverse Analysis Meta-Analysis and Meta-Regressions: Using the brms_ma_multiverse.R script, the different Bayesian random effects meta-analytic models (i.e., intercept only and with covariates) are estimated using different subsets of correlation coefficients from the complete_agg_intercor_yb[5/10/20].csv files and a summary of the output is saved to assess the consistency of the results.

processing/output/convergent_val/complete_agg_intercor_yb[5/10/20].csv

analysis/code/convergent_val/brms_ma_multiverse.R

analysis/output/convergent_val/ multiverse_fit_convergent_ma_[overall/measure].rds



Visualisation

Test-retest correlations

No. Description Input Scripts Output

1

The plot_measure_cor_overview.R script reads the complete_retest.csv file and plots the number of measures across retest intervals and the distributions of retest correlations.

processing/output/temp_stability/complete_retest.csv

figures/code/temp_stability/plot_measure_cor_overview.R

figures/output/temp_stability/meas_count.png

2

The plot_var_decomposition.R script reads the shapley_values_[omni/pro/fre/beh]retest.csv and shapley_values[omni/pro/fre/beh]_retest_boot.csv to plot the proportion of variance explained by different predictors.

analysis/output/temp_stability/shapley_values_[omni/pro/fre/beh]retest.csv
analysis/output/temp_stability/shapley_values[omni/pro/fre/beh]_retest_boot.csv

figures/code/temp_stability/plot_var_decomposition.R

figures/output/temp_stability/shapley_decomp_retest_fig.png

3

The plot_masc_pred.R script reads the masc_[pro/fre/beh]..rds files and plots the parameter estimates and model predictions.

analysis/output/temp_stability/masc_[pro/fre/beh].rds

figures/code/temp_stability/plot_masc_pred.R

figures/output/temp_stability/masc_pred_fig.png


Inter-correlations

No. Description Input Scripts Output

1

The plot_var_decomposition.R script reads the shapley_values_intercor.csv and shapley_values_intercor_boot.csv files to plot the proportion of variance explained by different predictors.

analysis/output/convergent_val/shapley_value_intercor.csv
analysis/output/convergent_val/shapley_values_intercor_boot.csv

figures/code/convergent_val/plot_var_decomposition.R

figures/output/convergent_val/shapley_decomp_convergent.png

1

The plot_cor_matrix.R script reads the fit_convergent_ma_[measure/domain].rds files and creates two correlation matrices displaying the pooled estimates.

analysis/output/convergent_val/fit_convergent_ma_[ measure/domain].rds

figures/code/convergent_val/plot_cor_matrix.R

figures/output/convergent_val/cor_matrix_fig.png