fl-prog

Federated disease progression modelling.

Setup

Check pyproject.toml for supported Python version(s).

In a Python environment managed by uv:

uv sync

In other types of Python environments (venv, conda, etc.):

pip install -e .

Steps

These scripts should be run sequentially.

Get data

Synthetic

# iid case
./scripts/simulate_data.py --tag iid

# non-iid cases
./scripts/simulate_data.py --t0-min 0 --t0-max 0.32 --t0-min 0.33 --t0-max 0.65 --t0-min 0.66 --t0-max 1 --tag non_overlapping_t0
./scripts/simulate_data.py --sigma 0.05 --sigma 0.1 --sigma 0.15 --tag unequal_sigma

ADNI-calibrated simulated data

First, save the diagnosis-specific time-shift distributions from the fitted model by running model_fits_adni.ipynb, making sure to uncomment the cell titled "For ADNI-Calibrated Simulator". Then:

./scripts/simulate_data_adni_timeshift.py --tag simulated_adni_timeshift

This will generate a ./data/simulated_adni_onset/ folder containing three simulated files that have the same names as their real ADNI counterparts and one metadata .json file.

Run the following command, passing the three simulated files' paths as CLI arguments:

./scripts/get_adni_data.py --tag simulated_adni_timeshift --iid/non-iid \
--idps \<SIMULATED FREESURFER CSV PATH\> \
--config \<SIMULATED CONFIG JSON PATH\> \
-adni-merge \<SIMULATED ADNIMERGE CSV PATH\>

ADNI

./scripts/get_adni_data.py --tag adni_iid --iid
./scripts/get_adni_data.py --tag adni_noniid --non-iid
./scripts/get_adni_data.py --tag adni_noniid_diag --non-iid --non-iid-strategy diagnosis

Optional: Regress chronological age out from biomarkers

Only for datasets with age information.

# Future option if injected artificial age effects into ADNI-calibrated simulated data
# ./scripts/regress_age_out.py --tag simulated_adni_timeshift

./scripts/regress_age_out.py --tag adni_iid --mode pooled-sites --min-max

./scripts/regress_age_out.py --tag adni_noniid --mode site --min-max

Then, must run all downstream scripts (e.g., merge_data.py, create_nodes.py, etc.) with --tag adni_<iid / non_iid>_age_adjusted_<minmax / NOTHING>

Merge data for centralized case

./scripts/merge_data.py --tag iid
./scripts/merge_data.py --tag non_overlapping_t0
./scripts/merge_data.py --tag unequal_sigma

./scripts/merge_data.py --tag simulated_adni_timeshift

./scripts/merge_data.py --tag adni_iid
./scripts/merge_data.py --tag adni_noniid
./scripts/merge_data.py --tag adni_noniid_diag

Create Fed-BioMed nodes and update their config.ini

This only needs to be run if some of the nodes haven't been created yet.

./scripts/create_nodes.py --tag iid
./scripts/create_nodes.py --tag non_overlapping_t0
./scripts/create_nodes.py --tag unequal_sigma

./scripts/create_nodes.py --tag simulated_adni_timeshift

./scripts/create_nodes.py --tag adni_iid
./scripts/create_nodes.py --tag adni_noniid
./scripts/create_nodes.py --tag adni_noniid_diag

Add data to nodes

If needed, use --wipe to clear existing datasets from each node.

./scripts/add_datasets_to_nodes.py --tag iid
./scripts/add_datasets_to_nodes.py --tag non_overlapping_t0
./scripts/add_datasets_to_nodes.py --tag unequal_sigma

./scripts/add_datasets_to_nodes.py --tag simulated_adni_timeshift

./scripts/add_datasets_to_nodes.py --tag adni_iid
./scripts/add_datasets_to_nodes.py --tag adni_noniid
./scripts/add_datasets_to_nodes.py --tag adni_noniid_diag

Start the nodes in separate processes

Start each node in a separate Terminal. These need to be running for the next script (run_fedbiomed.py) to work. Use Ctrl+C to stop a node when done.

# replace <NODE_ID> by '1', '2', '3', ..., or 'centralized'
fedbiomed node -p ./fedbiomed/node-<NODE_ID> start

Run model fitting

./scripts/run_fedbiomed.py --tag iid --n-rounds 5 --n-updates 100 --learning-rate 0.05 --time-shift-range 0 1
./scripts/run_fedbiomed.py --tag non_overlapping_t0 --n-rounds 5 --n-updates 100 --learning-rate 0.05 --time-shift-range 0 1
./scripts/run_fedbiomed.py --tag unequal_sigma --n-rounds 5 --n-updates 100 --learning-rate 0.05 --time-shift-range 0 1

./scripts/run_fedbiomed.py --tag simulated_adni_timeshift --learning-rate 0.05 --n-rounds 6 --n-updates 25 --time-shift-range 0 3 --lambda 10 --training-replies --aggregated-params

./scripts/run_fedbiomed.py --tag adni_iid --learning-rate 0.05 --n-rounds 6 --n-updates 25 --time-shift-range 0 3 --lambda 10 --training-replies --aggregated-params
./scripts/run_fedbiomed.py --tag adni_noniid --learning-rate 0.05 --n-rounds 6 --n-updates 25 --time-shift-range 0 3 --lambda 10 --training-replies --aggregated-params
./scripts/run_fedbiomed.py --tag adni_noniid_diag --learning-rate 0.05 --n-rounds 6 --n-updates 25 --time-shift-range 0 3 --lambda 10 --training-replies --aggregated-params

Plot

Synthetic data experiments

Run cells in ./notebooks/model_fits_synthetic.ipynb

ADNI-calibrated simulated data experiments

Run cells in ./notebooks/model_fits_simulated_adni_timeshift.ipynb

ADNI data experiments

Run cells in ./notebooks/model_fits_adni.ipynb