Functional MRI Pre-Processing Pipeline

The fMRI Pre-Processing Pipeline is a lightweight, end-to-end workflow built around fMRIPrep. FMRIPrep handles motion/distortion correction, anatomical/functional alignment to standard spaces, surface mapping, confound modeling, and we added connectivity analysis, with clear QC visualizations considering QC on the number of volumes (Adjustable), FD, high-motion ratio; QC with visual connectivity matrices; and tabular outputs for downstream ML/AI.

• Python ≥ 3.8
• Singularity/Apptainer for containerized execution
• Python packages: numpy, pandas, nilearn, nibabel, tqdm
• fMRIPrep Singularity image: fmriprep_latest.sif (auto-downloaded)
• FreeSurfer license: Valid license file
• (Optional) GPU support with CUDA drivers
• (Optional) conda for environment management

Example Environment Setup

conda create -y -n fmri python=3.10
conda activate fmri
pip install numpy pandas nilearn nibabel tqdm

All paths and behavior are controlled by config.py (shared by all scripts). The pipeline is config-driven with minimal CLI flags for convenience.

Key Configuration Items:

Paths:

• INPUT_DIR: BIDS input directory
• OUTPUT_DIR: fMRIPrep outputs
• WORK_DIR: Temporary working directory
• LOG_DIR: Pipeline logs

Hardware:

• USE_GPU: Enable/disable GPU acceleration
• GPUS: List of available GPU IDs
• MAX_PARALLEL: Maximum parallel subjects
• MEMORY_MB, NPROCS, OMP_NTHREADS: Resource allocation

fMRIPrep:

• OUTPUT_SPACES: Target output spaces
• FS_NO_RECONALL: Skip FreeSurfer reconstruction
• LONGITUDINAL: Enable longitudinal processing

Connectivity:

• GENERATE_CONNECTIVITY: Enable connectivity matrix generation
• DEFAULT_ATLASES: Brain atlases for parcellation
• DEFAULT_CONFOUNDS: Confound regressors
• CONNECTIVITY_OUTPUT_DIR: Output directory for matrices

fMRIPrep Output Structure Configuration:

• HANDLE_SESSIONS: Handle session-based fMRIPrep output structure (default: True)
• SESSION_PREFIX: Session directory prefix (default: "ses-")
• REQUIRED_BOLD_SUFFIX: Required BOLD file suffix (default: _desc-preproc_bold.nii.gz)
• REQUIRED_CONFOUNDS_SUFFIX: Required confounds suffix (default: _desc-confounds_timeseries.tsv)

File Selection Strategy:

• USE_LARGEST_FILE: Use the file with the most time points if multiple are available (default: True)
• MIN_TIME_POINTS: Minimum required time points for valid data (default: 50)

QC thresholds:

• MIN_VOLUMES_PASS: 180 (minimum valid volumes for PASS)
• MIN_VOLUMES_FAIL: 120 (below this = FAIL)
• FD_PASS_THRESHOLD: 0.3 mm (mean FD)
• FD_WARN_THRESHOLD: 0.5 mm (mean FD)
• HIGH_MOTION_PASS_RATIO: 20% (% frames with FD > 0.5 mm)
• HIGH_MOTION_WARN_RATIO: 30% (% frames with FD > 0.5 mm)

Verifies all required components are available:

• Singularity image
• FreeSurfer license
• Input directory structure
• BIDS filter file

./launcher.sh --check

Discovers available subjects in BIDS directory and creates processing queue.

Inputs: BIDS directory structure (sub-*/)
Outputs: Subject list file (remaining_subjects_array.txt)

python subject_manager.py --action discover --output subjects.txt
python subject_manager.py --action add --subjects sub-01 sub-02
python subject_manager.py --action remove --subjects sub-01

Includes automatic connectivity generation and visualization.

Processing Steps:

1. BIDS validation and input parsing
2. Anatomical preprocessing (skull stripping, registration)
3. Functional preprocessing (motion correction, distortion correction)
4. Surface generation and volume-to-surface mapping
5. Confound estimation and timeseries extraction
6. Output space transformation and resampling
7. Automatic connectivity matrix generation (if GENERATE_CONNECTIVITY = True)
8. Automatic visualization generation

Outputs (per subject in OUTPUT_DIR):

• *_desc-preproc_bold.nii.gz: Preprocessed BOLD timeseries
• *_desc-confounds_timeseries.tsv: Confound regressors
• *_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz: Anatomical in MNI space
• HTML reports with quality metrics
• Connectivity matrices and visualizations (automatic)

Execution Modes:

Single Subject:

python pipeline_runner.py --mode single --subject ON00400 --cpu-only

Parallel Processing:

python pipeline_runner.py --mode parallel --subjects ON00400 ON01016 --cpu-only

Watch Mode (Queue Processing):

python pipeline_runner.py --mode watch

Status Check:

python pipeline_runner.py --mode status

Note: All modes automatically include connectivity generation and visualization unless disabled with --no-connectivity flag.

Automatically runs after successful fMRIPrep completion (if GENERATE_CONNECTIVITY = True).

Processing Steps:

1. Load preprocessed fMRI data and confounds
2. Apply brain atlas parcellation (AAL, Harvard-Oxford, Schaefer, etc.)
3. Extract regional timeseries with confound regression
4. Compute connectivity matrices (correlation, partial correlation, tangent, covariance)
5. Generate automatic visualizations (connectivity heatmaps + distribution plots + statistics)
6. Save matrices, timeseries, and visualizations

Outputs (per subject in CONNECTIVITY_OUTPUT_DIR):

• sub-*_*_timeseries.csv: Regional timeseries
• sub-*_*_*_connectivity.csv: Connectivity matrices
• sub-*_*_*_visualization.png: Connectivity heatmaps with statistics

Available Atlases: aal, harvard_oxford, schaefer-100, schaefer-400, difumo-256, difumo-1024

python connectivity_matrix.py --subjects sub-01 sub-02 --atlases aal schaefer-100
# OR use standalone runner:
python run_connectivity.py --all-completed --atlases aal harvard_oxford

Note: When using pipeline_runner.py, connectivity generation (including visualization) happens automatically after each subject completes fMRIPrep successfully.

Processing Steps:

1. Find all subjects with fMRIPrep outputs
2. Locate best confounds file per subject/session (≥50 volumes)
3. Extract motion parameters (framewise_displacement)
4. Compute QC metrics: Mean FD, Max FD, High-Motion Ratio, Volumes
5. Apply thresholds to assign PASS / REVIEW / FAIL
6. Generate subject-level and group-level QC HTML reports
7. Link QC results with connectivity matrices and fMRIPrep HTML reports

Outputs (per subject in qc_reports/subjects):

• sub-*_ses-*_qc.html: Detailed subject/session QC report

Outputs (group level in qc_reports/):

• fmri_qc_summary.html: Interactive QC summary dashboard
• fmri_qc_data.csv: Tabular QC data (metrics + status)

QC Thresholds:

• PASS: Mean FD ≤ 0.3 mm, High-Motion Ratio ≤ 20%, Volumes ≥ 180
• REVIEW: Mean FD 0.3–0.5 mm OR High-Motion Ratio ≤ 30% OR Volumes 120–179
• FAIL: Mean FD ≥ 0.5 mm OR High-Motion Ratio > 30% OR Volumes < 120

# Run QC for all subjects
python fmri_qc.py --all

# Run QC for specific subjects
python fmri_qc.py --subjects 001 002

# Export group-level QC summary
python fmri_qc.py --all --export-summary

• Unified Workflow: Single command runs fMRIPrep → Connectivity → Visualization automatically
• Smart Processing: Only generates connectivity for successfully completed fMRIPrep subjects
• Automatic Visualization: Every connectivity matrix includes heatmap + statistics plots
• Resource Optimization: Connectivity generation uses the same resource allocation as fMRIPrep
• Error Resilience: Pipeline continues if connectivity generation fails for individual subjects
• Duplicate Prevention: Skips connectivity generation if matrices already exist
• Configuration Control: Enable/disable via GENERATE_CONNECTIVITY in config.py

# 1) Setup environment
conda create -y -n fmri python=3.10
conda activate fmri
pip install numpy pandas nilearn nibabel tqdm

# 2) Configure pipeline
# Edit config.py with your paths and settings

# 3) Check prerequisites
./launcher.sh --check

# 4) Discover subjects
python subject_manager.py --action discover

# 5) Single subject test (includes fMRIPrep + Connectivity + Visualization)
python pipeline_runner.py --mode single --subject ON00400 --cpu-only

# 6) Parallel processing (includes automatic connectivity + visualization)
python pipeline_runner.py --mode parallel --subjects ON00400 ON01016

# 7) Alternative: Standalone connectivity generation
python run_connectivity.py --all-completed

# 8) Run QC (Quality Control)
python fmri_qc.py --all

# 9) Monitor status
python pipeline_runner.py --mode status

Note: Steps 5-6 automatically include connectivity matrix generation and visualization. No separate connectivity command needed!

High-Performance Single Subject

# config.py
MAX_PARALLEL = 1
MEMORY_MB = 100000    # 100GB
NPROCS = 24
OMP_NTHREADS = 8

Balanced Parallel Processing

# config.py
MAX_PARALLEL = 2
MEMORY_MB = 50000     # 50GB per subject
NPROCS = 12
OMP_NTHREADS = 6

CPU-Only Processing

# config.py
USE_GPU = False
MAX_PARALLEL = 4
MEMORY_MB = 32000     # 32GB per subject
NPROCS = 8
OMP_NTHREADS = 4

OUTPUT_DIR/
├── sub-*/
│   ├── anat/
│   │   ├── *_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
│   │   └── *_desc-brain_mask.nii.gz
│   ├── func/
│   │   ├── *_desc-preproc_bold.nii.gz
│   │   ├── *_desc-confounds_timeseries.tsv
│   │   └── *_desc-carpetplot_bold.svg
│   └── figures/
│       └── *.svg  # fMRIPrep figures
├── sub-*.html              # fMRIPrep per-subject HTML report
├── logs/
│   ├── sub-*.log
│   ├── sub-*.error
│   └── watcher.log
├── connectivity_matrices/
│   └── sub-*/
│       ├── *_timeseries.csv
│       ├── *_*_connectivity.csv
│       └── *_*_visualization.png
└── qc_reports/             # QC script
    ├── fmri_qc_summary.html
    └── subjects/
        └── sub-*_ses-*_qc.html

Pipeline Runner

# Core processing (includes automatic connectivity + visualization)
python pipeline_runner.py --mode {single|parallel|watch|status}
python pipeline_runner.py --mode single --subject SUB_ID [--cpu-only] [--no-connectivity]
python pipeline_runner.py --mode parallel --subjects SUB1 SUB2 [--cpu-only]

# Options
--cpu-only          # Force CPU-only processing
--no-connectivity   # Skip connectivity matrix generation and visualization
--check             # Check prerequisites only

Subject Manager

# Subject management
python subject_manager.py --action {discover|add|remove|list}
python subject_manager.py --action discover [--output FILE]
python subject_manager.py --action add --subjects SUB1 SUB2
python subject_manager.py --action remove --subjects SUB1

Connectivity Generation

# Connectivity matrices + automatic visualization
python connectivity_matrix.py --subjects SUB1 SUB2 --atlases aal schaefer-100
python run_connectivity.py --all-completed --atlases aal harvard_oxford
python run_connectivity.py --subjects SUB1 SUB2 --corr-kinds full-corr partial-corr

# All commands automatically generate both CSV matrices AND PNG visualizations

Launcher (Alternative Interface)

# Bash interface (includes automatic connectivity + visualization)
./launcher.sh --mode {watch|parallel|status|single}
./launcher.sh --mode single --subject SUB_ID [--cpu-only]
./launcher.sh --check

Common Issues

Exit code 126: Permission issues with Singularity

chmod +x /path/to/fmriprep_latest.sif

Exit code 2: Subject not found in BIDS directory

ls /path/to/bids/directory/sub-*
python subject_manager.py --action list

FreeSurfer license error: Invalid or missing license

# Check license file exists and is valid
cat /path/to/license.txt

Out of memory: Reduce resource allocation

# config.py
MEMORY_MB = 32000  # Reduce from higher values
NPROCS = 8         # Reduce parallel processes

GPU not found: Force CPU-only processing

python pipeline_runner.py --mode single --subject SUB_ID --cpu-only

Log Analysis

# Check pipeline status
python pipeline_runner.py --mode status

# View subject logs
tail -f /path/to/logs/sub-SUB_ID.log
cat /path/to/logs/sub-SUB_ID.error

# Monitor system resources
top -u $USER
nvidia-smi  # For GPU usage

fMRI QC Manual Review Summary

Overall QC Review

All subjects and sessions are reviewed and classified as PASS, REVIEW, or FAIL based on motion (framewise displacement, FD) and scan length. PASS indicates high-quality data, REVIEW marks usable but moderate-quality data, and FAIL means the data are not usable. The thresholds for these categories (volumes, FD cutoffs, motion ratio) are fully configurable in the QC settings.

Per-Subject QC Reports

Each subject has its own HTML report showing QC status, task and scan details, number of volumes, and motion metrics such as mean and maximum FD. The report also links to the subject’s fMRIPrep output for visual inspection. If connectivity analysis is included, the report provides the connectivity matrix, visualizations, and ROI time series.

fMRIPrep HTML Report Manual Review Guidelines

For detailed quality assessment, review the fMRIPrep HTML reports. Key areas to check:

• Red line: Should outline the brain (not skull/dura)
• Blue line: Should separate white and grey matter
• Check: No brain tissue cutoff

• Ventricles: Should be aligned between subject and template
• White/grey matter boundaries: Should match
• Check: No stretching or distortion

@software{abbasi2025fmri,
  author       = {Mohammad Hassan Abbasi and Ehsan Adeli},
  title        = {f{MRI} {P}rocessing {P}ipeline: A lightweight, end-to-end framework for {BIDS}-compatible f{MRI} preprocessing and quality control},
  year         = {2025},
  institution  = {Stanford Translational AI Lab, Stanford University},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17516368},
  url          = {https://doi.org/10.5281/zenodo.17516604},
  note = {Zenodo, doi: 10.5281/zenodo.17516604},
}