Skip to content

getwilds/cbioformatter

BranchesTags

Repository files navigation

cBioFormatter

A Python package for streamlined preparation and formatting of clinical and molecular genomic data for upload to cBioPortal.

Overview

cBioFormatter simplifies the process of converting your genomic data into cBioPortal-compatible formats. Designed for data scientists with basic Python knowledge, this package handles all the complexity of cBioPortal file formatting, validation, and metadata generation.

What it does:

  • Converts clinical data (patient and sample attributes) into cBioPortal format
  • Processes VCF files into MAF format for mutation data
  • Generates all required metadata files automatically
  • Validates your study using cBioPortal's official validator
  • Creates case lists for sample grouping

What you need:

  • Basic Python knowledge (pandas DataFrames, module imports)
  • Your clinical data (Excel, CSV, database query, anything that can be converted to a pandas DataFrame)
  • VCF files for mutation data (optional)
  • mafsmith installed (for VCF processing, optional)

Installation

pip install cbioportal-formatter

Additional requirements:

  • mafsmith (for mutation data processing, if using VCF files) - see mafsmith repo

Development

For local development, clone the repository and install in editable mode with dev dependencies.

Using uv (recommended)

uv is a fast Python package manager. If you don't have it installed:

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then set up the project:

git clone https://github.com/getwilds/cbioformatter.git
cd cbioformatter
uv sync --extra dev

To run commands in the virtual environment:

uv run pytest              # Run tests
uv run pytest --cov        # Run tests with coverage
uv run ruff check .        # Run linter
uv run ruff format .       # Format code
uv run ipython             # Interactive Python shell (or: uv run python)

Using pip

git clone https://github.com/getwilds/cbioformatter.git
cd cbioformatter
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"

To run tests and linting:

pytest                     # Run tests
pytest --cov               # Run tests with coverage
ruff check .               # Run linter
ruff format .              # Format code
ipython                    # Interactive Python shell (or: python)

Quick Start

Basic Study with Clinical Data Only

import pandas as pd
from cbioformatter import ClinicalStudy

# Prepare your sample-level clinical data
# (typically loaded from a CSV, Excel file, or database query)
sample_df = pd.DataFrame({
    'SAMPLE_ID': ['S001', 'S002', 'S003'],
    'PATIENT_ID': ['P001', 'P001', 'P002'],
    'TUMOR_TYPE': ['Primary', 'Metastasis', 'Primary'],
    'AGE_AT_DIAGNOSIS': [45, 45, 67]
})

# sample_df looks like:
# | SAMPLE_ID | PATIENT_ID | TUMOR_TYPE | AGE_AT_DIAGNOSIS |
# |-----------|------------|------------|------------------|
# | S001      | P001       | Primary    | 45               |
# | S002      | P001       | Metastasis | 45               |
# | S003      | P002       | Primary    | 67               |

# Prepare your patient-level clinical data (optional)
patient_df = pd.DataFrame({
    'PATIENT_ID': ['P001', 'P002'],
    'SEX': ['Female', 'Male'],
    'ETHNICITY': ['Hispanic', 'Asian']
})

# patient_df looks like:
# | PATIENT_ID | SEX    | ETHNICITY |
# |------------|--------|-----------|
# | P001       | Female | Hispanic  |
# | P002       | Male   | Asian     |

# Create and validate the study
study = ClinicalStudy(
    study_id="brca_ocdo_2026",
    name="Breast Cancer Study (Office of the Chief Data Officer 2026)",
    description="Clinical and genomic data from breast cancer patients",
    cancer_type="brca",  # must be a valid cBioPortal cancer type
    genome_build="GRCh38",  # Options: "GRCh37", "hg19", or "GRCh38"
    sample_data=sample_df,
    patient_data=patient_df  # optional
)

# Validate the study (generates temp files, runs validator, cleans up)
result = study.validate()

if result.is_valid:
    print("✓ Study is valid!")
    print(f"Validation report: {result.report_path}")
    
    # Write files to disk
    study.write_files(output_dir="./my_studies")
    print(f"Study files written to: ./my_studies/brca_ocdo_2026/")
else:
    print("✗ Validation failed. Check the report for details:")
    print(f"Report: {result.report_path}")

Study with Mutation Data

# Add VCF file paths to your sample DataFrame
sample_df = pd.DataFrame({
    'SAMPLE_ID': ['S001', 'S002', 'S003'],
    'PATIENT_ID': ['P001', 'P001', 'P002'],
    'TUMOR_TYPE': ['Primary', 'Metastasis', 'Primary'],
    'VCF_PATH': [
        '/data/vcf/S001.vcf',
        '/data/vcf/S002.vcf',
        None  # This sample has no mutation data
    ]
})

# The rest is identical - mutation data is automatically detected
study = ClinicalStudy(
    study_id="brca_ocdo_2026",
    name="Breast Cancer Study (Office of the Chief Data Officer 2026)",
    description="Clinical and genomic data from breast cancer patients",
    cancer_type="brca",
    genome_build="GRCh38",
    sample_data=sample_df
)

result = study.validate()
if result.is_valid:
    study.write_files(output_dir="./my_studies")

Features

Clinical Data Handling

Required columns:

  • SAMPLE_ID in sample DataFrame (must be unique)
  • PATIENT_ID in patient DataFrame if provided (must be unique)

Smart defaults:

  • If patient_data is not provided, it's auto-generated from unique PATIENT_ID values in sample_data
  • If PATIENT_ID column is missing from sample_data, each sample is assigned its own patient (PATIENT_ID = SAMPLE_ID)
  • Column names are automatically cleaned for cBioPortal compatibility while preserving display names
  • Data types are automatically inferred: NUMBER (int/float), BOOLEAN (bool), STRING (everything else)

Validation:

  • Ensures all SAMPLE_ID values are unique
  • Ensures all PATIENT_ID values are unique (if patient data provided)
  • Validates referential integrity (all patient IDs in samples exist in patient data)
  • Failures raise clear exceptions with specific issues identified

Mutation Data Processing

Input: VCF files (one per sample)

How it works:

  1. Add a VCF_PATH column to your sample_data DataFrame with file paths
  2. VCF files are automatically converted to MAF format using mafsmith
  3. All MAF files are concatenated into a single mutation file
  4. Sample IDs are correctly mapped to Tumor_Sample_Barcode

Flexible data availability:

  • If VCF_PATH column is missing entirely → no mutation data included
  • If some samples have VCF paths and others don't → mutation data included only for samples with valid paths
  • At least one valid VCF path must be provided if the column exists

Requirements:

  • mafsmith must be installed (see the mafsmith repo for build instructions; cargo build --release produces a single binary)
  • Reference data for your genome build must be downloaded: mafsmith fetch --genome grch38 populates ~/.mafsmith/GRCh38/ (~3.8 GB). If the FASTA index is missing, build it with samtools faidx
  • VCF files must match the specified genome build (GRCh37 or GRCh38)
  • Optionally pass ref_fasta_path=... to ClinicalStudy (or set CBIOFORMATTER_REF_FASTA) to override the bundled reference

Known limitation: mafsmith's embedded annotation (fastVEP) does not currently populate the optional SWISSPROT column. As a result, cBioPortal's Pfam-domain mutations view will not be available for mafsmith-produced studies until nf-osi/mafsmith closes that gap. Mutations still load and display normally; only the Pfam diagram is affected.

Study Validation

The validate() method:

  1. Creates temporary files in cBioPortal format
  2. Runs the official cBioPortal validator (from cBioPortal datahub-study-curation-tools)
  3. Generates an HTML validation report
  4. Cleans up temporary files
  5. Returns a validation result object

Validation result object:

result.is_valid      # True if validation passed (clean or warnings-only)
result.report_path   # Path to HTML validation report
result.errors        # Errors AND/OR warnings emitted by the validator

is_valid is True for a clean validation and for warnings-only results; in the warnings-only case, result.errors is populated and write_files(validate=True) proceeds with a UserWarning. Errors (validator exit code 1 or 2) raise ValidationError from write_files(validate=True) and study files are not written.

Validator acquisition: The cBioPortal validator is AGPL-3.0 licensed and lives in a separate repository, so cbioformatter does not bundle it. On first validate() call, the validator is cloned into ~/.cache/cbioformatter/validator/ (~5 MB, requires git and internet). Subsequent calls reuse the cache.

For air-gapped or CI environments, pre-clone the validator and set CBIOFORMATTER_VALIDATOR_PATH:

git clone --depth 1 https://github.com/cBioPortal/datahub-study-curation-tools.git
export CBIOFORMATTER_VALIDATOR_PATH=$(pwd)/datahub-study-curation-tools/validation/validator

File Output

The write_files() method generates a complete cBioPortal study directory:

my_studies/
└── brca_ocdo_2026/
    ├── meta_study.txt
    ├── meta_clinical_patient.txt
    ├── data_clinical_patient.txt
    ├── meta_clinical_sample.txt
    ├── data_clinical_sample.txt
    ├── meta_mutations.txt      # if mutation data provided
    ├── data_mutations.txt      # if mutation data provided
    ├── case_lists/
    │   ├── cases_all.txt
    │   └── cases_sequenced.txt          # if mutation data provided

Parameters:

  • output_dir (default: ".") - Base directory for output. Study files are created in {output_dir}/{study_id}/
  • validate (default: True) - If True, runs validation before writing files. Set to False to skip validation (use with caution).

API Reference

ClinicalStudy

ClinicalStudy(
    study_id: str,
    name: str,
    description: str,
    cancer_type: str,
    genome_build: str,
    sample_data: pd.DataFrame,
    patient_data: pd.DataFrame = None
)

Parameters:

  • study_id: Unique identifier for the study (no spaces, lowercase recommended)
  • name: Human-readable study name
  • description: Brief description of the study
  • cancer_type: Valid cBioPortal cancer type (see cBioPortal documentation)
  • genome_build: Reference genome build. Accepts UCSC names ("hg19", "hg38", "mm10") or NCBI/Ensembl aliases ("GRCh37", "GRCh38", "GRCm38"); aliases are translated to the UCSC form on write since cBioPortal's validator only accepts UCSC names
  • sample_data: pandas DataFrame with sample-level clinical attributes. Must include SAMPLE_ID. Optionally includes PATIENT_ID and VCF_PATH
  • patient_data: Optional pandas DataFrame with patient-level clinical attributes. Must include PATIENT_ID if provided

Methods:

validate()

Validates the study using cBioPortal's official validator.

Returns: ValidationResult object with:

  • is_valid (bool): Whether validation passed
  • report_path (str): Path to HTML validation report
  • errors (list): List of validation errors if validation failed

write_files(output_dir=".", validate=True)

Writes all study files to disk.

Parameters:

  • output_dir (str): Base output directory (default: current directory)
  • validate (bool): If True, runs validation before writing files (default: True)

Returns: Path to the created study directory ({output_dir}/{study_id}/)

Raises:

  • ValidationError if validate=True and the cBioPortal validator reports errors. Study files are not written. Pass validate=False to skip validation.

Example Workflow

See the example notebook for a complete walkthrough using simulated data.

Supported Data Types (Current Version)

  • ✅ Clinical data (patient and sample attributes)
  • ✅ Mutation data (VCF → MAF conversion)
  • ⏳ Copy number alterations (CNA) - planned for future release
  • ⏳ Gene expression data - planned for future release
  • ⏳ Methylation data - planned for future release

Requirements

  • Python 3.10+
  • pandas
  • mafsmith (optional, for VCF processing)

External Tools

This package relies on the following external tools for mutation data processing:

mafsmith (optional, for VCF processing):

  • Required only if you're including mutation data from VCF files
  • Build from source per the mafsmith repo (cargo build --release)
  • After install, download reference data with mafsmith fetch --genome grch38 (or grch37 / grcm39)
  • Includes embedded fastVEP annotation — no separate VEP install needed
  • Known limitation: the optional SWISSPROT column is not populated, so cBioPortal's Pfam-domain mutations view will be unavailable

Troubleshooting

Common Issues

"SAMPLE_ID duplicates found"

  • Ensure all values in your SAMPLE_ID column are unique
  • Check for accidentally duplicated rows in your data

"PATIENT_ID 'P123' not found in patient data"

  • Every patient ID referenced in sample data must exist in patient data
  • If you didn't provide patient data, this shouldn't happen (it's auto-generated)

"VCF file not found: /path/to/file.vcf"

  • Check that all file paths in the VCF_PATH column are correct
  • Ensure files are accessible from your current working directory

"Could not locate mafsmith"

  • Build mafsmith following the mafsmith repo and ensure the resulting binary is on your PATH, or set CBIOFORMATTER_MAFSMITH_PATH to its location

"Cannot read FASTA index reference.fa.fai"

  • mafsmith's fetch may not generate a FASTA index. Build it manually with samtools faidx ~/.mafsmith/<genome>/reference.fa

Validation fails with complex errors

  • Review the HTML validation report at the path provided
  • Common issues: incorrect cancer type, malformed column names, missing required fields

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Citation

If you use cBioFormatter in your research, please mention the GitHub repository:

cBioFormatter: https://github.com/getwilds/cbioportal-formatter

Future aim: We plan to submit cBioFormatter to the Journal of Open Source Software (JOSS) for peer review. Once published, a formal citation will be provided here.

Contact

Fred Hutch users:

External users:

Acknowledgments

  • Built to support the Fred Hutch Cancer Center cBioPortal instance
  • Uses cBioPortal's official validation tools
  • Part of the WILDS ecosystem

About

cBioPortal Study Formatting Python Package

Resources

Readme

License

MIT license

Code of conduct

Code of conduct

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases 3

cBioFormatter v0.3.0 Latest
Jun 22, 2026
+ 2 releases

Contributors

Languages