{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Nested Cross-Validation ML Pipeline\n", "## Liver Preservation Transcriptomics — Local (M1 Mac) Version\n", "\n", "**Implements:** True nested cross-validation — feature selection (EN + XGBoost top-20 genes) happens **strictly within each outer training fold**. No test samples leak into gene panel selection.\n", "\n", "### Setup (one-time)\n", "```bash\n", "conda create -n genetics python=3.10\n", "conda activate genetics\n", "conda install -c conda-forge numpy pandas scikit-learn xgboost shap jupyter\n", "pip install mygene tabulate\n", "```\n", "\n", "### Set your data path\n", "Edit `DATA_DIR` in the **Config** cell below to point to the folder containing your `*_EXPR_ML.csv` and `*_PHENO_ML.csv` files.\n", "\n", "### Design\n", "- **Outer loop:** `RepeatedStratifiedKFold(n_splits=5, n_repeats=3)` = 15 folds\n", "- **Within each outer fold:**\n", " 1. Adaptive variance pre-filter (≤10× n_train features, max 1500) — keeps SAGA well-conditioned for small cohorts\n", " 2. Fit `LogisticRegressionCV` (Elastic Net) on outer *training* data → best C / l1_ratio\n", " 3. Fit XGBoost on outer *training* data → gene importance ranking\n", " 4. Form `EN ∪ XGB` panel (top-20 from each)\n", " 5. Re-train a fixed logistic regression on panel-restricted *training* data\n", " 6. Evaluate on outer *test* fold → AUC, sensitivity, specificity, PPV\n", "- **Consensus panel:** genes selected in ≥33% of outer folds (adaptive threshold for small-n cohorts)\n", "- **Reports:** mean ± SD across all 15 folds" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Local path configuration ─────────────────────────────────────────────────\n", "# Point DATA_DIR to the folder containing your *_EXPR_ML.csv / *_PHENO_ML.csv files.\n", "# OUTPUT_DIR is set to the same folder by default; change it if you prefer\n", "# results written elsewhere.\n", "\n", "import os\n", "\n", "# *** EDIT THIS LINE ***\n", "DATA_DIR = os.path.expanduser('~/path/to/data')\n", "\n", "if not os.path.isdir(DATA_DIR):\n", " raise FileNotFoundError(\n", " f\"DATA_DIR not found: {DATA_DIR!r}\\n\"\n", " \"Please update DATA_DIR above to the folder containing your CSV files.\"\n", " )\n", "\n", "print(f'Data directory : {DATA_DIR}')\n", "print(f'Contents : {os.listdir(DATA_DIR)[:8]} ...')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Environment verification ─────────────────────────────────────────────────\n", "# Confirms all required packages are installed in the current kernel.\n", "# If any import fails, activate your conda env and re-launch Jupyter:\n", "# conda activate genetics && jupyter notebook\n", "\n", "import sys\n", "required = ['numpy', 'pandas', 'sklearn', 'xgboost', 'shap', 'mygene', 'tabulate', 'joblib']\n", "missing = []\n", "for pkg in required:\n", " try:\n", " __import__(pkg)\n", " except ImportError:\n", " missing.append(pkg)\n", "\n", "if missing:\n", " print(f'Missing packages: {missing}')\n", " print('Run in terminal: pip install ' + ' '.join(missing))\n", "else:\n", " import numpy, pandas, sklearn, xgboost, shap\n", " print('All packages OK')\n", " print(f' Python {sys.version.split()[0]}')\n", " print(f' numpy {numpy.__version__}')\n", " print(f' pandas {pandas.__version__}')\n", " print(f' sklearn {sklearn.__version__}')\n", " print(f' xgboost {xgboost.__version__}')\n", " print(f' shap {shap.__version__}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# IMPORTS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import os\n", "import warnings\n", "import numpy as np\n", "import pandas as pd\n", "from collections import defaultdict\n", "\n", "from sklearn.model_selection import RepeatedStratifiedKFold, StratifiedKFold\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.linear_model import LogisticRegressionCV, LogisticRegression\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.metrics import (\n", " roc_auc_score, confusion_matrix\n", ")\n", "\n", "from joblib import Parallel, delayed\n", "import xgboost as xgb\n", "import shap\n", "\n", "warnings.filterwarnings('ignore')\n", "print('All imports OK.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# CONFIG\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "RANDOM_STATE = 42\n", "\n", "# Inherit DATA_DIR set in the path cell above\n", "OUTPUT_DIR = DATA_DIR\n", "EXPORT_DIR = os.path.join(OUTPUT_DIR, 'ml_manuscript_outputs')\n", "os.makedirs(EXPORT_DIR, exist_ok=True)\n", "\n", "# ── Outer CV ────────────────────────────────────────────────────────────────\n", "OUTER_N_SPLITS = 5\n", "OUTER_N_REPEATS = 3 # 15 folds total\n", "\n", "# ── Feature selection ────────────────────────────────────────────────────────\n", "# Increased from 20 → 30 per model.\n", "# Union panel per fold = up to 60 unique genes (was ~33), giving the\n", "# consensus filter more candidates to work with across all datasets.\n", "TOP_N_GENES = 30\n", "\n", "# Consensus threshold: 0.33 (≥5/15 folds).\n", "# The stricter 0.50 threshold produces 0 consensus genes for small cohorts\n", "# (n<60). 0.33 is scientifically defensible and should be reported in Methods.\n", "CONSENSUS_MIN_FRAC = 0.33\n", "\n", "# ── Result quality flags ─────────────────────────────────────────────────────\n", "# Datasets whose results should be interpreted with extra caution:\n", "# GSE14951 (n=10): perfect AUC=1.0 is unreliable at this sample size;\n", "# report but flag as exploratory.\n", "# GSE15480 (n=24): AUC=1.0 plausible but CI extremely wide; treat cautiously.\n", "# GSE276531 (n=32): AUC=0.41 (below chance); label parsing or genuine lack\n", "# of signal — investigate pos/neg term assignment.\n", "MIN_SAMPLES_RELIABLE = 30 # datasets below this are flagged in output\n", "\n", "# ── EN inner CV ──────────────────────────────────────────────────────────────\n", "EN_L1_RATIOS = [0.1, 0.9] # two extremes capture the EN regularisation range\n", "EN_CS = np.logspace(-2, 2, 8) # 8 values over 4 orders of magnitude\n", "EN_INNER_CV = 5\n", "EN_MAX_ITER = 2000\n", "\n", "# ── Adaptive variance pre-filter ─────────────────────────────────────────────\n", "# Cap at 10× n_train to keep SAGA well-conditioned for small cohorts.\n", "# n_train=38 → min(38×10, 2000) = 380 features (small cohort)\n", "# n_train=64 → min(64×10, 2000) = 640 features (medium cohort)\n", "# n_train=200 → min(200×10,2000) = 2000 features (large cohort)\n", "# Raised MAX from 1500 → 2000 to give larger datasets more candidates now\n", "# that TOP_N_GENES is 30 (need a deeper pool to draw from).\n", "VARIANCE_TOP_K_MAX = 2000\n", "VARIANCE_TOP_K_MULTIPLIER = 10\n", "\n", "# ── Sex chromosome gene filter ───────────────────────────────────────────────\n", "# Y-linked and X-inactivation genes reflect donor sex imbalance between\n", "# classes, not liver preservation biology.\n", "# Extended after round-2 results: TXLNGY appeared in GSE151648_IRI panel.\n", "SEX_CHROMOSOME_GENES = {\n", " # Y-linked protein-coding (canonical chrY set)\n", " 'DDX3Y','EIF1AY','KDM5D','NLGN4Y','PRKY','PCDH11Y','RBMY1A1',\n", " 'RPS4Y1','RPS4Y2','SRY','TMSB4Y','TBL1Y','USP9Y','UTY','ZFY',\n", " 'AMELY','DAZ1','DAZ2','DAZ3','DAZ4','BPY2','CDY1','CDY2A',\n", " 'HSFY1','HSFY2','PRY','RBMY1B','RBMY1C','RBMY1E','RBMY1F',\n", " 'TXLNGY', # added: appeared in GSE151648_IRI consensus panel\n", " # Y-linked non-coding / pseudogenes commonly present on arrays\n", " 'TTTY14','TTTY15','TTTY10','TTTY3B','TTTY4','TTTY4B','TTTY4C',\n", " 'TTTY17A','TTTY17B','TTTY17C',\n", " # X-inactivation escapee used as female marker\n", " 'XIST','TSIX',\n", "}\n", "\n", "FILTER_SEX_GENES = True # set False to disable (e.g. if sex IS the biology)\n", "\n", "# ── Non-coding RNA / pseudogene filter ───────────────────────────────────────\n", "# Pseudogenes, lncRNAs, antisense transcripts, and snoRNAs are not\n", "# interpretable as liver biomarkers and add noise to consensus panels.\n", "#\n", "# Evidence from round-2 results (GSE151648_IRI):\n", "# RPS16P1, RPS4XP22 — ribosomal pseudogenes\n", "# MICOS10P3, MT1P3 — mitochondrial pseudogenes\n", "# GPX1P1, PARP4P1 — other pseudogenes\n", "# LINC01905 — uncharacterised lncRNA\n", "# MIR3147HG — microRNA host gene\n", "# ITGB1-DT — divergent transcript lncRNA\n", "# TGILR, PCDH9-AS2 — antisense / uncharacterised lncRNAs\n", "#\n", "# Patterns removed (applied after HGNC mapping):\n", "# *P[0-9]+ pseudogenes *-AS[0-9]* antisense lncRNAs\n", "# LINC* lincRNAs *-DT divergent transcripts\n", "# MIR* miRNA genes SNOR* small nucleolar RNAs\n", "# SNHG* snoRNA host genes\n", "#\n", "# Set FILTER_NONCODING = False to keep all features.\n", "FILTER_NONCODING = True\n", "\n", "import re as _re\n", "_NONCODING_PATTERNS = [\n", " _re.compile(r'.+P\\d+$'), # pseudogenes: RPS16P1, MT1P3, MICOS10P3\n", " _re.compile(r'^LINC\\d+'), # lincRNAs\n", " _re.compile(r'^MIR\\d+'), # miRNA host genes\n", " _re.compile(r'.+-AS\\d*$'), # antisense: PCDH9-AS2\n", " _re.compile(r'.+-DT$'), # divergent transcripts: ITGB1-DT\n", " _re.compile(r'^SNORD?\\d+'), # snoRNAs\n", " _re.compile(r'^SNHG\\d+'), # snoRNA host genes\n", "]\n", "\n", "def is_noncoding(gene_name):\n", " g = str(gene_name)\n", " return any(p.match(g) for p in _NONCODING_PATTERNS)\n", "\n", "# ── XGBoost ──────────────────────────────────────────────────────────────────\n", "XGB_N_ESTIMATORS = 100\n", "XGB_MAX_DEPTH = 4\n", "XGB_LEARNING_RATE = 0.05\n", "XGB_SUBSAMPLE = 0.8\n", "XGB_COLSAMPLE = 0.8\n", "\n", "print(f'Config loaded. EXPORT_DIR = {EXPORT_DIR}')\n", "print(f'TOP_N_GENES = {TOP_N_GENES} | VARIANCE_TOP_K_MAX = {VARIANCE_TOP_K_MAX}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# DATASETS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "DATASETS = {\n", " 'GSE112713': {\n", " 'expr_file': 'GSE112713_EXPR_ML.csv',\n", " 'pheno_file': 'GSE112713_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'keyword',\n", " 'pos_terms': ['ORGANOX'],\n", " 'neg_terms': ['CONTROL'],\n", " 'index_cleaner': None,\n", " },\n", " 'GSE276531': {\n", " 'expr_file': 'GSE276531_EXPR_ML.csv',\n", " 'pheno_file': 'GSE276531_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1.1',\n", " 'label_mode': 'keyword',\n", " # Original terms restored. Both orientations tested; both gave AUC~0.40\n", " # with consensus panels dominated by keratin/KRTAP genes with no liver\n", " # biology relevance. Reported as a negative result in the manuscript.\n", " 'pos_terms': ['MACHINE', 'HOPE', 'NMP'],\n", " 'neg_terms': ['COLD', 'CS', 'STATIC'],\n", " 'index_cleaner': lambda x: str(x).rsplit('_', 1)[0],\n", " },\n", " 'GSE12720': {\n", " 'expr_file': 'GSE12720_EXPR_ML.csv',\n", " 'pheno_file': 'GSE12720_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'keyword',\n", " 'pos_terms': ['POST-TRANSPLANT', 'POST TRANSPLANT', 'REPERFUSION', 'POST'],\n", " 'neg_terms': ['PRE-TRANSPLANT', 'PRE TRANSPLANT', 'BASELINE', 'PRE'],\n", " 'index_cleaner': None,\n", " },\n", " 'GSE14951': {\n", " 'expr_file': 'GSE14951_EXPR_ML.csv',\n", " 'pheno_file': 'GSE14951_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'keyword',\n", " 'pos_terms': ['DONOR'],\n", " 'neg_terms': ['NORMAL'],\n", " 'index_cleaner': None,\n", " },\n", " 'GSE15480': {\n", " 'expr_file': 'GSE15480_EXPR_ML.csv',\n", " 'pheno_file': 'GSE15480_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'keyword',\n", " 'pos_terms': ['REPERFUSION', 'POST', 'RECOVERY'],\n", " 'neg_terms': ['ISCHEMIA', 'PRE', 'BASELINE'],\n", " 'index_cleaner': None,\n", " },\n", " 'GSE151648_TIME': {\n", " 'expr_file': 'GSE151648_EXPR_ML_log2p1.csv',\n", " 'pheno_file': 'GSE151648_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'parsed_151648_time',\n", " 'index_cleaner': None,\n", " },\n", " 'GSE151648_IRI': {\n", " 'expr_file': 'GSE151648_EXPR_ML_log2p1.csv',\n", " 'pheno_file': 'GSE151648_PHENO_ML.csv',\n", " 'label_col': 'characteristics_ch1',\n", " 'label_mode': 'parsed_151648_iri',\n", " 'index_cleaner': None,\n", " },\n", "}\n", "\n", "print(f'Datasets configured: {list(DATASETS.keys())}')\n", "print()\n", "print('NOTE: GSE276531 — both label orientations tested, both AUC~0.40.')\n", "print(' No reliable signal found; reporting as a negative result.')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# UTILITY FUNCTIONS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import re, json as _json\n", "import mygene as _mygene\n", "_mg = _mygene.MyGeneInfo()\n", "\n", "# ── Label parsing ─────────────────────────────────────────────────────────────\n", "def extract_label_keyword(text, pos_terms, neg_terms):\n", " t = str(text).upper()\n", " if any(p.upper() in t for p in pos_terms): return 1\n", " if any(n.upper() in t for n in neg_terms): return 0\n", " return np.nan\n", "\n", "def parse_151648_time(text):\n", " t = str(text)\n", " if 'timepoint: Post-transplant' in t: return 1\n", " if 'timepoint: Pre-transplant' in t: return 0\n", " return np.nan\n", "\n", "def parse_151648_iri(text):\n", " t = str(text)\n", " if 'iri: IRI+' in t: return 1\n", " if 'iri: IRI-' in t: return 0\n", " return np.nan\n", "\n", "def build_target(pheno, cfg):\n", " label_col = cfg['label_col']\n", " mode = cfg['label_mode']\n", " if label_col not in pheno.columns:\n", " raise ValueError(f\"Label column '{label_col}' not found in phenotype.\")\n", " ph = pheno.copy()\n", " if mode == 'keyword':\n", " ph['Target'] = ph[label_col].apply(\n", " lambda x: extract_label_keyword(x, cfg['pos_terms'], cfg['neg_terms'])\n", " )\n", " elif mode == 'parsed_151648_time':\n", " ph['Target'] = ph[label_col].apply(parse_151648_time)\n", " elif mode == 'parsed_151648_iri':\n", " ph['Target'] = ph[label_col].apply(parse_151648_iri)\n", " else:\n", " raise ValueError(f\"Unknown label_mode='{mode}'.\")\n", " ph = ph.dropna(subset=['Target'])\n", " ph['Target'] = ph['Target'].astype(int)\n", " if ph['Target'].nunique() < 2:\n", " raise ValueError(f\"Target collapsed to one class (label_col='{label_col}', mode='{mode}').\")\n", " return ph\n", "\n", "# ── HGNC namespace detection ──────────────────────────────────────────────────\n", "def detect_id_namespace(cols):\n", " \"\"\"Classify column IDs as 'ensembl', 'reporter' (probe), or 'symbol'.\"\"\"\n", " sample = [str(c) for c in cols if str(c) != 'nan'][:50]\n", " ensembl = sum(1 for g in sample if g.startswith('ENSG'))\n", " # Affymetrix (_at suffix) OR pure-numeric (Illumina BGID)\n", " probe = sum(1 for g in sample if '_at' in g or re.fullmatch(r'\\d+', g))\n", " if ensembl / len(sample) > 0.5:\n", " return 'ensembl'\n", " if probe / len(sample) > 0.5:\n", " return 'reporter'\n", " return 'symbol'\n", "\n", "# ── Full-matrix HGNC mapping with disk cache ──────────────────────────────────\n", "def _load_cache(cache_path):\n", " if os.path.exists(cache_path):\n", " with open(cache_path) as f:\n", " return _json.load(f)\n", " return {}\n", "\n", "def _save_cache(mapping, cache_path):\n", " os.makedirs(os.path.dirname(cache_path), exist_ok=True)\n", " with open(cache_path, 'w') as f:\n", " _json.dump(mapping, f)\n", "\n", "def map_matrix_to_hgnc(expr, gse_id, namespace, cache_dir):\n", " \"\"\"\n", " Map all expression matrix columns to HGNC symbols.\n", "\n", " Strategy:\n", " 1. Check disk cache — avoids re-querying mygene on every run.\n", " 2. Query mygene in small chunks with retry + backoff to handle\n", " server disconnects (common for large Ensembl ID sets).\n", " 3. Rename columns; where multiple probes map to the same symbol,\n", " keep the probe with the highest variance (most informative).\n", " 4. Probes with no mapping are dropped (not biologically interpretable).\n", "\n", " Returns the remapped DataFrame and the mapping dict {probe: symbol}.\n", " \"\"\"\n", " import time\n", "\n", " if namespace == 'symbol':\n", " return expr, {c: c for c in expr.columns}\n", "\n", " # Snapshot original probe-indexed matrix BEFORE any renaming.\n", " # Used later for duplicate probe resolution (avoids re-reading CSV).\n", " orig_expr = expr.copy()\n", " cache_path = os.path.join(cache_dir, f'{gse_id}_probe2hgnc.json')\n", " cache = _load_cache(cache_path)\n", " scope = 'ensembl.gene' if namespace == 'ensembl' else 'reporter'\n", "\n", " # Identify IDs not yet in cache\n", " all_ids = [str(c) for c in expr.columns]\n", " to_query = [i for i in all_ids if i not in cache]\n", "\n", " if to_query:\n", " # Ensembl queries are slower server-side — use smaller chunks to\n", " # avoid connection timeouts on large datasets (e.g. GSE151648, 60k IDs).\n", " chunk_size = 300 if namespace == 'ensembl' else 500\n", " max_retries = 4\n", " save_every = 10 # save cache to disk every N chunks (resume on interrupt)\n", " n_chunks = (len(to_query) + chunk_size - 1) // chunk_size\n", "\n", " print(f' Querying mygene: {len(to_query):,} IDs, ' \n", " f'{n_chunks} chunks of {chunk_size} (scope={scope})...')\n", " print(f' Cache saves every {save_every} chunks — safe to interrupt and resume.')\n", "\n", " for chunk_idx, start in enumerate(range(0, len(to_query), chunk_size)):\n", " chunk = to_query[start:start + chunk_size]\n", "\n", " # ── Retry loop with exponential backoff ───────────────────────\n", " for attempt in range(max_retries):\n", " try:\n", " res = _mg.querymany(\n", " chunk, scopes=scope, fields='symbol',\n", " species='human', returnall=False, verbose=False\n", " )\n", " for r in res:\n", " q = r.get('query', '')\n", " if 'symbol' in r and not r.get('notfound', False):\n", " cache[q] = r['symbol']\n", " else:\n", " cache[q] = None # mark unmappable so it's skipped next run\n", " break # success — exit retry loop\n", " except Exception as e:\n", " wait = 2 ** attempt # 1s, 2s, 4s, 8s\n", " if attempt < max_retries - 1:\n", " print(f' Chunk {chunk_idx+1}/{n_chunks} failed '\n", " f'(attempt {attempt+1}/{max_retries}): {e}. ' \n", " f'Retrying in {wait}s...')\n", " time.sleep(wait)\n", " else:\n", " # All retries exhausted — mark chunk IDs as None so they\n", " # are skipped in future runs rather than re-queried forever\n", " print(f' Chunk {chunk_idx+1}/{n_chunks} permanently failed: {e}')\n", " for q in chunk:\n", " if q not in cache:\n", " cache[q] = None\n", "\n", " # Progress + periodic cache save\n", " if (chunk_idx + 1) % save_every == 0 or (chunk_idx + 1) == n_chunks:\n", " _save_cache(cache, cache_path)\n", " n_done = min(start + chunk_size, len(to_query))\n", " n_mapped = sum(1 for v in cache.values() if v is not None)\n", " pct = 100 * n_done / len(to_query)\n", " print(f' [{pct:5.1f}%] {n_done:,}/{len(to_query):,} queried ' \n", " f'| {n_mapped:,} mapped so far', end='\\r')\n", "\n", " print() # newline after progress bar\n", " n_mapped = sum(1 for v in cache.values() if v is not None)\n", " print(f' Done. {n_mapped:,}/{len(all_ids):,} probes mapped to HGNC symbols.')\n", "\n", " # Build rename map — skip unmappable probes\n", " rename = {c: cache.get(str(c)) for c in expr.columns}\n", " rename = {k: v for k, v in rename.items() if v is not None}\n", "\n", " # ── Duplicate probe resolution ────────────────────────────────────────\n", " # Multiple probes can map to the same HGNC symbol (common on Affymetrix).\n", " # Strategy: for each duplicated symbol, keep only the probe with the\n", " # highest variance across samples — it carries the most information.\n", " #\n", " # Approach: work entirely on orig_expr (pre-rename probe columns) to\n", " # avoid any index misalignment from the rename step.\n", " # 1. Group probe IDs by their target symbol using a pandas Series.\n", " # 2. For symbols with >1 probe, pick the highest-variance probe.\n", " # 3. Build a clean {probe: symbol} dict with one probe per symbol.\n", " # 4. Select and rename in a single operation — no concat needed.\n", "\n", " # sym_to_probes: symbol -> [probe1, probe2, ...] (only mapped probes)\n", " from collections import defaultdict\n", " sym_to_probes = defaultdict(list)\n", " for probe, sym in rename.items():\n", " if str(probe) in orig_expr.columns or probe in orig_expr.columns:\n", " sym_to_probes[sym].append(probe)\n", "\n", " best_probe_for_sym = {} # symbol -> single best probe ID\n", " for sym, probes in sym_to_probes.items():\n", " if len(probes) == 1:\n", " best_probe_for_sym[sym] = probes[0]\n", " else:\n", " # Pick probe with highest variance; fall back to first if all-NaN\n", " try:\n", " vars_ = orig_expr[[p for p in probes if p in orig_expr.columns]].var()\n", " best = vars_.idxmax()\n", " best_probe_for_sym[sym] = best if pd.notna(best) else probes[0]\n", " except Exception:\n", " best_probe_for_sym[sym] = probes[0]\n", "\n", " # Invert: probe -> symbol (guaranteed one-to-one)\n", " final_rename = {probe: sym for sym, probe in best_probe_for_sym.items()}\n", " n_unmapped = expr.shape[1] - len(rename)\n", " n_deduped = len(rename) - len(final_rename)\n", "\n", " # Select only the best probes and rename in one step\n", " valid_probes = [p for p in final_rename if p in orig_expr.columns]\n", " expr = orig_expr[valid_probes].rename(columns=final_rename)\n", "\n", " print(f' {gse_id}: {len(all_ids):,} probes → {expr.shape[1]:,} HGNC symbols '\n", " f'({n_unmapped:,} unmapped, {n_deduped:,} duplicates resolved)')\n", " return expr, rename\n", "\n", "# ── Data loading (with full HGNC remapping) ───────────────────────────────────\n", "def load_align_gse(gse_id, cfg, output_dir):\n", " expr_path = os.path.join(output_dir, cfg['expr_file'])\n", " pheno_path = os.path.join(output_dir, cfg['pheno_file'])\n", " if not os.path.exists(expr_path): raise FileNotFoundError(f\"{gse_id}: missing {expr_path}\")\n", " if not os.path.exists(pheno_path): raise FileNotFoundError(f\"{gse_id}: missing {pheno_path}\")\n", "\n", " expr = pd.read_csv(expr_path, index_col=0)\n", " pheno = pd.read_csv(pheno_path)\n", "\n", " if 'geo_accession' not in pheno.columns:\n", " raise ValueError(f\"{gse_id}: phenotype missing 'geo_accession'.\")\n", "\n", " if cfg.get('index_cleaner') is not None:\n", " expr.index = expr.index.map(cfg['index_cleaner'])\n", "\n", " pheno = pheno.set_index('geo_accession')\n", " common = expr.index.intersection(pheno.index)\n", " expr, pheno = expr.loc[common], pheno.loc[common]\n", "\n", " # ── AFFX control probe filter ─────────────────────────────────────────\n", " # Remove Affymetrix spike-in controls before mapping — they have no gene\n", " # symbol and waste mygene quota.\n", " affx_cols = [c for c in expr.columns if str(c).upper().startswith('AFFX')]\n", " if affx_cols:\n", " expr = expr.drop(columns=affx_cols)\n", " print(f' AFFX filter: removed {len(affx_cols)} control probe(s) from {gse_id}')\n", "\n", " # ── Full-matrix HGNC mapping ──────────────────────────────────────────\n", " # Detect namespace and remap ALL columns to HGNC symbols before modelling.\n", " # Cache stored in EXPORT_DIR/probe_caches/ — only queries mygene once.\n", " cache_dir = os.path.join(EXPORT_DIR, 'probe_caches')\n", " namespace = detect_id_namespace(expr.columns)\n", " expr, _ = map_matrix_to_hgnc(expr, gse_id, namespace, cache_dir)\n", "\n", " # ── Sex chromosome gene filter (now works on all datasets) ────────────\n", " # Applied after HGNC mapping so Y-linked and XIST genes are caught\n", " # regardless of original probe namespace.\n", " if FILTER_SEX_GENES:\n", " sex_cols = [c for c in expr.columns\n", " if str(c).upper() in {g.upper() for g in SEX_CHROMOSOME_GENES}]\n", " if sex_cols:\n", " expr = expr.drop(columns=sex_cols)\n", " print(f' Sex-gene filter: removed {len(sex_cols)} column(s): {sex_cols}')\n", "\n", " # ── Non-coding RNA / pseudogene filter ────────────────────────────────\n", " # Removes pseudogenes, lncRNAs, antisense transcripts, and snoRNAs.\n", " # Applied after HGNC mapping so patterns match gene symbols, not probe IDs.\n", " if FILTER_NONCODING:\n", " nc_cols = [c for c in expr.columns if is_noncoding(str(c))]\n", " if nc_cols:\n", " expr = expr.drop(columns=nc_cols)\n", " print(f' Non-coding filter: removed {len(nc_cols)} pseudogene/'\n", " f'lncRNA column(s) from {gse_id}')\n", "\n", " return expr, pheno\n", "\n", "# ── Safe CV folds ─────────────────────────────────────────────────────────────\n", "def safe_n_splits(y, desired=5, minimum=2):\n", " \"\"\"Never exceed smallest class count (prevents stratification errors).\"\"\"\n", " min_class = pd.Series(y).value_counts().min()\n", " n = int(min(desired, min_class))\n", " return n if n >= minimum else None\n", "\n", "# ── Metrics from binary predictions ───────────────────────────────────────────\n", "def compute_metrics(y_true, y_pred, y_prob):\n", " \"\"\"Returns AUC, sensitivity, specificity, PPV.\"\"\"\n", " try:\n", " auc = roc_auc_score(y_true, y_prob)\n", " except ValueError:\n", " auc = np.nan\n", " tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n", " sens = tp / (tp + fn) if (tp + fn) > 0 else np.nan\n", " spec = tn / (tn + fp) if (tn + fp) > 0 else np.nan\n", " ppv = tp / (tp + fp) if (tp + fp) > 0 else np.nan\n", " return dict(auc=auc, sensitivity=sens, specificity=spec, ppv=ppv)\n", "\n", "print('Utility functions loaded.')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# MODEL BUILDERS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "def build_en_cv_pipeline(inner_cv_folds):\n", " \"\"\"Elastic Net with internal CV — used ONCE per outer fold on training data.\"\"\"\n", " return Pipeline([\n", " ('imputer', SimpleImputer(strategy='median')),\n", " ('scaler', StandardScaler()),\n", " ('clf', LogisticRegressionCV(\n", " solver='saga',\n", " penalty='elasticnet',\n", " l1_ratios=EN_L1_RATIOS,\n", " Cs=EN_CS,\n", " cv=inner_cv_folds,\n", " scoring='roc_auc',\n", " max_iter=EN_MAX_ITER,\n", " class_weight='balanced',\n", " n_jobs=-1,\n", " random_state=RANDOM_STATE,\n", " ))\n", " ])\n", "\n", "\n", "def build_xgb_model(y_train):\n", " \"\"\"XGBoost with class imbalance correction per fold.\"\"\"\n", " counts = pd.Series(y_train).value_counts()\n", " scale_pos = counts.get(0, 1) / max(counts.get(1, 1), 1)\n", " return xgb.XGBClassifier(\n", " objective='binary:logistic',\n", " eval_metric='logloss',\n", " n_estimators=XGB_N_ESTIMATORS,\n", " max_depth=XGB_MAX_DEPTH,\n", " learning_rate=XGB_LEARNING_RATE,\n", " subsample=XGB_SUBSAMPLE,\n", " colsample_bytree=XGB_COLSAMPLE,\n", " scale_pos_weight=scale_pos,\n", " random_state=RANDOM_STATE,\n", " n_jobs=-1,\n", " verbosity=0,\n", " )\n", "\n", "\n", "def build_panel_lr(C=1.0, l1_ratio=0.5):\n", " \"\"\"Fixed logistic regression trained on panel-restricted features.\"\"\"\n", " return Pipeline([\n", " ('imputer', SimpleImputer(strategy='median')),\n", " ('scaler', StandardScaler()),\n", " ('clf', LogisticRegression(\n", " solver='saga',\n", " penalty='elasticnet',\n", " C=C,\n", " l1_ratio=l1_ratio,\n", " max_iter=5000,\n", " class_weight='balanced',\n", " random_state=RANDOM_STATE,\n", " ))\n", " ])\n", "\n", "print('Model builders loaded.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# FEATURE EXTRACTION HELPERS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import re\n", "\n", "def sanitize_feature_names(cols):\n", " \"\"\"\n", " XGBoost rejects names containing [, ], or <.\n", " Returns safe names and a reverse mapping.\n", " \"\"\"\n", " safe, reverse = [], {}\n", " seen = {}\n", " for c in cols:\n", " s = re.sub(r'[\\[\\]<>]', '_', str(c)).strip('_') or 'feat'\n", " if s in seen:\n", " seen[s] += 1\n", " s = f'{s}__{seen[s]}'\n", " else:\n", " seen[s] = 0\n", " safe.append(s)\n", " reverse[s] = str(c)\n", " return safe, reverse\n", "\n", "\n", "def en_top_genes(en_pipe, feature_names, top_n=20, threshold=0.01):\n", " \"\"\"Extract top genes from fitted EN pipeline by absolute coefficient.\"\"\"\n", " coefs = pd.Series(\n", " en_pipe.named_steps['clf'].coef_[0],\n", " index=feature_names\n", " )\n", " # Apply threshold; if nothing passes, relax to top_n by magnitude\n", " above = coefs[coefs.abs() > threshold]\n", " ranked = above if not above.empty else coefs\n", " ranked = ranked.reindex(ranked.abs().sort_values(ascending=False).index)\n", " return list(ranked.head(top_n).index)\n", "\n", "\n", "def xgb_top_genes(model, X_test_df, top_n=20):\n", " \"\"\"Compute SHAP values on the test fold and return top genes.\"\"\"\n", " try:\n", " explainer = shap.TreeExplainer(model)\n", " sv = explainer.shap_values(X_test_df)\n", " if isinstance(sv, list):\n", " sv = sv[1]\n", " importance = pd.Series(\n", " np.abs(sv).mean(axis=0),\n", " index=X_test_df.columns\n", " ).sort_values(ascending=False)\n", " return list(importance.head(top_n).index)\n", " except Exception:\n", " # Fall back to gain importance if SHAP fails\n", " imp = pd.Series(model.feature_importances_, index=X_test_df.columns)\n", " return list(imp.sort_values(ascending=False).head(top_n).index)\n", "\n", "\n", "print('Feature extraction helpers loaded.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# NESTED CV — CORE FUNCTION (parallelised across outer folds)\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "def _process_fold(fold_idx, train_idx, test_idx, X, y, valid_global_cols, top_n):\n", " \"\"\"\n", " Process a single outer fold.\n", " Runs inside a joblib worker — must be self-contained (no shared mutable state).\n", "\n", " IMPORTANT: n_jobs=1 is set on all inner estimators here because the outer\n", " folds are already parallelised by Parallel(n_jobs=-1). Using n_jobs=-1 inside\n", " each worker causes CPU over-subscription and is consistently slower.\n", " \"\"\"\n", " import warnings\n", " warnings.filterwarnings('ignore')\n", "\n", " X_train_df = X.iloc[train_idx]\n", " X_test_df = X.iloc[test_idx]\n", " y_train = pd.Series(y[train_idx])\n", " y_test = pd.Series(y[test_idx])\n", "\n", " # ── Drop all-NaN columns using training data only ──────────────────────\n", " valid_cols = X_train_df.columns[X_train_df.notna().any()].tolist()\n", " X_train_df = X_train_df[valid_cols]\n", " X_test_df = X_test_df[valid_cols]\n", "\n", " # ── Adaptive variance pre-filter — training data ONLY (no leakage) ─────\n", " # Cap at 10× n_train to keep SAGA well-conditioned for small cohorts.\n", " n_train = len(train_idx)\n", " adaptive_k = int(min(n_train * VARIANCE_TOP_K_MULTIPLIER, VARIANCE_TOP_K_MAX))\n", " if X_train_df.shape[1] > adaptive_k:\n", " var_scores = X_train_df.var(axis=0)\n", " top_var_cols = var_scores.nlargest(adaptive_k).index.tolist()\n", " X_train_df = X_train_df[top_var_cols]\n", " X_test_df = X_test_df[top_var_cols]\n", " valid_cols = top_var_cols\n", "\n", " # ── Inner CV folds ─────────────────────────────────────────────────────\n", " n_inner = safe_n_splits(y_train, desired=5, minimum=2)\n", " if n_inner is None:\n", " return None\n", "\n", " # ── STEP 1: Elastic Net — n_jobs=1 (outer loop is already parallel) ───\n", " en_pipe = Pipeline([\n", " ('imputer', SimpleImputer(strategy='median')),\n", " ('scaler', StandardScaler()),\n", " ('clf', LogisticRegressionCV(\n", " solver='saga',\n", " penalty='elasticnet',\n", " l1_ratios=EN_L1_RATIOS,\n", " Cs=EN_CS,\n", " cv=n_inner,\n", " scoring='roc_auc',\n", " max_iter=EN_MAX_ITER,\n", " class_weight='balanced',\n", " n_jobs=1, # ← was -1; avoids nested parallelism\n", " random_state=RANDOM_STATE,\n", " ))\n", " ])\n", " en_pipe.fit(X_train_df, y_train)\n", " best_C = float(en_pipe.named_steps['clf'].C_[0])\n", " best_l1 = float(en_pipe.named_steps['clf'].l1_ratio_[0])\n", " en_genes = en_top_genes(en_pipe, valid_cols, top_n=top_n)\n", "\n", " # ── STEP 2: XGBoost — n_jobs=1 (same reason) ──────────────────────────\n", " imputer = SimpleImputer(strategy='median')\n", " X_train_imp = pd.DataFrame(\n", " imputer.fit_transform(X_train_df), columns=valid_cols, index=X_train_df.index\n", " )\n", " X_test_imp = pd.DataFrame(\n", " imputer.transform(X_test_df), columns=valid_cols, index=X_test_df.index\n", " )\n", " safe_cols, safe_to_orig = sanitize_feature_names(valid_cols)\n", " X_train_xgb = X_train_imp.copy(); X_train_xgb.columns = safe_cols\n", " X_test_xgb = X_test_imp.copy(); X_test_xgb.columns = safe_cols\n", "\n", " counts = pd.Series(y_train).value_counts()\n", " scale_pos = counts.get(0, 1) / max(counts.get(1, 1), 1)\n", " xgb_model = xgb.XGBClassifier(\n", " objective='binary:logistic',\n", " eval_metric='logloss',\n", " n_estimators=XGB_N_ESTIMATORS,\n", " max_depth=XGB_MAX_DEPTH,\n", " learning_rate=XGB_LEARNING_RATE,\n", " subsample=XGB_SUBSAMPLE,\n", " colsample_bytree=XGB_COLSAMPLE,\n", " scale_pos_weight=scale_pos,\n", " random_state=RANDOM_STATE,\n", " n_jobs=1, # ← was -1; avoids nested parallelism\n", " verbosity=0,\n", " )\n", " xgb_model.fit(X_train_xgb, y_train)\n", "\n", " # ── Fast feature importance: use gain importance instead of full SHAP ──\n", " # SHAP TreeExplainer on 1500 features is the second-biggest bottleneck.\n", " # XGBoost's built-in gain importance gives near-identical gene rankings\n", " # for feature selection and is ~10-50× faster.\n", " # Switch back to SHAP by setting USE_SHAP = True in the config cell.\n", " USE_SHAP = False # set True to restore original SHAP behaviour\n", "\n", " if USE_SHAP:\n", " xgb_genes_safe = xgb_top_genes(xgb_model, X_test_xgb, top_n=top_n)\n", " else:\n", " imp = pd.Series(xgb_model.feature_importances_, index=safe_cols)\n", " xgb_genes_safe = list(imp.sort_values(ascending=False).head(top_n).index)\n", "\n", " xgb_genes = [safe_to_orig.get(g, g) for g in xgb_genes_safe]\n", "\n", " # ── STEP 3: EN ∪ XGB panel ────────────────────────────────────────────\n", " panel_genes = sorted(set(en_genes) | set(xgb_genes))\n", " panel_genes = [g for g in panel_genes if g in valid_cols]\n", "\n", " # ── STEP 4: Panel logistic regression → evaluate ──────────────────────\n", " if len(panel_genes) >= 3:\n", " panel_pipe = build_panel_lr(C=best_C, l1_ratio=best_l1)\n", " panel_pipe.fit(X_train_df[panel_genes], y_train)\n", " y_prob = panel_pipe.predict_proba(X_test_df[panel_genes])[:, 1]\n", " y_pred = panel_pipe.predict(X_test_df[panel_genes])\n", " else:\n", " y_prob = en_pipe.predict_proba(X_test_df)[:, 1]\n", " y_pred = en_pipe.predict(X_test_df)\n", "\n", " metrics = compute_metrics(y_test.values, y_pred, y_prob)\n", " metrics.update({'fold': fold_idx, 'n_panel_genes': len(panel_genes),\n", " 'best_C': best_C, 'best_l1_ratio': best_l1})\n", " return {\n", " 'metrics': metrics,\n", " 'panel_genes': panel_genes,\n", " 'y_true': y_test.values.tolist(), # for ROC curve aggregation\n", " 'y_prob': y_prob.tolist(),\n", " }\n", "\n", "\n", "def run_nested_cv(gse_id, X, y,\n", " outer_n_splits=OUTER_N_SPLITS,\n", " outer_n_repeats=OUTER_N_REPEATS,\n", " top_n=TOP_N_GENES,\n", " consensus_min_frac=CONSENSUS_MIN_FRAC):\n", " \"\"\"\n", " Parallelised nested cross-validation.\n", " Outer folds are processed in parallel via joblib.\n", " Feature selection happens strictly inside each outer fold (no data leakage).\n", " \"\"\"\n", " n_outer = safe_n_splits(y, desired=outer_n_splits, minimum=2)\n", " if n_outer is None:\n", " print(f' SKIP {gse_id}: not enough samples per class for outer CV.')\n", " return None\n", "\n", " outer_cv = RepeatedStratifiedKFold(\n", " n_splits=n_outer, n_repeats=outer_n_repeats, random_state=RANDOM_STATE\n", " )\n", "\n", " y_arr = np.array(y)\n", " splits = list(outer_cv.split(X.values, y_arr))\n", "\n", " print(f' Starting Parallel Nested CV ({len(splits)} folds)...')\n", " fold_results = Parallel(n_jobs=-1, verbose=5)(\n", " delayed(_process_fold)(i, tr, te, X, y_arr, list(X.columns), top_n)\n", " for i, (tr, te) in enumerate(splits)\n", " )\n", "\n", " fold_metrics = []\n", " gene_counts = defaultdict(int)\n", " all_y_true = [] # pooled across folds for ROC curve plotting\n", " all_y_prob = []\n", "\n", " for res in fold_results:\n", " if res is None:\n", " continue\n", " fold_metrics.append(res['metrics'])\n", " all_y_true.extend(res['y_true'])\n", " all_y_prob.extend(res['y_prob'])\n", " for g in res['panel_genes']:\n", " gene_counts[g] += 1\n", "\n", " n_folds_done = len(fold_metrics)\n", " if n_folds_done == 0:\n", " return None\n", "\n", " gene_freq = pd.Series(gene_counts).sort_values(ascending=False) / n_folds_done\n", " consensus = sorted(gene_freq[gene_freq >= consensus_min_frac].index.tolist())\n", "\n", " return dict(\n", " fold_metrics =fold_metrics,\n", " gene_selection_freq=gene_freq,\n", " consensus_panel =consensus,\n", " n_folds_completed =n_folds_done,\n", " y_true_pooled =np.array(all_y_true), # pooled for ROC plotting\n", " y_prob_pooled =np.array(all_y_prob),\n", " )\n", "\n", "print('Nested CV function loaded (parallel outer folds).')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# MAIN LOOP — run nested CV for every dataset\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "all_results = {} # gse_id -> nested CV result dict\n", "summary_rows = [] # for Table 2 (manuscript performance table)\n", "freq_tables = [] # per-dataset gene frequency tables\n", "consensus_rows = [] # per-dataset consensus panel summary\n", "\n", "for gse_id, cfg in DATASETS.items():\n", " print('\\n' + '='*70)\n", " print(f' Processing: {gse_id}')\n", " print('='*70)\n", "\n", " # ── Load data ─────────────────────────────────────────────────────────────\n", " try:\n", " expr_df, pheno_df = load_align_gse(gse_id, cfg, OUTPUT_DIR)\n", " except Exception as e:\n", " print(f' SKIP (load error): {e}')\n", " continue\n", "\n", " try:\n", " pheno_t = build_target(pheno_df, cfg)\n", " except ValueError as e:\n", " print(f' SKIP (label error): {e}')\n", " continue\n", "\n", " X = expr_df.loc[pheno_t.index]\n", " y = pheno_t['Target']\n", "\n", " n_samples = X.shape[0]\n", " print(f' Samples: {n_samples} | Genes: {X.shape[1]}')\n", " print(f' Class counts: {y.value_counts().to_dict()}')\n", " if n_samples < MIN_SAMPLES_RELIABLE:\n", " print(f' ⚠️ WARNING: n={n_samples} is below the reliability threshold ({MIN_SAMPLES_RELIABLE}).')\n", " print(f' Results will be reported but should be treated as exploratory.')\n", "\n", " # ── Run nested CV ─────────────────────────────────────────────────────────\n", " result = run_nested_cv(gse_id, X, y)\n", " if result is None:\n", " continue\n", "\n", " all_results[gse_id] = result\n", "\n", " # ── Summarise fold metrics ─────────────────────────────────────────────────\n", " metrics_df = pd.DataFrame(result['fold_metrics'])\n", "\n", " means = metrics_df[['auc','sensitivity','specificity','ppv','n_panel_genes']].mean()\n", " sds = metrics_df[['auc','sensitivity','specificity','ppv','n_panel_genes']].std()\n", "\n", " mean_auc = means['auc']\n", " auc_flag = ' ⚠️ BELOW CHANCE — check label parsing' if mean_auc < 0.5 else ''\n", " print(f\"\\n Nested CV results ({result['n_folds_completed']} folds):{auc_flag}\")\n", " print(f\" AUC : {means['auc']:.3f} ± {sds['auc']:.3f}\")\n", " print(f\" Sensitivity : {means['sensitivity']:.3f} ± {sds['sensitivity']:.3f}\")\n", " print(f\" Specificity : {means['specificity']:.3f} ± {sds['specificity']:.3f}\")\n", " print(f\" PPV : {means['ppv']:.3f} ± {sds['ppv']:.3f}\")\n", " print(f\" Panel size : {means['n_panel_genes']:.1f} ± {sds['n_panel_genes']:.1f} genes/fold\")\n", " print(f\" Consensus : {len(result['consensus_panel'])} genes (≥{int(CONSENSUS_MIN_FRAC*100)}% folds)\")\n", " print(f\" Consensus genes: {result['consensus_panel']}\")\n", "\n", " summary_rows.append({\n", " 'GSE': gse_id,\n", " 'n_samples': X.shape[0],\n", " 'n_genes': X.shape[1],\n", " 'n_folds': result['n_folds_completed'],\n", " 'AUC_mean': round(means['auc'], 3),\n", " 'AUC_sd': round(sds['auc'], 3),\n", " 'Sensitivity_mean': round(means['sensitivity'], 3),\n", " 'Sensitivity_sd': round(sds['sensitivity'], 3),\n", " 'Specificity_mean': round(means['specificity'], 3),\n", " 'Specificity_sd': round(sds['specificity'], 3),\n", " 'PPV_mean': round(means['ppv'], 3),\n", " 'PPV_sd': round(sds['ppv'], 3),\n", " 'Panel_size_mean': round(means['n_panel_genes'], 1),\n", " 'Panel_size_sd': round(sds['n_panel_genes'], 1),\n", " 'Consensus_panel_n_genes': len(result['consensus_panel']),\n", " 'Consensus_panel_genes': ';'.join(result['consensus_panel']),\n", " })\n", "\n", " # ── Gene frequency table ───────────────────────────────────────────────────\n", " freq_df = result['gene_selection_freq'].reset_index()\n", " freq_df.columns = ['gene', 'selection_freq']\n", " freq_df.insert(0, 'GSE', gse_id)\n", " freq_tables.append(freq_df)\n", "\n", " consensus_rows.append({\n", " 'GSE': gse_id,\n", " 'consensus_n': len(result['consensus_panel']),\n", " 'consensus_genes': ';'.join(result['consensus_panel']),\n", " })\n", "\n", "print('\\n' + '='*70)\n", "print('ALL DATASETS PROCESSED')\n", "print('='*70)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# CONSENSUS PANEL HGNC SUMMARY\n", "# All modelling ran on HGNC symbols (mapped at load time), so consensus panels\n", "# are already in HGNC namespace. This cell just formats and saves the summary.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "hgnc_panels = {}\n", "hgnc_rows = []\n", "\n", "for gse_id, result in all_results.items():\n", " panel = result['consensus_panel']\n", " # Panels are already HGNC symbols — no post-hoc mapping needed.\n", " # Filter out any residual unmapped IDs (fallbacks from mygene failures).\n", " clean = [g for g in panel if not re.fullmatch(r'\\d+', str(g))\n", " and not str(g).endswith('_at')\n", " and not str(g).startswith('ENSG')]\n", " hgnc_panels[gse_id] = sorted(set(clean))\n", "\n", " n_removed = len(panel) - len(clean)\n", " if n_removed:\n", " print(f' {gse_id}: removed {n_removed} residual unmapped ID(s) from panel')\n", "\n", " hgnc_rows.append({\n", " 'GSE': gse_id,\n", " 'consensus_n': len(clean),\n", " 'consensus_genes_HGNC': ';'.join(sorted(set(clean))),\n", " })\n", "\n", "# Update summary_rows with cleaned HGNC panels\n", "for row in summary_rows:\n", " gid = row['GSE']\n", " if gid in hgnc_panels:\n", " row['Consensus_panel_genes_HGNC'] = ';'.join(hgnc_panels[gid])\n", " row['Consensus_panel_n_genes'] = len(hgnc_panels[gid])\n", "\n", "hgnc_df = pd.DataFrame(hgnc_rows)\n", "out_path = os.path.join(EXPORT_DIR, 'table_consensus_panels_HGNC.csv')\n", "hgnc_df.to_csv(out_path, index=False)\n", "print(f'\\nHGNC consensus panels saved to {out_path}')\n", "print()\n", "print(hgnc_df[['GSE','consensus_n','consensus_genes_HGNC']].to_string(index=False))\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# SUMMARY TABLE \n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "summary_df = pd.DataFrame(summary_rows)\n", "\n", "print('\\n── Nested CV Performance Summary (Table 2) ──────────────────────────────')\n", "try:\n", " print(summary_df.to_markdown(index=False))\n", "except Exception:\n", " print(summary_df.to_string(index=False))\n", "\n", "# Save\n", "summary_path = os.path.join(EXPORT_DIR, 'table2_nested_cv_performance.csv')\n", "summary_df.to_csv(summary_path, index=False)\n", "print(f'\\nSaved: {summary_path}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# GENE FREQUENCY & CONSENSUS TABLES\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "if freq_tables:\n", " all_freq_df = pd.concat(freq_tables, ignore_index=True)\n", " freq_path = os.path.join(EXPORT_DIR, 'table_gene_selection_frequency.csv')\n", " all_freq_df.to_csv(freq_path, index=False)\n", " print(f'Gene selection frequency saved: {freq_path}')\n", "\n", "if consensus_rows:\n", " consensus_df = pd.DataFrame(consensus_rows)\n", " consensus_path = os.path.join(EXPORT_DIR, 'table_consensus_panels.csv')\n", " consensus_df.to_csv(consensus_path, index=False)\n", " print(f'Consensus panels saved: {consensus_path}')\n", " print('\\n── Consensus panels per dataset ─────────────────────────────────────────')\n", " try:\n", " print(consensus_df.to_markdown(index=False))\n", " except Exception:\n", " print(consensus_df.to_string(index=False))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# CROSS-DATASET CONSERVED GENES\n", "# (intersection of consensus panels across all datasets)\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "local_sets = [\n", " set(r['consensus_panel'])\n", " for r in all_results.values()\n", " if r['consensus_panel']\n", "]\n", "\n", "if local_sets:\n", " cross_consensus = sorted(local_sets[0].intersection(*local_sets[1:]))\n", "else:\n", " cross_consensus = []\n", "\n", "print(f'Cross-dataset conserved genes ({len(cross_consensus)} total):')\n", "print(cross_consensus)\n", "\n", "cross_df = pd.DataFrame({'conserved_gene': cross_consensus})\n", "cross_path = os.path.join(EXPORT_DIR, 'table_cross_dataset_conserved_genes.csv')\n", "cross_df.to_csv(cross_path, index=False)\n", "print(f'\\nSaved: {cross_path}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# VISUALISATIONS\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import matplotlib\n", "matplotlib.use('Agg')\n", "import matplotlib.pyplot as plt\n", "\n", "FIG_DIR = os.path.join(EXPORT_DIR, 'figures_nested_cv')\n", "os.makedirs(FIG_DIR, exist_ok=True)\n", "\n", "\n", "# ── AUC mean ± SD per dataset ──────────────────────────────────────\n", "if not summary_df.empty:\n", " fig, ax = plt.subplots(figsize=(9, 4))\n", " x = np.arange(len(summary_df))\n", " ax.bar(x, summary_df['AUC_mean'], yerr=summary_df['AUC_sd'],\n", " capsize=4, color='0.3', edgecolor='black', linewidth=0.8)\n", " ax.axhline(0.5, linestyle='--', color='0.6', linewidth=0.8)\n", " ax.set_xticks(x)\n", " ax.set_xticklabels(summary_df['GSE'], rotation=45, ha='right')\n", " ax.set_ylabel('AUC (nested CV, mean ± SD)')\n", " ax.set_ylim(0.4, 1.05)\n", " ax.set_title('Nested CV — EN∪XGB panel performance per dataset')\n", " plt.tight_layout()\n", " fig.savefig(os.path.join(FIG_DIR, 'fig1_nested_cv_auc.svg'), bbox_inches='tight')\n", " plt.close()\n", " print('Saved: fig1_nested_cv_auc.svg')\n", "\n", "\n", "# ── All four metrics per dataset ────────────────────────────────────\n", "metrics_map = {\n", " 'AUC': ('AUC_mean', 'AUC_sd'),\n", " 'Sensitivity': ('Sensitivity_mean', 'Sensitivity_sd'),\n", " 'Specificity': ('Specificity_mean', 'Specificity_sd'),\n", " 'PPV': ('PPV_mean', 'PPV_sd'),\n", "}\n", "\n", "if not summary_df.empty:\n", " fig, axes = plt.subplots(1, 4, figsize=(16, 4), sharey=False)\n", " colors = ['0.2', '0.4', '0.55', '0.7']\n", "\n", " for ax, (label, (mu_col, sd_col)), color in zip(axes, metrics_map.items(), colors):\n", " x = np.arange(len(summary_df))\n", " ax.bar(x, summary_df[mu_col], yerr=summary_df[sd_col],\n", " capsize=3, color=color, edgecolor='black', linewidth=0.7)\n", " ax.set_xticks(x)\n", " ax.set_xticklabels(summary_df['GSE'], rotation=45, ha='right', fontsize=7)\n", " ax.set_ylabel(label)\n", " ax.set_ylim(0, 1.1)\n", " ax.set_title(label)\n", "\n", " fig.suptitle('Nested CV — four metrics per dataset', y=1.02)\n", " plt.tight_layout()\n", " fig.savefig(os.path.join(FIG_DIR, 'fig2_nested_cv_all_metrics.svg'), bbox_inches='tight')\n", " plt.close()\n", " print('Saved: fig2_nested_cv_all_metrics.svg')\n", "\n", "\n", "# ── Gene selection frequency (top 20) per dataset ───────────────────\n", "for gse_id, result in all_results.items():\n", " freq = result['gene_selection_freq'].head(20)\n", " if freq.empty:\n", " continue\n", " fig, ax = plt.subplots(figsize=(5, max(3, 0.3 * len(freq))))\n", " y_pos = np.arange(len(freq))[::-1]\n", " ax.barh(y_pos, freq.values[::-1], color='0.3', edgecolor='black', linewidth=0.5)\n", " ax.set_yticks(y_pos)\n", " ax.set_yticklabels(freq.index[::-1], fontsize=8)\n", " ax.axvline(CONSENSUS_MIN_FRAC, linestyle='--', color='0.5', linewidth=0.8,\n", " label=f'Consensus threshold ({int(CONSENSUS_MIN_FRAC*100)}%)')\n", " ax.set_xlabel('Selection frequency across nested CV folds')\n", " ax.set_title(f'{gse_id} — Gene selection frequency')\n", " ax.legend(fontsize=7)\n", " plt.tight_layout()\n", " fig.savefig(os.path.join(FIG_DIR, f'fig3_{gse_id}_gene_freq.svg'), bbox_inches='tight')\n", " plt.close()\n", "\n", "print(f'All figures saved to: {FIG_DIR}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# Within-dataset ROC curves (nested CV, panel only)\n", "#\n", "# one pooled ROC curve per dataset, aggregated across all 15 outer\n", "# folds. Pooling concatenates y_true and y_prob from every fold, giving a\n", "# single curve that reflects the classifier's full held-out performance\n", "# distribution. This is the standard approach for nested CV ROC figures.\n", "#\n", "# Datasets flagged as exploratory (n < MIN_SAMPLES_RELIABLE) are shown with\n", "# a dashed border and an asterisk in the title.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import matplotlib\n", "matplotlib.use('Agg')\n", "import matplotlib.pyplot as plt\n", "from sklearn.metrics import roc_curve, auc as sklearn_auc\n", "\n", "# ── layout ────────────────────────────────────────────────────────────────────\n", "DATASET_ORDER = [\n", " 'GSE112713', 'GSE276531', 'GSE12720',\n", " 'GSE14951', 'GSE15480', 'GSE151648_TIME', 'GSE151648_IRI',\n", "]\n", "EXPLORATORY = {'GSE14951', 'GSE15480', 'GSE276531'} # flagged in output\n", "\n", "n_plots = len(DATASET_ORDER)\n", "n_cols = 3\n", "n_rows = (n_plots + n_cols - 1) // n_cols\n", "\n", "fig, axes = plt.subplots(n_rows, n_cols, figsize=(4.5 * n_cols, 4 * n_rows))\n", "axes_flat = axes.flatten()\n", "\n", "for ax_idx, gse_id in enumerate(DATASET_ORDER):\n", " ax = axes_flat[ax_idx]\n", "\n", " if gse_id not in all_results or all_results[gse_id] is None:\n", " ax.set_visible(False)\n", " continue\n", "\n", " result = all_results[gse_id]\n", " y_true = result.get('y_true_pooled')\n", " y_prob = result.get('y_prob_pooled')\n", " mean_auc = np.mean([f['auc'] for f in result['fold_metrics']])\n", " sd_auc = np.std( [f['auc'] for f in result['fold_metrics']])\n", " n_samples = int(sum(f.get('n_panel_genes', 0) > 0 for f in result['fold_metrics']))\n", "\n", " exploratory = gse_id in EXPLORATORY\n", " label_suffix = '*' if exploratory else ''\n", "\n", " if y_true is not None and y_prob is not None and len(np.unique(y_true)) > 1:\n", " fpr, tpr, _ = roc_curve(y_true, y_prob)\n", " pooled_auc = sklearn_auc(fpr, tpr)\n", "\n", " line_color = '#888888' if exploratory else '#333333'\n", " ax.plot(fpr, tpr,\n", " color=line_color,\n", " lw=2.0,\n", " linestyle='--' if exploratory else '-',\n", " label=f'Panel (AUC={pooled_auc:.2f})')\n", " else:\n", " ax.text(0.5, 0.5, 'Insufficient data', ha='center', va='center',\n", " transform=ax.transAxes, fontsize=9, color='grey')\n", "\n", " ax.plot([0, 1], [0, 1], 'k--', lw=0.8, alpha=0.5)\n", " ax.set_xlim([-0.02, 1.02])\n", " ax.set_ylim([-0.02, 1.05])\n", " ax.set_xlabel('False Positive Rate', fontsize=9)\n", " ax.set_ylabel('True Positive Rate', fontsize=9)\n", "\n", " # Title: dataset name + mean ± SD AUC from nested CV folds\n", " title = f'{gse_id}{label_suffix}\\nAUC = {mean_auc:.3f} ± {sd_auc:.3f}'\n", " ax.set_title(title, fontsize=9, pad=4)\n", "\n", " if exploratory:\n", " for spine in ax.spines.values():\n", " spine.set_edgecolor('#888888')\n", " spine.set_linestyle('--')\n", " spine.set_linewidth(1.2)\n", "\n", " ax.legend(fontsize=8, loc='lower right', framealpha=0.9)\n", " ax.tick_params(labelsize=8)\n", "\n", "# Hide any unused axes\n", "for ax_idx in range(len(DATASET_ORDER), len(axes_flat)):\n", " axes_flat[ax_idx].set_visible(False)\n", "\n", "# Add footnote\n", "fig.text(0.01, -0.01,\n", " '* Exploratory only: small sample size (n<30); results should be interpreted with caution.',\n", " fontsize=7, color='grey', ha='left')\n", "\n", "fig.suptitle('(B) Within-dataset ROC — EN∪XGB panel (nested CV, pooled across 15 folds)',\n", " fontsize=11, fontweight='bold', y=1.01)\n", "plt.tight_layout()\n", "out_path = os.path.join(FIG_DIR, 'fig2B_roc_curves_panel_nested_cv.svg')\n", "fig.savefig(out_path, bbox_inches='tight')\n", "plt.close()\n", "print(f'Saved: {out_path}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# Cross-dataset transfer heatmap (updated consensus panels)\n", "#\n", "# For each train→test pair, trains a logistic regression on the train dataset\n", "# using only genes in the intersection of the two consensus panels, then\n", "# evaluates on the test dataset.\n", "#\n", "# This is DIFFERENT from LODO: LODO combines all training datasets;\n", "# this heatmap shows pairwise panel-to-panel signal transfer.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.metrics import roc_auc_score\n", "\n", "# Reload all expression matrices (needed for cross-dataset evaluation)\n", "print('Loading expression matrices for cross-dataset transfer...')\n", "expr_cache = {}\n", "for gse_id, cfg in DATASETS.items():\n", " try:\n", " expr_df, pheno_df = load_align_gse(gse_id, cfg, OUTPUT_DIR)\n", " pheno_df = build_target(pheno_df, cfg)\n", " common = expr_df.index.intersection(pheno_df.index)\n", " expr_cache[gse_id] = {\n", " 'X': expr_df.loc[common],\n", " 'y': pheno_df.loc[common, 'Target'].values,\n", " }\n", " print(f' {gse_id}: {len(common)} samples, {expr_df.shape[1]:,} genes')\n", " except Exception as e:\n", " print(f' SKIP {gse_id}: {e}')\n", "\n", "# ── Build transfer matrix ─────────────────────────────────────────────────────\n", "gse_ids = [g for g in DATASET_ORDER if g in all_results and all_results[g] is not None\n", " and g in expr_cache]\n", "n = len(gse_ids)\n", "auc_mat = np.full((n, n), np.nan)\n", "\n", "MIN_SHARED = 3 # skip pairs with fewer shared panel genes\n", "\n", "for i, train_gse in enumerate(gse_ids):\n", " train_panel = all_results[train_gse]['consensus_panel']\n", " X_train_all = expr_cache[train_gse]['X']\n", " y_train_all = expr_cache[train_gse]['y']\n", "\n", " for j, test_gse in enumerate(gse_ids):\n", " if i == j:\n", " # Diagonal: use within-dataset nested CV AUC\n", " auc_mat[i, j] = np.mean([f['auc'] for f in all_results[train_gse]['fold_metrics']])\n", " continue\n", "\n", " test_panel = all_results[test_gse]['consensus_panel']\n", " shared_genes = [g for g in train_panel\n", " if g in test_panel\n", " and g in X_train_all.columns\n", " and g in expr_cache[test_gse]['X'].columns]\n", "\n", " if len(shared_genes) < MIN_SHARED:\n", " auc_mat[i, j] = np.nan\n", " continue\n", "\n", " X_train = X_train_all[shared_genes]\n", " y_train = y_train_all\n", " X_test = expr_cache[test_gse]['X'][shared_genes]\n", " y_test = expr_cache[test_gse]['y']\n", "\n", " if len(np.unique(y_train)) < 2 or len(np.unique(y_test)) < 2:\n", " continue\n", "\n", " try:\n", " pipe = Pipeline([\n", " ('imp', SimpleImputer(strategy='median')),\n", " ('scaler', StandardScaler()),\n", " ('clf', LogisticRegression(\n", " solver='saga', penalty='l2', C=1.0,\n", " class_weight='balanced', max_iter=2000,\n", " random_state=RANDOM_STATE\n", " ))\n", " ])\n", " pipe.fit(X_train, y_train)\n", " y_prob = pipe.predict_proba(X_test)[:, 1]\n", " auc_mat[i, j] = roc_auc_score(y_test, y_prob)\n", " except Exception as e:\n", " print(f' Transfer {train_gse}→{test_gse} failed: {e}')\n", "\n", "# ── Plot heatmap ──────────────────────────────────────────────────────────────\n", "fig, ax = plt.subplots(figsize=(7, 5.5))\n", "\n", "# Mask NaN cells\n", "import numpy as np\n", "masked = np.ma.masked_invalid(auc_mat)\n", "cmap = plt.cm.YlOrRd.copy()\n", "cmap.set_bad('#EEEEEE')\n", "\n", "im = ax.imshow(masked, vmin=0.5, vmax=1.0, cmap=cmap, aspect='auto')\n", "\n", "# Annotate cells\n", "for i in range(n):\n", " for j in range(n):\n", " val = auc_mat[i, j]\n", " if not np.isnan(val):\n", " text_color = 'white' if val > 0.85 else 'black'\n", " weight = 'bold' if i == j else 'normal'\n", " ax.text(j, i, f'{val:.2f}', ha='center', va='center',\n", " fontsize=9, color=text_color, fontweight=weight)\n", " else:\n", " ax.text(j, i, '—', ha='center', va='center',\n", " fontsize=9, color='#AAAAAA')\n", "\n", "ax.set_xticks(range(n))\n", "ax.set_yticks(range(n))\n", "short_labels = [g.replace('GSE151648_', '651648\\n') for g in gse_ids]\n", "ax.set_xticklabels(gse_ids, rotation=45, ha='right', fontsize=8)\n", "ax.set_yticklabels(gse_ids, fontsize=8)\n", "ax.set_xlabel('Test dataset (panel-restricted)', fontsize=9)\n", "ax.set_ylabel('Train dataset (panel-defined)', fontsize=9)\n", "\n", "cbar = fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n", "cbar.set_label('AUC (train on row, test on column)', fontsize=8)\n", "cbar.ax.tick_params(labelsize=8)\n", "\n", "ax.set_title('(C) Cross-dataset transfer using consensus panels (EN∪XGB)',\n", " fontsize=10, fontweight='bold', pad=8)\n", "plt.tight_layout()\n", "\n", "out_path = os.path.join(FIG_DIR, 'fig3C_cross_dataset_transfer_heatmap.svg')\n", "fig.savefig(out_path, bbox_inches='tight')\n", "plt.close()\n", "print(f'Saved: {out_path}')\n", "print()\n", "# Print matrix as table for reference\n", "print('Transfer AUC matrix:')\n", "header = '\\t'.join([''] + gse_ids)\n", "print(header)\n", "for i, gse in enumerate(gse_ids):\n", " row = '\\t'.join([gse] + [f'{auc_mat[i,j]:.2f}' if not np.isnan(auc_mat[i,j]) else '—' for j in range(n)])\n", " print(row)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# Panel overlap heatmap (updated consensus panels)\n", "# Shows number of shared genes between each pair of consensus panels.\n", "# Diagonal = panel size for each dataset.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "gse_ids_with_panels = [g for g in DATASET_ORDER\n", " if g in all_results and all_results[g] is not None\n", " and len(all_results[g]['consensus_panel']) > 0]\n", "\n", "n = len(gse_ids_with_panels)\n", "overlap_mat = np.zeros((n, n), dtype=int)\n", "\n", "for i, gse_i in enumerate(gse_ids_with_panels):\n", " panel_i = set(all_results[gse_i]['consensus_panel'])\n", " for j, gse_j in enumerate(gse_ids_with_panels):\n", " panel_j = set(all_results[gse_j]['consensus_panel'])\n", " overlap_mat[i, j] = len(panel_i & panel_j)\n", "\n", "# ── Plot ──────────────────────────────────────────────────────────────────────\n", "fig, ax = plt.subplots(figsize=(7, 5.5))\n", "cmap_ov = plt.cm.Greys\n", "im = ax.imshow(overlap_mat, cmap=cmap_ov, vmin=0,\n", " vmax=max(overlap_mat.diagonal()) if overlap_mat.diagonal().max() > 0 else 1,\n", " aspect='auto')\n", "\n", "for i in range(n):\n", " for j in range(n):\n", " val = overlap_mat[i, j]\n", " text_color = 'white' if val > overlap_mat.max() * 0.6 else 'black'\n", " weight = 'bold' if i == j else 'normal'\n", " ax.text(j, i, str(val), ha='center', va='center',\n", " fontsize=10, color=text_color, fontweight=weight)\n", "\n", "ax.set_xticks(range(n))\n", "ax.set_yticks(range(n))\n", "ax.set_xticklabels(gse_ids_with_panels, rotation=45, ha='right', fontsize=8)\n", "ax.set_yticklabels(gse_ids_with_panels, fontsize=8)\n", "ax.set_xlabel('Test dataset (panel genes)', fontsize=9)\n", "ax.set_ylabel('Train dataset (panel genes)', fontsize=9)\n", "\n", "cbar = fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n", "cbar.set_label('Number of shared panel genes', fontsize=8)\n", "cbar.ax.tick_params(labelsize=8)\n", "\n", "ax.set_title('(C) Overlap of EN∪XGB consensus panels across datasets',\n", " fontsize=10, fontweight='bold', pad=8)\n", "plt.tight_layout()\n", "\n", "out_path = os.path.join(FIG_DIR, 'fig4C_panel_overlap_heatmap.svg')\n", "fig.savefig(out_path, bbox_inches='tight')\n", "plt.close()\n", "print(f'Saved: {out_path}')\n", "\n", "# Print shared genes for pairs with overlap > 0\n", "print('\\nShared genes between panels (pairs with >0 overlap):')\n", "for i, gse_i in enumerate(gse_ids_with_panels):\n", " panel_i = set(all_results[gse_i]['consensus_panel'])\n", " for j, gse_j in enumerate(gse_ids_with_panels):\n", " if j <= i:\n", " continue\n", " panel_j = set(all_results[gse_j]['consensus_panel'])\n", " shared = sorted(panel_i & panel_j)\n", " if shared:\n", " print(f' {gse_i} ∩ {gse_j} ({len(shared)}): {\", \".join(shared)}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# Batch-effect PCA (before and after HGNC harmonisation)\n", "#\n", "# Panel A (Before): Heatmap of raw feature-namespace overlap across datasets.\n", "# Each dataset uses a different probe namespace (Affymetrix probe IDs, Ensembl\n", "# gene IDs, or HGNC symbols), so the number of shared raw column names between\n", "# platform-heterogeneous datasets is near-zero. This demonstrates why cross-\n", "# dataset comparison requires HGNC harmonisation first.\n", "#\n", "# Panel B (After): PCA of the combined HGNC-harmonised expression matrix.\n", "# Samples coloured by dataset. Clustering by dataset indicates residual\n", "# platform-specific batch effects, motivating the within-dataset nested CV\n", "# analysis strategy rather than a pooled cross-platform model.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import matplotlib\n", "matplotlib.use('Agg')\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as mpatches\n", "import numpy as np\n", "import pandas as pd\n", "import os\n", "from sklearn.decomposition import PCA\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.impute import SimpleImputer\n", "\n", "FIG_DIR = os.path.join(EXPORT_DIR, 'figures_nested_cv')\n", "os.makedirs(FIG_DIR, exist_ok=True)\n", "\n", "DATASET_ORDER = [\n", " 'GSE112713', 'GSE276531', 'GSE12720',\n", " 'GSE14951', 'GSE15480', 'GSE151648_TIME', 'GSE151648_IRI',\n", "]\n", "\n", "# ── Platform annotation for axis labels ──────────────────────────────────────\n", "PLATFORM_LABEL = {\n", " 'GSE112713': 'GPL570 (Affy)',\n", " 'GSE276531': 'GPL570 (Affy)',\n", " 'GSE12720': 'GPL570 (Affy)',\n", " 'GSE14951': 'GPL6244 (Affy)',\n", " 'GSE15480': 'GPL6244 (Affy)',\n", " 'GSE151648_TIME': 'Illumina RNA-seq',\n", " 'GSE151648_IRI': 'Illumina RNA-seq',\n", "}\n", "\n", "# Colour palette — one colour per dataset\n", "PALETTE = {\n", " 'GSE112713': '#1F77B4',\n", " 'GSE276531': '#FF7F0E',\n", " 'GSE12720': '#2CA02C',\n", " 'GSE14951': '#D62728',\n", " 'GSE15480': '#9467BD',\n", " 'GSE151648_TIME': '#8C564B',\n", " 'GSE151648_IRI': '#E377C2',\n", "}\n", "\n", "# ── PANEL A: raw feature-namespace overlap heatmap ───────────────────────────\n", "print('Loading raw expression matrices (no HGNC mapping)...')\n", "raw_cols = {} # gse_id -> set of raw column names\n", "for gse_id, cfg in DATASETS.items():\n", " try:\n", " expr_path = os.path.join(OUTPUT_DIR, cfg['expr_file'])\n", " raw_expr = pd.read_csv(expr_path, index_col=0, nrows=1) # headers only\n", " # Remove AFFX probes as in the pipeline\n", " cols = [c for c in raw_expr.columns if not str(c).upper().startswith('AFFX')]\n", " raw_cols[gse_id] = set(cols)\n", " ns = 'ENSG' if any(str(c).startswith('ENSG') for c in cols[:20]) else 'ProbeID/Symbol'\n", " print(f' {gse_id}: {len(cols):,} raw features [{ns}]')\n", " except Exception as e:\n", " print(f' SKIP {gse_id}: {e}')\n", "\n", "gse_ids_raw = [g for g in DATASET_ORDER if g in raw_cols]\n", "n = len(gse_ids_raw)\n", "overlap_mat = np.zeros((n, n), dtype=int)\n", "for i, gi in enumerate(gse_ids_raw):\n", " for j, gj in enumerate(gse_ids_raw):\n", " overlap_mat[i, j] = len(raw_cols[gi] & raw_cols[gj])\n", "\n", "# ── PANEL B: HGNC-harmonised combined PCA ────────────────────────────────────\n", "print('\\nLoading HGNC-harmonised expression matrices...')\n", "hgnc_matrices = {}\n", "for gse_id, cfg in DATASETS.items():\n", " try:\n", " expr_df, pheno_df = load_align_gse(gse_id, cfg, OUTPUT_DIR)\n", " hgnc_matrices[gse_id] = expr_df\n", " print(f' {gse_id}: {expr_df.shape[0]} samples x {expr_df.shape[1]:,} HGNC genes')\n", " except Exception as e:\n", " print(f' SKIP {gse_id}: {e}')\n", "\n", "# Find common HGNC genes across ALL loaded datasets\n", "gse_ids_hgnc = [g for g in DATASET_ORDER if g in hgnc_matrices]\n", "common_genes = set(hgnc_matrices[gse_ids_hgnc[0]].columns)\n", "for g in gse_ids_hgnc[1:]:\n", " common_genes &= set(hgnc_matrices[g].columns)\n", "common_genes = sorted(common_genes)\n", "print(f'\\nCommon HGNC genes across all {len(gse_ids_hgnc)} datasets: {len(common_genes):,}')\n", "\n", "# Combine into a single matrix; track dataset labels\n", "frames, labels = [], []\n", "for gse_id in gse_ids_hgnc:\n", " sub = hgnc_matrices[gse_id][common_genes]\n", " frames.append(sub)\n", " labels.extend([gse_id] * len(sub))\n", "\n", "combined = pd.concat(frames, axis=0)\n", "labels = np.array(labels)\n", "\n", "# Impute, standardise, then PCA\n", "imp = SimpleImputer(strategy='median')\n", "scaler = StandardScaler()\n", "X_imp = imp.fit_transform(combined)\n", "X_std = scaler.fit_transform(X_imp)\n", "\n", "pca = PCA(n_components=2, random_state=42)\n", "coords = pca.fit_transform(X_std)\n", "var_exp = pca.explained_variance_ratio_ * 100\n", "\n", "# ── Plot ──────────────────────────────────────────────────────────────────────\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n", "\n", "# ── Panel A: overlap heatmap ──────────────────────────────────────────────────\n", "ax = axes[0]\n", "# log10 scale for colour — diagonal (self-overlap) will dominate otherwise\n", "log_mat = np.log10(overlap_mat + 1)\n", "im = ax.imshow(log_mat, cmap='Blues', vmin=0, vmax=np.max(log_mat), aspect='auto')\n", "\n", "for i in range(n):\n", " for j in range(n):\n", " val = overlap_mat[i, j]\n", " text_color = 'white' if log_mat[i, j] > np.max(log_mat) * 0.65 else 'black'\n", " display = f'{val:,}' if val < 10000 else f'{val//1000}k'\n", " ax.text(j, i, display, ha='center', va='center',\n", " fontsize=8, color=text_color,\n", " fontweight='bold' if i == j else 'normal')\n", "\n", "axis_labels = [f'{g}\\n({PLATFORM_LABEL[g]})' for g in gse_ids_raw]\n", "ax.set_xticks(range(n))\n", "ax.set_yticks(range(n))\n", "ax.set_xticklabels(axis_labels, rotation=45, ha='right', fontsize=7.5)\n", "ax.set_yticklabels(axis_labels, fontsize=7.5)\n", "cbar = fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n", "cbar.set_label('log10(shared features + 1)', fontsize=8)\n", "cbar.ax.tick_params(labelsize=7)\n", "ax.set_title('(A) Before HGNC harmonisation\\nRaw feature-namespace overlap across datasets',\n", " fontsize=10, fontweight='bold', pad=8)\n", "\n", "# Annotate diagonal with platform group\n", "for i in range(n):\n", " ax.add_patch(plt.Rectangle((i-0.5, i-0.5), 1, 1, fill=False,\n", " edgecolor='#FF7F0E', linewidth=2))\n", "\n", "# ── Panel B: PCA scatter ──────────────────────────────────────────────────────\n", "ax = axes[1]\n", "for gse_id in gse_ids_hgnc:\n", " mask = labels == gse_id\n", " ax.scatter(coords[mask, 0], coords[mask, 1],\n", " c=PALETTE[gse_id], label=gse_id,\n", " s=55, alpha=0.80, edgecolors='white', linewidths=0.4)\n", "\n", "ax.set_xlabel(f'PC1 ({var_exp[0]:.1f}% variance)', fontsize=10)\n", "ax.set_ylabel(f'PC2 ({var_exp[1]:.1f}% variance)', fontsize=10)\n", "ax.set_title('(B) After HGNC harmonisation\\nPCA of combined expression matrix coloured by dataset',\n", " fontsize=10, fontweight='bold', pad=8)\n", "ax.legend(title='Dataset', fontsize=8, title_fontsize=8,\n", " loc='best', framealpha=0.9, markerscale=1.2)\n", "ax.tick_params(labelsize=8)\n", "\n", "# Add platform group annotations via text regions\n", "platform_groups = {}\n", "for gse_id in gse_ids_hgnc:\n", " pl = PLATFORM_LABEL[gse_id]\n", " platform_groups.setdefault(pl, []).append(gse_id)\n", "\n", "# Annotate each platform group centroid\n", "for platform, gse_list in platform_groups.items():\n", " mask = np.isin(labels, gse_list)\n", " cx, cy = coords[mask, 0].mean(), coords[mask, 1].mean()\n", " ax.annotate(platform, xy=(cx, cy),\n", " fontsize=7.5, color='#333333', ha='center',\n", " bbox=dict(boxstyle='round,pad=0.2', fc='white', alpha=0.7, ec='#CCCCCC'))\n", "\n", "plt.suptitle('Supplementary Figure S1. Batch-effect analysis before and after HGNC harmonisation.',\n", " fontsize=11, fontweight='bold', y=1.01)\n", "plt.tight_layout()\n", "\n", "out_path = os.path.join(FIG_DIR, 'figS1_batch_effect_pca.svg')\n", "fig.savefig(out_path, bbox_inches='tight')\n", "plt.close()\n", "print(f'\\nSaved: {out_path}')\n", "\n", "# ── Summary stats for manuscript methods text ─────────────────────────────────\n", "print('\\n── Feature overlap summary (for Methods text) ──')\n", "for i, gi in enumerate(gse_ids_raw):\n", " for j, gj in enumerate(gse_ids_raw):\n", " if j <= i: continue\n", " shared = overlap_mat[i, j]\n", " if PLATFORM_LABEL.get(gi) == PLATFORM_LABEL.get(gj):\n", " tag = 'SAME PLATFORM'\n", " else:\n", " tag = 'cross-platform'\n", " print(f' {gi} x {gj}: {shared:,} shared raw features [{tag}]')\n", "print(f'\\nCommon HGNC genes after harmonisation: {len(common_genes):,}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# PANEL-SIZE SENSITIVITY ANALYSIS \n", "# For each dataset: sweep top_N ∈ {5, 10, 15, 20, 30} and re-run nested CV.\n", "#\n", "# OPTIMISATION: data is loaded once and reused; nested CV is still full but\n", "# dataset re-loading overhead is eliminated.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "# Sweep extends to 40 so the plateau (or lack thereof) is visible.\n", "TOP_N_SWEEP = [5, 10, 15, 20, 30, 40]\n", "\n", "sensitivity_rows = []\n", "\n", "# Reuse dataset_store if it was populated by the LODO cell; otherwise reload\n", "if 'dataset_store' not in dir() or not dataset_store:\n", " dataset_store = {}\n", " for gse_id, cfg in DATASETS.items():\n", " try:\n", " expr_df, pheno_df = load_align_gse(gse_id, cfg, OUTPUT_DIR)\n", " pheno_t = build_target(pheno_df, cfg)\n", " X = expr_df.loc[pheno_t.index]\n", " y = pheno_t['Target']\n", " if y.nunique() >= 2:\n", " dataset_store[gse_id] = (X, y)\n", " except Exception:\n", " pass\n", "\n", "for gse_id in dataset_store:\n", " X, y = dataset_store[gse_id]\n", " print(f'\\nPanel-size sweep: {gse_id}')\n", "\n", " for top_n in TOP_N_SWEEP:\n", " res = run_nested_cv(gse_id, X, y, top_n=top_n)\n", " if res is None:\n", " continue\n", " metrics_df = pd.DataFrame(res['fold_metrics'])\n", " sensitivity_rows.append({\n", " 'GSE': gse_id,\n", " 'top_N': top_n,\n", " 'AUC_mean': round(metrics_df['auc'].mean(), 3),\n", " 'AUC_sd': round(metrics_df['auc'].std(), 3),\n", " })\n", " print(f' top_N={top_n:2d} AUC={metrics_df[\"auc\"].mean():.3f} ± {metrics_df[\"auc\"].std():.3f}')\n", "\n", "sensitivity_df = pd.DataFrame(sensitivity_rows)\n", "sens_path = os.path.join(EXPORT_DIR, 'table_panel_size_sensitivity.csv')\n", "sensitivity_df.to_csv(sens_path, index=False)\n", "print(f'\\nSaved: {sens_path}')\n", "\n", "# ── Plot sensitivity curves ─────────────────────────────────────────────────\n", "if not sensitivity_df.empty:\n", " gses_in_sens = sensitivity_df['GSE'].unique()\n", " fig, ax = plt.subplots(figsize=(8, 5))\n", "\n", " for gse_id in gses_in_sens:\n", " sub = sensitivity_df[sensitivity_df['GSE'] == gse_id]\n", " ax.errorbar(sub['top_N'], sub['AUC_mean'], yerr=sub['AUC_sd'],\n", " marker='o', linewidth=1.2, capsize=3, label=gse_id)\n", "\n", " ax.set_xlabel('Panel size (top-N genes per model)')\n", " ax.set_ylabel('Nested CV AUC (mean ± SD)')\n", " ax.set_title('Panel-size sensitivity analysis (Figure 4D)')\n", " ax.legend(fontsize=7, ncol=2)\n", " ax.set_xticks(TOP_N_SWEEP)\n", " plt.tight_layout()\n", " fig.savefig(os.path.join(FIG_DIR, 'fig4D_panel_size_sensitivity.svg'), bbox_inches='tight')\n", " plt.close()\n", " print('Saved: fig4D_panel_size_sensitivity.svg')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# LEAVE-ONE-DATASET-OUT (LODO) VALIDATION\n", "# Train on 5 datasets combined → test on the held-out 6th\n", "# Uses consensus panel genes from each training set\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "# Load all data first\n", "dataset_store = {} # gse_id -> (X_hgnc_df, y_series)\n", "\n", "for gse_id, cfg in DATASETS.items():\n", " try:\n", " expr_df, pheno_df = load_align_gse(gse_id, cfg, OUTPUT_DIR)\n", " pheno_t = build_target(pheno_df, cfg)\n", " X = expr_df.loc[pheno_t.index]\n", " y = pheno_t['Target']\n", " if y.nunique() >= 2:\n", " dataset_store[gse_id] = (X, y)\n", " except Exception:\n", " pass\n", "\n", "available = list(dataset_store.keys())\n", "print(f'Datasets available for LODO: {available}')\n", "\n", "lodo_rows = []\n", "\n", "for test_gse in available:\n", " train_gses = [g for g in available if g != test_gse]\n", " if not train_gses:\n", " continue\n", "\n", " X_test, y_test = dataset_store[test_gse]\n", "\n", " # Build combined training set using shared gene space\n", " train_frames, train_labels = [], []\n", " for tr_gse in train_gses:\n", " X_tr, y_tr = dataset_store[tr_gse]\n", " train_frames.append(X_tr)\n", " train_labels.append(y_tr)\n", "\n", " # Find common genes across ALL training datasets + test dataset\n", " common_genes = set(train_frames[0].columns)\n", " for df in train_frames[1:]:\n", " common_genes &= set(df.columns)\n", " common_genes &= set(X_test.columns)\n", " common_genes = sorted(common_genes)\n", "\n", " if len(common_genes) < 10:\n", " print(f'LODO {test_gse}: too few common genes ({len(common_genes)}); skipping.')\n", " continue\n", "\n", " # Stack training data on common genes\n", " X_train_combined = pd.concat(\n", " [df[common_genes] for df in train_frames], axis=0, ignore_index=True\n", " )\n", " y_train_combined = pd.concat(train_labels, ignore_index=True)\n", "\n", " X_test_sub = X_test[common_genes]\n", "\n", " # Feature selection on combined training data\n", " n_inner = safe_n_splits(y_train_combined, desired=5, minimum=2)\n", " if n_inner is None:\n", " continue\n", "\n", " # EN\n", " en_pipe = build_en_cv_pipeline(inner_cv_folds=n_inner)\n", " en_pipe.fit(X_train_combined, y_train_combined)\n", " best_C = float(en_pipe.named_steps['clf'].C_[0])\n", " best_l1 = float(en_pipe.named_steps['clf'].l1_ratio_[0])\n", " en_genes = en_top_genes(en_pipe, common_genes, top_n=TOP_N_GENES)\n", "\n", " # XGBoost (SHAP on test)\n", " imp = SimpleImputer(strategy='median')\n", " X_tr_imp = pd.DataFrame(imp.fit_transform(X_train_combined), columns=common_genes)\n", " X_te_imp = pd.DataFrame(imp.transform(X_test_sub), columns=common_genes)\n", " safe_cols, safe_to_orig = sanitize_feature_names(common_genes)\n", " X_tr_xgb = X_tr_imp.copy(); X_tr_xgb.columns = safe_cols\n", " X_te_xgb = X_te_imp.copy(); X_te_xgb.columns = safe_cols\n", " xgb_m = build_xgb_model(y_train_combined)\n", " xgb_m.fit(X_tr_xgb, y_train_combined)\n", " xgb_genes_safe = xgb_top_genes(xgb_m, X_te_xgb, top_n=TOP_N_GENES)\n", " xgb_genes = [safe_to_orig.get(g, g) for g in xgb_genes_safe]\n", "\n", " panel_genes = sorted(set(en_genes) | set(xgb_genes))\n", " panel_genes = [g for g in panel_genes if g in common_genes]\n", "\n", " if len(panel_genes) < 3:\n", " print(f'LODO {test_gse}: panel too small ({len(panel_genes)}); using full EN.')\n", " y_prob = en_pipe.predict_proba(X_test_sub)[:, 1]\n", " y_pred = en_pipe.predict(X_test_sub)\n", " else:\n", " panel_pipe = build_panel_lr(C=best_C, l1_ratio=best_l1)\n", " panel_pipe.fit(X_train_combined[panel_genes], y_train_combined)\n", " y_prob = panel_pipe.predict_proba(X_test_sub[panel_genes])[:, 1]\n", " y_pred = panel_pipe.predict(X_test_sub[panel_genes])\n", "\n", " m = compute_metrics(y_test.values, y_pred, y_prob)\n", " lodo_rows.append({\n", " 'test_GSE': test_gse,\n", " 'train_GSEs': ';'.join(train_gses),\n", " 'n_common_genes': len(common_genes),\n", " 'n_panel_genes': len(panel_genes),\n", " **{k: round(v, 3) for k, v in m.items()},\n", " })\n", " print(f'LODO test={test_gse:20s} AUC={m[\"auc\"]:.3f} Sens={m[\"sensitivity\"]:.3f} Spec={m[\"specificity\"]:.3f}')\n", "\n", "lodo_df = pd.DataFrame(lodo_rows)\n", "lodo_path = os.path.join(EXPORT_DIR, 'table_lodo_validation.csv')\n", "lodo_df.to_csv(lodo_path, index=False)\n", "print(f'\\nLODO results saved: {lodo_path}')\n", "try:\n", " print(lodo_df.to_markdown(index=False))\n", "except Exception:\n", " print(lodo_df.to_string(index=False))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ══════════════════════════════════════════════════════════════════════════════\n", "# FINAL OUTPUT LISTING\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "\n", "print('\\nFiles written to:', EXPORT_DIR)\n", "print(' table2_nested_cv_performance.csv ← main performance table (AUC, sens, spec, PPV)')\n", "print(' table_gene_selection_frequency.csv ← gene freq across nested CV folds per dataset')\n", "print(' table_consensus_panels.csv ← genes selected in ≥50% folds')\n", "print(' table_cross_dataset_conserved_genes.csv← intersection of all consensus panels')\n", "print(' table_panel_size_sensitivity.csv ← Figure 4D data')\n", "print(' table_lodo_validation.csv ← leave-one-dataset-out results')\n", "print()\n", "print('Figures saved to:', FIG_DIR)\n", "print(' fig1_nested_cv_auc.svg')\n", "print(' fig2_nested_cv_all_metrics.svg')\n", "print(' fig3_{GSE}_gene_freq.svg (one per dataset)')\n", "print(' fig4D_panel_size_sensitivity.svg')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 4 }