# BioSignal reliability study protocol — v1.1
Author: Prem Kumar Perumal. Research demo, 2 November 2026.

## Question and scope
Does model performance and probability reliability change across independent collection batches, and what error/review trade-off results from a fixed score-margin deferral rule?
The supplied fixture uses invented count channels and labels. It is NOT external metagenomic validation. The new evaluation workspace accepts predictions from separately trained models and never maps real sequencing data to the invented channels.

## Required study design
Define the target population, intended prediction task, reference standard and independent unit before fitting. Assign all related participants, longitudinal samples and technical replicates to one split. Use training for preprocessing/model fitting, validation for criteria, and a separate held-out study/site where feasible. The CSV group check catches declared batch overlap only. Model leakage and reference-label validity require independent review.

Record data source, access date, dataset-level licence/permissions, exclusions, sample flow, outcome definition, model/preprocessing versions and split manifest. Only pseudonymous IDs and scores belong in the browser. Computation stays on the device; exported reports include supplied sample IDs and labels.

## Analysis implemented
Two-class reference labels and probability of class 1. Primary and baseline must have matching samples and targets. One common decision threshold is explicitly fixed before test reveal; studies requiring separately optimised thresholds should predefine an extended protocol rather than tune on test data.
Accuracy, sensitivity, specificity, FPR, FNR, confusion counts, coverage and Brier score. Brier mixes discrimination and calibration; it is not pure calibration error. Calibration uses ten half-open bins [0,.1), ... [.9,1], with n, mean prediction and positive fraction. Curves are descriptive with no hypothesis test.
Defer if abs(primary probability - threshold) < margin. Primary and baseline retained analyses use the SAME retained subset. Referrals are unresolved and never counted as true negatives. Zero denominators produce null / Not estimable. The browser locks threshold/margin after test reveal; reloading is possible and does not restore an unseen test set.
400 seeded whole-batch percentile bootstrap replicates. Resampling preserves all samples per selected batch. 95% endpoints use sorted index floor((n-1)*q); report valid replicates for each statistic. These are conditional on fitted predictions and do not include training variation. Fewer than two batches means no interval; few batches, rare labels or extensive deferral weaken inference. Independence between batches is a study assumption.

## Reproduction
python3 evaluation/build_example.py (run from the BioSignal research pack root after installing source/requirements.txt).
node evaluation/test.cjs
The builder fits a fraction-A logistic baseline only on the shipped training split. The primary model is unchanged. example-provenance.json records coefficients and SHA-256 hashes. Benchmark scripts compare calculations against scikit-learn. Reports contain input SHA-256, selected partition, criteria, predictions, descriptive curves, interval counts and reviewer notes. A browser hash is provenance, not a signature or tamper-proof audit trail.

## Public-data plan — pending, no results claimed
Candidate: curatedMetagenomicData, https://bioconductor.org/packages/curatedMetagenomicData/ . Its documentation describes processed human shotgun microbiome profiles and metadata. It is not respiratory outbreak ground truth. Select an outcome justified by the original study, verify dataset-level permissions and labels, exclude confounded comparisons, split by participant and hold out an entire study. Train a NEW appropriate abundance model with preprocessing fitted on training only; do not reuse the synthetic model. Export its validation/test probabilities to this workspace. A microbiome phenotype benchmark would test transportability methods, not establish pathogen surveillance performance.
Independent laboratory/bioinformatics review is still required; no NHS/UKHSA data, integration or endorsement is claimed.

## References
https://scikit-learn.org/stable/modules/calibration.html
https://scikit-learn.org/stable/modules/cross_validation.html
https://bioconductor.org/packages/release/data/experiment/vignettes/curatedMetagenomicData/inst/doc/curatedMetagenomicData.html
