Threshold Tuning & Calibration

Threshold tuning and calibration is the governance stage that owns the numbers every other validation node reads — the reference bounds, critical bounds, and delta limits that decide whether a patient result is released, flagged, or held. It is not an operator editing a cutoff in a form; it is a versioned, evidence-backed proposal-and-promotion pipeline in which a candidate threshold set is derived from quality-control and historical result evidence, validated against live traffic in shadow mode, and promoted into an effective-dated configuration store only after dual clinical approval. This page specifies the stage’s ingress and egress contracts, the statistical models that turn control data into defensible bounds, the Python patterns that keep every retune reproducible, and the tests that prove a bad threshold set can never silently reach production.

Context and Pipeline Position

Within the Clinical Result Validation & Rule Engine Architecture, threshold tuning sits beside the runtime result path rather than inside it. The runtime path — canonicalization, then the Reference Range Check Implementation, then Delta Validation & Trend Analysis and Critical Value Alert Routing — is deliberately stateless and reads its bounds from configuration. This stage is the owner of that configuration: it consumes quality-control series, reagent-lot metadata, and de-identified historical results, and it produces new versioned interval and threshold rows that the range check will later resolve.

The separation is the whole point. Because the reference range node reads intervals rather than hard-coding them, a bound can be retuned, reviewed, and dated without redeploying the validation engine, and any past decision remains reproducible against the interval that was live at collection time. Upstream, the harmonized units and resolved analyte identities delivered by the Instrument Data Ingestion & HL7/CSV Pipelines tier — units normalized to UCUM, analytes resolved to LOINC through the Test Code Taxonomy & Standards — are the same canonical keys this stage tunes against, so a threshold is always attached to an unambiguous (loinc, specimen, method_version) partition.

Because it never touches a patient result in flight, this stage can afford to be slow, statistical, and human-gated. Its output is data — a new configuration version — not a verdict.

Stage Boundaries

Threshold tuning exposes one ingress contract and two egress contracts, with explicit failure semantics at each edge. Nothing enters production configuration except through the promotion egress.

Ingress — the tuning evidence bundle. The stage accepts a bundle scoped to a single (loinc, specimen, method_version) partition: a Levey-Jennings quality-control series (control level, target mean, assigned SD, per-run observed values with timestamps), the reagent-lot and calibrator-lot identifiers spanning that window, and a de-identified sample of historical patient results for the same analyte and cohort. Demographic cohort keys (age_low_days, age_high_days, sex, condition) travel with the historical sample so the derived bounds stay cohort-specific. Any bundle that fails schema validation — missing assigned SD, non-UCUM units, a QC series shorter than the configured minimum run count — is rejected at the boundary and never partially processed.

Egress A — the candidate threshold set. The stage emits a CandidateThresholdSet: proposed lower, upper, critical_low, critical_high, and delta limits for the partition, each annotated with the estimator used, the evidence window, and a diff against the currently effective version. A candidate is inert — it changes no runtime behavior until promoted.

Egress B — the promotion record. After a shadow-mode evaluation and dual clinical approval, the candidate becomes an effective-dated interval row written to the same versioned store the range check reads, carrying a new version, an effective_from date, and the approval signatures. Promotion is the only path by which a number reaches production.

Failure semantics. A transport failure (QC store unreachable, historical query timeout) is retryable and re-driven with backoff; it never yields a silent “no change” that masks stale evidence. A statistical failure — insufficient in-control runs, a QC series that fails its own Westgard rules, an estimator that cannot converge — is not retryable: it blocks candidate generation and routes the partition to a clinical-review queue with a machine-readable reason. The controlling invariant: the absence of a validated, approved candidate leaves the currently effective threshold set untouched. Tuning degrades toward the last known-good bounds, never toward an unvalidated guess.

Statistical Specification

Fixed cutoffs cannot absorb reagent-lot shifts, calibrator changes, or seasonal cohort drift. This stage derives bounds from control data and robust estimators rather than analyst intuition, and it records exactly which estimator and window produced each number. Two evidence sources feed the derivation: the quality-control series governs whether the method is stable enough to tune at all, and the historical result distribution shapes the reference bounds within a stable method.

Quality-control acceptability is decided by Westgard multi-rules over the Levey-Jennings series before any bound is proposed. A partition whose control data violates a rejection rule is out of control, and out-of-control data may not be used to move a patient-facing threshold.

Only once the control series is in control does the stage estimate reference bounds from the historical result distribution. Robust estimators are mandatory because raw laboratory distributions carry outliers, hemolysis artifacts, and mixed-cohort contamination that classical mean±2SD bounds would absorb into the interval:

• Tukey fences derive lower/upper from the interquartile range (Q1 − k·IQR, Q3 + k·IQR), resistant to the tail contamination classical SD is not.
• Rolling median absolute deviation (MAD) tracks slow method drift across reagent lots without being dragged by single aberrant runs, and scales to a robust SD estimate via the 1.4826 consistency constant.
• Exponentially weighted moving average (EWMA) on the QC series surfaces sustained shifts earlier than a fixed-window mean, so a calibration drift is caught before it widens the reference bound.

Every derivation is pinned: the estimator name, its parameters (k, EWMA lambda, window length), the deterministic evidence window, and the resulting bounds are all recorded on the candidate so an inspector — or a future engineer investigating a shifted bound — can recompute the number byte-for-byte.

Implementation Patterns

Model every contract with Pydantic v2 so the evidence bundle is validated by construction and the candidate is serializable for the approval and audit sinks. The estimators are pure functions over the evidence; only the promoter touches the configuration store.

from __future__ import annotations

from datetime import date, datetime
from decimal import Decimal
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field


class Estimator(str, Enum):
    TUKEY = "tukey"
    MAD = "mad"
    EWMA = "ewma"


class QCPoint(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    observed: Decimal
    run_at: datetime
    reagent_lot: str


class EvidenceBundle(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    loinc: str
    specimen: str
    method_version: str
    target_mean: Decimal
    assigned_sd: Decimal = Field(gt=0)
    qc_series: tuple[QCPoint, ...]
    historical: tuple[Decimal, ...]
    min_in_control_runs: int = 20


class CandidateThresholdSet(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    loinc: str
    specimen: str
    method_version: str
    estimator: Estimator
    window_from: date
    window_to: date
    lower: Decimal
    upper: Decimal
    critical_low: Decimal | None = None
    critical_high: Decimal | None = None
    supersedes_version: str | None = None

Westgard evaluation and the robust estimators are deterministic and side-effect free, which is what makes a retune reproducible:

from statistics import median


def westgard_1_3s(series: list[Decimal], mean: Decimal, sd: Decimal) -> bool:
    """True when any point exceeds 3 SD — the run must be rejected."""
    return any(abs(x - mean) > 3 * sd for x in series)


def westgard_4_1s(series: list[Decimal], mean: Decimal, sd: Decimal) -> bool:
    """True when four consecutive points fall on one side beyond 1 SD."""
    for i in range(len(series) - 3):
        window = series[i : i + 4]
        if all(x - mean > sd for x in window) or all(mean - x > sd for x in window):
            return True
    return False


def tukey_fences(values: list[Decimal], k: Decimal = Decimal("1.5")) -> tuple[Decimal, Decimal]:
    ordered = sorted(values)
    n = len(ordered)
    q1 = ordered[n // 4]
    q3 = ordered[(3 * n) // 4]
    iqr = q3 - q1
    return (q1 - k * iqr, q3 + k * iqr)


def robust_sd_via_mad(values: list[Decimal]) -> Decimal:
    med = Decimal(median(values))
    deviations = sorted(abs(x - med) for x in values)
    mad = deviations[len(deviations) // 2]
    return Decimal("1.4826") * mad  # MAD → robust SD estimate

The tuner orchestrates the derivation but performs no promotion. It gates on quality control first, then derives bounds, and returns either a candidate or a blocked reason — never a live change:

import asyncio


class ThresholdTuner:
    def __init__(self, store: "ConfigStore") -> None:
        self._store = store
        self._sem = asyncio.Semaphore(16)

    async def derive(self, bundle: EvidenceBundle) -> CandidateThresholdSet | None:
        series = [p.observed for p in bundle.qc_series]
        mean, sd = bundle.target_mean, bundle.assigned_sd

        if len(series) < bundle.min_in_control_runs:
            return None  # insufficient evidence → route to review
        if westgard_1_3s(series, mean, sd) or westgard_4_1s(series, mean, sd):
            return None  # out of control → block, do not tune

        lower, upper = tukey_fences(list(bundle.historical))
        async with self._sem:
            current = await self._store.effective_version(
                bundle.loinc, bundle.specimen, bundle.method_version
            )
        return CandidateThresholdSet(
            loinc=bundle.loinc,
            specimen=bundle.specimen,
            method_version=bundle.method_version,
            estimator=Estimator.TUKEY,
            window_from=bundle.qc_series[0].run_at.date(),
            window_to=bundle.qc_series[-1].run_at.date(),
            lower=lower,
            upper=upper,
            supersedes_version=current,
        )

Before promotion, the candidate runs in shadow mode: the classification engine evaluates a replayed slice of recent traffic against both the effective and the candidate bounds, and the tuner reports the reclassification delta — how many results would flip flag, and in which direction. A candidate that would newly auto-verify a run of previously-flagged criticals is caught here, before a human ever approves it. Because the underlying classify function is pure, the shadow run is a deterministic fan-out across the event loop bounded by the tuner’s semaphore, reusing the same async batch processing workers that drive live ingestion.

Error Classification and Handling

Errors are tiered so each class has exactly one correct disposition. Collapsing them — treating an out-of-control QC series the same as a store timeout — is how a laboratory ends up moving a patient-facing bound on unstable evidence.

Transport failures never degrade to a “no candidate, all clear” outcome; they raise, the orchestrator quarantines the bundle, and a bounded exponential backoff re-drives it — the same discipline the upstream schema validation and error-handling layer applies at ingest. Schema failures are caught by Pydantic construction: an extra="forbid", frozen=True model turns a malformed evidence bundle into a ValidationError at the boundary rather than a silently truncated series feeding an estimator. Statistical failures are the designed-for path to clinical review — they carry a stable reason (insufficient_runs, qc_out_of_control, estimator_nonconvergent) so dashboards distinguish a genuine method problem from an operational blip. Governance failures — a shadow reclassification delta exceeding its guardrail, or a promotion attempted with fewer than two approvers — refuse the promotion and hold the candidate for adjudication.

The controlling invariant is the safety default: any unresolved or unvalidated state leaves the currently effective threshold set in force. A blocked retune is a signal to investigate a method, never a reason to widen a bound blindly.

Regulatory Touchpoints

Threshold tuning is a defined analytic-decision-governance point under several accreditation regimes, and each one constrains the pipeline above.

• CLIA §493.1253 (establishment and verification of performance specifications) is why every candidate pins its estimator, evidence window, and QC provenance: a reference interval or decision threshold must be traceable to the data and method that established it.
• CLIA §493.1256 (control procedures) is why Westgard evaluation gates derivation — control data must demonstrate the method is in control before its results shape a patient-facing bound.
• CLIA §493.1291© requires the reference interval to appear with the reported result, so promotion writes an effective-dated version and the runtime path persists the exact interval_version used; a bound retuned today never rewrites a result released last month.
• CAP GEN.41350 / automated-verification checklist items require auto-verification rules — and the thresholds that drive them — to be documented, version-controlled, and dual-approved, satisfied by the shadow-mode gate and the dual-signature promotion built on the CLIA/CAP Data Boundaries defined at the platform layer.
• 21 CFR Part 11 §11.10 / §11.50 demand attributable, tamper-evident records with signatures: every promotion writes the candidate, the shadow delta, and both approver identities to an append-only sink with a monotonic sequence number.
• HIPAA §164.312(b) audit controls apply because the historical evidence sample carries PHI-derived distributions; access to the tuning workspace and audit sink is role-based and logged, and the historical sample is de-identified before it leaves the runtime boundary.

Because these clauses attach to a change in the decision surface, the promotion audit event must be written before the new version becomes effective, and it must include enough to reconstruct the derivation: the evidence window, the estimator and parameters, the QC verdict, the shadow reclassification delta, and both approvals.

Testing and Validation

A tuning pipeline is only trustworthy if it demonstrably cannot promote a bad bound. Combine property-based tests over the estimators, golden-file fixtures over real retune scenarios, and a promotion contract test.

Property-based tests with hypothesis assert the invariants that must hold for every input — most importantly that a derived interval is well-ordered and that critical bounds never cross their reference bounds:

from decimal import Decimal

from hypothesis import given, strategies as st

from tuning import robust_sd_via_mad, tukey_fences

samples = st.lists(
    st.decimals(min_value=0, max_value=1000, allow_nan=False, allow_infinity=False),
    min_size=8,
    max_size=500,
)


@given(values=samples)
def test_tukey_fences_are_ordered(values: list[Decimal]) -> None:
    lower, upper = tukey_fences(values)
    assert lower <= upper


@given(values=samples)
def test_robust_sd_is_non_negative(values: list[Decimal]) -> None:
    assert robust_sd_via_mad(values) >= 0

Golden-file fixtures pin real retune scenarios — a reagent-lot shift that legitimately widens a potassium interval, a 4_1s calibration drift that must block tuning, a historical sample contaminated with hemolysis outliers that Tukey fences must resist — as evidence-bundle/expected-candidate pairs, so any change to the estimators surfaces as a diff in review. A promotion contract test loads a candidate and asserts the governance invariants without touching live configuration: every critical_low below its lower, every critical_high above its upper, a shadow reclassification delta within guardrail, and at least two distinct approver identities. Run it in CI on every candidate so a malformed or under-approved promotion is refused before it can reach the store.

Round-trip every model through model_dump_json() / model_validate_json() in tests to guarantee the promotion record written to the audit sink deserializes back identically — the property an inspector relies on years later, and the same guarantee the downstream Handling out-of-range flags without manual intervention automation depends on when it reads these bounds to auto-resolve flags without a technologist in the loop.

Related

• Reference Range Check Implementation — the runtime node that reads the intervals this stage tunes and promotes.
• Delta Validation & Trend Analysis — the longitudinal stage whose delta limits are calibrated here from historical variation.
• Critical Value Alert Routing — the escalation path whose panic bounds this stage sets and shadow-tests before promotion.
• Handling out-of-range flags without manual intervention — how the promoted thresholds drive automated hold/release routing.
• Test Code Taxonomy & Standards — the LOINC identity that keys every partition this stage tunes.

Part of: Clinical Result Validation & Rule Engine Architecture.

Frequently Asked Questions