Motivation
Classifier evaluation can stratify MADCAP (and related plots) by demographics such as age, sex, or ethnicity, because those diagnostics are already patient-level: each row has a predicted probability and an observed outcome.
Distributional evaluation is harder. Specialty bed-demand predictions are aggregates: many patients contribute fractional expected admissions that are summed, then turned into a count distribution per specialty. Ethnicity (and most other demographics) never appear explicitly in that path today — specialty routing is conditioned on age/sex subgroups only — so any demographic effect enters only indirectly (via admission probabilities, via correlation with age/sex, or via who is in the snapshot cohort). Once you only look at a specialty-level EPUDD or PMF chart, those contributions are mixed together.
Simple representation tables (e.g. ethnicity × specialty shares among observed admissions) answer a different question: case-mix / pathway equity in the observed data. They do not say whether the model over- or under-assigns demand for a demographic group.
This note sketches a middle path: keep the patient-level building blocks that feed the specialty totals long enough to regroup by demographic, then compare predicted vs observed expected counts.
Per-patient specialty mass
At a snapshot, for patient (i) and specialty (s):
[
m_{i,s} = p^{\mathrm{admit}}i \times \pi{i,s}
]
where
- (p^{\mathrm{admit}}_i) is the admission classifier probability for patient (i)
- (\pi_{i,s}) is the specialty-routing probability that an admission for (i) goes to specialty (s) (from the subgroup specialty model)
So (m_{i,s}) is that patient’s expected admission mass to specialty (s) (usually a fraction of a person). Specialty demand predictions are built by summing such masses across patients (then combining uncertainty). The demographic label (e.g. ethnicity (e)) tags the patient; it is not an input to (p) or (\pi).
Predicted counts by group × specialty
For a chosen demographic grouping variable (ethnicity is the motivating example), restrict to patients with group (e) and sum:
[
\hat{N}{e,s} = \sum{i:, \mathrm{group}i = e} m{i,s}
]
Aggregate over the same eval windows / snapshot rules used for distribution evaluation, so the cohort matches the rest of the pipeline.
Interpretation: “How much admission mass did the model assign to group (e) ending in specialty (s)?”
Observed counts
For the same patients and windows, count realised destinations with the same observation definition used elsewhere (e.g. first admitting specialty / admitted-at-some-point):
[
N_{e,s} = \sum_{i:, \mathrm{group}_i = e}
\mathbf{1}{\text{patient } i \text{ admitted to specialty } s}
]
What to compare
For each cell ((e, s)), contrast (\hat{N}{e,s}) and (N{e,s}) (difference, ratio, or both). Normalising by the group’s total mass (share within (e)) can separate “this group is large” from “this group is mis-routed relative to its own total.”
Reading the pattern:
- If the admission classifier is poorly calibrated for group (e), expect (\hat{N}_{e,\cdot}) systematically high or low across specialties.
- If specialty routing mis-assigns destinations for group (e) (even when overall admission calibration looks fine), expect over-prediction in some (s) and under-prediction in others for that (e).
Missing / not-stated demographics need an explicit bucket. Thin (e \times s) cells will be unstable — the same thin-slice problem as stratified MADCAP.
Relation to other checks
| Check |
Question it answers |
| MADCAP by demographic |
Are patient-level admission (or departure) probabilities well calibrated / discriminating within each group? |
| Representation table (group × specialty among observed outcomes) |
How is observed case-mix distributed across clinical areas? |
| Specialty-mass attribution (this note) |
Of the mass that feeds specialty demand, how much is attributed to each group × specialty, and does that match observed counts? |
Attribution is closer to “distributional bias” than representation tables because it uses the same (p \times \pi) terms that build the specialty totals. It is still not a full predictive distribution by demographic: the comparison is of expected counts (means), not separate EPUDD/PMF charts per group. Building demographic-specific bed PMFs (sum (m_{i,s}) only over (i \in e), then form a discrete distribution) is possible in principle but noisier and a larger product change.
Scope notes
- Natural first application: ED current patients → specialty admissions. Yet-to-arrive rate models typically have no patient demographics, so ethnicity cannot be attributed the same way.
- Departure / transfer flows could use an analogous idea (patient-level departure probability × transfer weights), with the same caveats about aggregation and thin cells.
- This is an evaluation / analysis idea, not a change to how production demand is computed.
Motivation
Classifier evaluation can stratify MADCAP (and related plots) by demographics such as age, sex, or ethnicity, because those diagnostics are already patient-level: each row has a predicted probability and an observed outcome.
Distributional evaluation is harder. Specialty bed-demand predictions are aggregates: many patients contribute fractional expected admissions that are summed, then turned into a count distribution per specialty. Ethnicity (and most other demographics) never appear explicitly in that path today — specialty routing is conditioned on age/sex subgroups only — so any demographic effect enters only indirectly (via admission probabilities, via correlation with age/sex, or via who is in the snapshot cohort). Once you only look at a specialty-level EPUDD or PMF chart, those contributions are mixed together.
Simple representation tables (e.g. ethnicity × specialty shares among observed admissions) answer a different question: case-mix / pathway equity in the observed data. They do not say whether the model over- or under-assigns demand for a demographic group.
This note sketches a middle path: keep the patient-level building blocks that feed the specialty totals long enough to regroup by demographic, then compare predicted vs observed expected counts.
Per-patient specialty mass
At a snapshot, for patient (i) and specialty (s):
[
m_{i,s} = p^{\mathrm{admit}}i \times \pi{i,s}
]
where
So (m_{i,s}) is that patient’s expected admission mass to specialty (s) (usually a fraction of a person). Specialty demand predictions are built by summing such masses across patients (then combining uncertainty). The demographic label (e.g. ethnicity (e)) tags the patient; it is not an input to (p) or (\pi).
Predicted counts by group × specialty
For a chosen demographic grouping variable (ethnicity is the motivating example), restrict to patients with group (e) and sum:
[
\hat{N}{e,s} = \sum{i:, \mathrm{group}i = e} m{i,s}
]
Aggregate over the same eval windows / snapshot rules used for distribution evaluation, so the cohort matches the rest of the pipeline.
Interpretation: “How much admission mass did the model assign to group (e) ending in specialty (s)?”
Observed counts
For the same patients and windows, count realised destinations with the same observation definition used elsewhere (e.g. first admitting specialty / admitted-at-some-point):
[
N_{e,s} = \sum_{i:, \mathrm{group}_i = e}
\mathbf{1}{\text{patient } i \text{ admitted to specialty } s}
]
What to compare
For each cell ((e, s)), contrast (\hat{N}{e,s}) and (N{e,s}) (difference, ratio, or both). Normalising by the group’s total mass (share within (e)) can separate “this group is large” from “this group is mis-routed relative to its own total.”
Reading the pattern:
Missing / not-stated demographics need an explicit bucket. Thin (e \times s) cells will be unstable — the same thin-slice problem as stratified MADCAP.
Relation to other checks
Attribution is closer to “distributional bias” than representation tables because it uses the same (p \times \pi) terms that build the specialty totals. It is still not a full predictive distribution by demographic: the comparison is of expected counts (means), not separate EPUDD/PMF charts per group. Building demographic-specific bed PMFs (sum (m_{i,s}) only over (i \in e), then form a discrete distribution) is possible in principle but noisier and a larger product change.
Scope notes