Skip to content

Latest commit

 

History

History
498 lines (393 loc) · 25.5 KB

File metadata and controls

498 lines (393 loc) · 25.5 KB

Findings that are not about models

Things this benchmark turned up that are worth recording separately: measurement defects that produced published numbers, and cost results that contradict the reason people reach for these models.

The bug that produced two published results

Both TabNet and FT-Transformer were characterised in earlier notes on numbers generated by a run in which they were largely not training.

The wrappers split off a validation set for early stopping with rng.permutation seeded from a constant, unstratified. On any dataset where a class missed that split, the logloss metric raised — and because the seed was fixed, it raised on every trial, every fold, every time.

The failure was silent, and worse than silent: a dataset whose every fit raises completes fast. Optuna caught the exception, marked the trial failed, and moved on in milliseconds. So the dataset finished quickly, recorded a NaN or a score from whatever partial folds survived, and the run looked healthy.

The damage:

model datasets with no score what the published numbers meant
TabNet 26 of 146 cost measured on mostly-failing fits
FT-Transformer 79 of 146 mean over the 67 that happened to work

The clearest single case is abalone-7class: the log records 1736.9 s of TabNet time with zero completed fits — no early-stopping line, no max-epochs line. That 1736.9 s is pure wrapper overhead for 324 fits that each raised on entry.

Consequence for TabNet's headline number: its published 107.3 h total made it look like a cheap model that simply performs badly. Measured with every fit training, it costs 485.3 h, the most of any model, and still ranks last. TabNet is not a cheap weak model; it is an expensive weak one.

The lesson is about coverage, not about these two models. A mean is only meaningful next to the count it was taken over. Check the all-NaN count before trusting any aggregate — the run that produces the worst numbers is the one that looks fastest.

A default batch size trained TabNet on nothing

With the split fixed and 144 of 146 datasets scored, TabNet still averaged 0.5946 — last by 0.18, ahead of CatBoost on 2 of 144 datasets. The coverage check above passed.

batch_size was fixed at 1024 and never tuned, and pytorch-tabnet's fit defaults to drop_last=True. An inner fit trains on about 48 % of a dataset, so below roughly 2 100 rows its only batch is incomplete and gets dropped: zero gradient steps. Early stopping then fires on the untrained model, the fit returns in under a second, and the score lands next to the class-prevalence baseline. That was 82 of 146 datasets.

dataset prevalence baseline broken fixed CatBoost
connectionist-vowel 0.0909 0.1280 0.9853 0.9966
cnae-9 0.1111 0.1567 0.9419 0.9793
semeion 0.1000 0.1406 0.9113 0.9880
dermatology 0.2012 0.3070 0.9903 0.9942

Below 1024 training rows the batch is now an eighth of the split, and only a single-row remainder is dropped, since BatchNorm cannot take one. On the 82 affected datasets TabNet moves from 0.4662 to 0.7345 (CatBoost 0.8160); overall from 0.5946 to 0.7507, and from 2 to 11 datasets ahead of CatBoost. It still ranks last.

A full count of scores is not enough either. An untrained classifier returns a valid number on every dataset. The tell is a score sitting on the prevalence baseline, which scripts/check_prevalence_baseline.py now computes for every stored pair — see the next section.

The floor this metric starts from

ROC AUC begins at 0.5 no matter what the data looks like, so a useless model is obvious from the number alone. Weighted PR AUC begins at the class prevalence: the positive-class share for binary, the sum of squared class shares for multiclass. That floor ranges from 0.01 to 0.95 across these 146 datasets, so a score means nothing without it. benchmark.metrics.pr_auc_baseline computes it, and a test pins it to what a constant predictor actually scores.

scripts/check_prevalence_baseline.py compares every stored score against its own floor. Ten of roughly 2 300 pairs sit within 0.01 of it, and the two datasets behind them are worth knowing about:

dataset baseline flagged best model models above the floor
thoracic-surgery 0.8511 TabNet 0.8376, SVC 0.8515 TabFM 0.9132 15 / 16
planning-relax 0.2857 five, from LightGBM 0.2478 up SVC 0.4044 11 / 16

TabNet's 0.8376 on thoracic-surgery is the illustration: in a column of means it reads as one of the better results in the benchmark, and it is below what predicting the majority class achieves. The flagged pairs are not datasets that defeat everything — most models clear the floor on both — which is what makes them worth looking at.

The check prints a report and is not a test that fails. A dataset can legitimately defeat a model, and a build that breaks on an honest result gets switched off.

Label ordering silently changed the metric

pr_auc_score reads y_prob[:, 1] on binary problems, so which class counts as positive follows label ordering. The classical and foundation paths encoded labels differently, and on the affected binary datasets the two families were scored against opposite classes.

This did not produce obviously wrong numbers. It produced interesting ones, which is worse. It generated an entire published finding: a table of 13 datasets where foundation models supposedly collapsed, and two successive explanations for it — first "severe class imbalance", then a correction to "small binary data". Both described an artefact.

dataset published gap actual
blood-transfusion-service +0.3661 −0.0122 (foundation wins)
appendicitis +0.2361 −0.0314 (foundation wins)
seismic-bumps (TabICL vs CatBoost) +0.73 +0.0026
thoracic-surgery (TabICL vs CatBoost) +0.66 +0.0002

After the fix, exactly 2 of 146 datasets have any classical model ahead of every foundation model by more than 0.02.

A finding that is large, clean, and explainable deserves more suspicion than a messy one, not less. Both explanations fit the bad data comfortably.

AutoML rows were joined by position

The AutoML runs cover 142 of the 146 datasets. The figure notebooks sliced evaluated_datasets positionally, which assumes the four missing datasets are last. They are not — the gaps fall at indices 61, 62, 68 and 69 (movement-libras-10, movement-libras, ozone-eighthr, ozone-onehr).

81 of 142 rows were attributed to the wrong dataset. Recovered by DP sequence alignment against the score vectors: the aligned mapping costs 12.797 / 12.729 with correlation 0.80 between the two frameworks, against 19.889 / 19.798 and correlation 0.37 for the positional assumption. The names are now written into the joblib files and joined by name.

The corrected numbers move AutoML from "comparable to individual gradient boosters" to the top of the benchmark by a clear margin.

TabNet was on the CPU the whole time

ResNet and FT-Transformer use _get_device(), which selects MPS. TabNet is built through pytorch-tabnet, whose define_device("auto") is:

if device_name == "auto":
    if torch.cuda.is_available():
        return "cuda"
    else:
        return "cpu"

MPS is not in it. On Apple silicon "auto" silently means CPU, and no warning is emitted. Forcing device_name="mps" measured 3.9-4.3x faster with no cost in score: over 5 seeds per device, abalone-3class 0.5398 ± 0.0763 on CPU against 0.5733 ± 0.0245 on MPS, volcanoes-a3 0.8243 ± 0.0060 against 0.8140 ± 0.0265.

A single-seed comparison first looked alarming (0.6252 CPU vs 0.4670 MPS), but CPU's own five-seed range on that dataset is [0.4165, 0.6414]. Device comparisons on a stochastic model need distributions, not one run each.

Small data reverses the result. Below 1024 training rows the batches are tens of rows, MPS dispatch overhead outweighs the arithmetic, and CPU wins: autoUniv-au6-1000 (1000 x 40) takes 2 098 s on CPU with the four inner folds in parallel, against 9 994 s for autoUniv-au1-1000 (1000 x 20) on MPS with folds in sequence. The wrapper picks the device by split size.

TabNet's seed variance exceeds most between-model gaps

That same experiment: on abalone-3class, TabNet's PR AUC across 5 seeds spans 0.4165 to 0.6414 on CPU, a standard deviation of 0.076.

For scale, the entire classical block of this benchmark — CatBoost, LightGBM, LightGBM-linear, XGBoost, Random Forest, HistGradientBoosting, SVC — spans 0.015 in mean PR AUC. TabNet's run-to-run noise on one dataset is five times the spread the benchmark is trying to resolve between seven different algorithms.

Nested CV over 4 outer folds damps this, but not to nothing, and nothing in the published single-number-per-model format shows it.

SVC's 96 hours were memory thrash, not compute

SVC originally recorded 96.3 h, the second most expensive model in the benchmark. Re-run with identical scoring code on an otherwise idle machine it took 2.41 h, and arcene went from 22 886.9 s to 9.8 s — a factor of 2300 on one dataset.

The difference was not the algorithm. The original run shared the machine with processes that drove it into swap. The binding resource in this benchmark is RAM, not cores: 24 GB, and a single TabFM dataset uncapped allocated 13.55 GB.

A separate real defect: libsvm's SMO solver is unbounded by default and never terminates on some (config, fold) pairs — workers observed still spinning after 14 h 08 m. max_iter=2_000_000 bounds it and is verified non-binding: scores are byte-identical on all comparable datasets.

A month of stolen cores, invisible from inside

Partway through the ResNet run its rate drifted from 1.31x to 1.58x of historical over several days. The obvious reading was that the remaining datasets were more expensive. That was wrong.

Ten orphaned VSCode helper processes, each burning 90-125 % CPU with an RSS under 200 KB, had been running for 33 days and were taking roughly ten of the machine's fourteen cores. Load average was 82. After killing them the rate returned to 1.15x, better than the run had ever managed.

The run itself had no way to see this. Every in-process signal — its own CPU percentage, its memory, its progress per dataset — looked normal. Wall-clock per dataset was the only affected number, and it is exactly the number a benchmark publishes.

An upgrade that moved one model and not another

TabPFN-3.5 needs tabpfn 9.0.0; the benchmark had been running 8.2.0. Upgrading a library that holds every foundation-model number in the report is not free, so the question was what it changes.

The checkpoint filenames for v2, v2.5, v2.6 and v3 are byte-identical between the two releases — the weights do not move. Re-running one dataset before and after the upgrade settled the rest: TabPFN-3 on wholesale-channel reproduced its stored scores exactly, 0.963776 / 0.951391 / 0.925232 / 0.926493, to 0.0e+00.

TabPFN 2.6 did not. On teaching-assistant-evaluation it moved by up to 0.033, and GridSearch picked a different ensemble size, because 9.0.0 scales n_estimators for feature coverage where 8.2.0 did not. Same weights, different inference path.

So TabPFN 2.6 was dropped rather than republished on numbers the current code cannot reproduce. Its checkpoint is kept on disk under a .bak- name; 236 h of compute is not worth deleting to save 7 MB. The model was eleventh of fifteen and the most expensive foundation model in the benchmark, so nothing in the conclusions rested on it.

The general point: pinning a library version is not the same as pinning behaviour. The weights file was identical and the results still moved.

Which headline gaps the data actually supports

Every ranking in this benchmark is a list of means, and the gaps between neighbours run 0.002-0.009 — well inside the seed variance measured on a single model. The figures now test them: Friedman as an omnibus, then Wilcoxon signed-rank on every pair with Holm correction, over the 106 datasets every model scores. One dataset is one observation; the four folds of a dataset share their data and would inflate the sample fourfold.

The family is 18 models, so 153 pairs. The three GridSearch baselines from compare_baseline_models.py are left out of it: they are the same learners as their tuned entries, pairs between the two versions are not comparisons this report makes, and carrying them would widen the family to 210 pairs and weaken every verdict.

Nemenyi's critical distance is not used. It compares average ranks, and a model's average rank moves when an unrelated model joins the comparison, so the verdict on a pair depends on company it never met. Wilcoxon reads only the pair's own scores. The Holm multiplier still depends on the family size, which is why the family is stated rather than assumed.

comparison mean gap wins raw p Holm p verdict
MLJAR over TabFM +0.0295 61 / 106 0.015 0.36 not separable
TabFM over TabPFN-3 +0.0041 73 / 106 8e-05 0.004 separable
TabPFN-3.5 over TabPFN-3 +0.0040 71 / 106 1e-04 0.005 separable
TabPFN-3.5 over TabPFN-3.5-fast +0.0017 65 / 106 0.002 0.06 not separable
TabPFN-3.5-fast over TabPFN-3 +0.0022 67 / 106 0.013 0.33 not separable
TabPFN-3.5 over TabFM -0.0002 37 / 106 0.17 1 not separable
CatBoost over HistGradientBoosting +0.0067 75 / 106 4e-06 0.0003 separable
CatBoost over Random Forest +0.0021 70 / 106 0.0009 0.04 separable
CatBoost over LightGBM Linear -0.0006 60 / 106 0.08 1 not separable

The first row is the one that mattered. The README said AutoML wins; MLJAR's 0.03 lead over TabFM comes from large gains on a minority of datasets, and the paired test does not separate the two. That verdict rests on the correction — the raw p-value is 0.015 — which is the honest thing to report rather than either number alone.

The rest run the other way. Gaps of 0.002 to 0.007, small enough to read as noise in a table of means, are consistent enough across datasets to survive correction for 153 comparisons.

Two rows show the correction doing opposite things to near-identical evidence. TabPFN-3.5 over TabPFN-3 is +0.0040 on 71 datasets and survives; TabPFN-3.5-fast over TabPFN-3 is +0.0022 on 67 and does not. The gap between the two verdicts is not a difference in kind, it is where 0.05 happens to fall.

The TabFM row is worth reading twice. TabPFN-3.5 has the same mean to within 0.0002 but wins only 37 of the 106 head-to-head — TabFM wins more often, TabPFN-3.5 wins by more when it does. Means and win counts answer different questions, and neither alone is the ranking.

The critical-difference figure draws a bar only where every pair inside it is inseparable. Six models now sit under one bar: TabFM, TabPFN-3.5, TabICL, TabPFN-3.5-fast and both AutoML frameworks. TabPFN-3 falls just outside it, sharing the next bar down.

PR AUC cannot see calibration

PR AUC scores an ordering. A model that ranks every positive above every negative is perfect by that measure whether its 0.9 means 0.9 or 0.6. Nothing in this benchmark read the probabilities themselves until now.

Brier and top-label ECE over the stored out-of-fold predictions, 108 datasets scored by all sixteen models that kept predictions. ECE T and ECE iso are the same models after post-hoc repair, covered in the next section.

model PR AUC Brier ECE ECE T ECE iso PR AUC iso
TabFM 0.8596 0.1881 0.0367 0.0355 0.0352 0.8469
TabICL 0.8574 0.1905 0.0405 0.0347 0.0359 0.8420
TabPFN-3.5-fast 0.8577 0.2129 0.0427 0.0413 0.0364 0.8436
TabPFN-3.5 0.8595 0.2103 0.0429 0.0418 0.0367 0.8462
TabPFN-3 0.8557 0.2127 0.0443 0.0420 0.0395 0.8402
SVC 0.8240 0.2350 0.0477 0.0437 0.0436 0.8052
ResNet 0.8211 0.2446 0.0512 0.0487 0.0455 0.8014
XGBoost 0.8294 0.2394 0.0627 0.0521 0.0478 0.8114
LightGBM 0.8311 0.2443 0.0674 0.0604 0.0572 0.8098
Logistic Regression 0.7838 0.3084 0.0732 0.0687 0.0581 0.7564
HistGradientBoosting 0.8274 0.2526 0.0739 0.0561 0.0508 0.8094
Random Forest 0.8325 0.2410 0.0747 0.0459 0.0463 0.8194
LightGBM Linear 0.8348 0.2457 0.0756 0.0705 0.0581 0.8061
CatBoost 0.8342 0.2496 0.0848 0.0665 0.0565 0.8087
TabNet 0.7529 0.3238 0.0851 0.0665 0.0589 0.7362
SGD 0.7837 0.3254 0.1000 0.0980 0.0640 0.7515

The five foundation models hold the five best ECE values. CatBoost — the top classical model on PR AUC, tied there with LightGBM Linear — is fourteenth of the sixteen on calibration, with only TabNet and SGD behind it. Rank by PR AUC and rank by calibration disagree about which classical model to reach for.

The two TabPFN-3.5 variants land third and fourth, a little ahead of TabPFN-3 and a little behind TabFM and TabICL. Calibration is the one axis where the new generation did not move much.

PR AUC is the mean over the four outer folds, as everywhere else in this report. Brier and ECE pool the folds first: a 300-row fold spread over 15 bins is mostly binning noise, and averaging that per fold inflates ECE by 1.3x to 1.7x.

This cost nothing to measure: the per-fold probabilities and labels have been in results/ckpt/<model>.joblib all along. AutoGluon and MLJAR are the exception — their runners stored scores and times only, so the two models the README used to call the winners cannot be checked for calibration without a re-run.

Post-hoc calibration does not buy the gap, and it is not free

The obvious objection to the table above is that calibration is a solved problem: fit a one-dimensional map on held-out predictions and the cheap model catches up. It does not. Both repairs are cross-fitted over the model's own four outer folds — each fold is corrected by a map fitted on the other three, so the calibrator never reads the labels it is scored against.

Temperature scaling divides the log probabilities by one fitted scalar. It can only sharpen or soften confidence, never reorder the classes in a row, so on binary problems PR AUC is untouched. Isotonic regression is the non-parametric ceiling: one-vs-rest, then renormalised.

Three things come out of it.

The repair is real but small where it matters. Random Forest gains the most (0.0747 to 0.0459, a 39 % cut), then SGD, CatBoost and HistGradientBoosting, all around 31-36 %. The foundation models gain least, because they had least to give back.

It does not close the gap. Twelve of the sixteen models still sit above the untreated TabFM at 0.0367 after their better repair. The four that reach it are TabFM, TabICL and both TabPFN-3.5 variants — every one a foundation model. Repaired CatBoost lands at 0.0565, still worse than raw TabPFN-3 at 0.0443. "Just calibrate the gradient booster" does not produce a foundation model's probabilities.

The ceiling costs ranking. Isotonic loses PR AUC on all sixteen, from -0.0127 (TabFM) to -0.0322 (SGD). CatBoost gives up 0.0255 — twelve times the 0.0021 gap over Random Forest that the pairwise test calls significant. Temperature scaling is the honest free lunch, and it is the weaker of the two everywhere except TabICL and Random Forest.

So the calibration ordering is not an artefact of leaving the classical models unrepaired. It survives repair, and the repair that closes most of it costs more ranking than the ranking differences this benchmark is able to detect.

Ensembles over the stored predictions never help

Every model stores per-fold probability matrices, so combining them costs arithmetic rather than compute. 126 datasets covered by all nine models tested; probability, logit and rank averaging.

ensemble best combiner mean vs best single win rate
TabFM + TabICL + TabPFN-3 logit 0.8654 +0.0001 38.9 %
3 foundation + CatBoost logit 0.8647 −0.0006 42.1 %
TabICL + TabPFN-3 logit 0.8636 −0.0017 34.1 %
all 9 models logit 0.8620 −0.0033 30.2 %
TabICL + CatBoost logit 0.8612 −0.0041 29.4 %
CatBoost + LightGBM-linear + XGBoost logit 0.8491 −0.0162 17.5 %

Best single model is TabFM at 0.8653. Nothing beats it. The strongest ensemble ties it to within 0.0001 and wins on 39 % of datasets — worse than a coin flip.

Two specifics worth stating. Adding CatBoost to the foundation trio makes it worse (−0.0006), which refutes the "always co-train a cheap classical baseline" advice independently of the blind-spot table, itself a scoring bug. And logit averaging beats probability averaging beats rank averaging, consistently, in every combination — but the ordering does not matter much when none of them wins.

Cost does not track performance

Total wall clock for the nested CV, against mean PR AUC over each model's own coverage. The two TabPFN-3.5 rows cover 131 datasets rather than 146, because runs now skip the UCI++ duplicates; over those same 131 TabPFN-3 costs 63.6 h, which is the like-for-like number to compare them against.

model hours median/dataset mean PR AUC coverage
SGD 0.9 11 s 0.7729 146
SVC 2.4 10 s 0.8248 146
TabFM 4.3 27 s 0.8653 126
LogReg 5.9 24 s 0.7772 146
XGBoost 7.4 53 s 0.8328 146
Random Forest 7.6 105 s 0.8359 146
TabPFN-3.5-fast 9.3 85 s 0.8608 131
LightGBM 11.3 118 s 0.8345 146
LightGBM-linear 13.2 145 s 0.8374 146
TabPFN-3.5 17.8 180 s 0.8627 131
TabICL 34.7 364 s 0.8574 142
HistGradientBoosting 36.5 384 s 0.8303 146
TabPFN-3 73.6 1199 s 0.8591 146
CatBoost 75.7 268 s 0.8386 146
ResNet 207.7 2597 s 0.8234 146
TabNet 485.3 2754 s 0.7507 146

ResNet and TabNet together cost 693.0 h — more than every other model combined — to finish below Random Forest, which costs 7.6 h. TabNet's hours mix two execution modes, MPS with sequential folds on the larger datasets and CPU with parallel folds on the smaller ones, so they compare only roughly with the rest.

The sharpest illustration is inside one model family. TabPFN-3 costs 63.6 h over the 131 datasets a run now covers; TabPFN-3.5 costs 17.8 h and scores higher; the fast variant costs 9.3 h and is within 0.002 of it. Nearly seven times the price for a difference the test cannot find.

TabFM reaches the highest mean of any single model for 4.3 h, though on 126 datasets and with a GPU.

Fifteen datasets are duplicates

UCI++ reuses the same underlying data in different configurations, and 15 of the 146 are such variants. They used to be listed in each figure notebook and dropped after the fact, so they were computed and then discarded — 7 of the 9 volcanoes variants cost ResNet over 2 h each, and about 10 % of the benchmark's compute went to datasets no figure reads.

The list now lives in config.DUPLICATE_DATASETS and the runners skip it, so that compute is not spent again. The scores already on disk are kept and the aggregates still span all 146, so nothing published changed; a model added from here on covers 131. Duplicates also inflated the sample in the significance tests, which is the other reason to drop them before the statistics rather than after.

The pairs the timeout could not measure — now measured

A timed-out (dataset, model) pair records time = NaN, so the hours it burned never enter the cost figures. Each was re-run with the cap raised. Every answer was worth the machine time.

CatBoost on plant-species-leaves-shape (100 classes, 1600 rows, 64 features) needs 25.16 h. It had recorded NaN twice before — 14 h with the timeout defeated, then a clean 12 h cut — so the number simply did not exist. What it buys:

model PR AUC time
TabICL 0.8998 0.34 h
TabPFN-3 0.8987 0.60 h
ResNet 0.8250 1.57 h
SVC 0.7620 0.01 h
CatBoost 0.7321 25.16 h
Random Forest 0.7203 0.17 h
LightGBM 0.6665 0.55 h

CatBoost places fifth. SVC scores higher in 36 seconds — 2500x cheaper for +0.03 PR AUC. TabICL scores +0.17 higher for 1/74th of the cost. This one dataset is a third of CatBoost's entire 75.7 h benchmark cost, and it is the single clearest case in the benchmark of compute buying nothing.

The failure mode is many-class data: CatBoost trains one-vs-all across 100 classes inside a 50-trial nested search. Nothing about the data is hard — every other model finishes in under two hours.

ResNet on multiple-features (2000 x 649) needs 10.40 h and scores 0.9987. Above 500 features _train_rtdl_on_device falls back from MPS to CPU. Historically 1.20 h at 20 trials; 2.48x more fits predicts 2.98 h on the GPU, so the measured 10.40 h puts the cost of losing MPS at 3.49x — close to the 3.5x estimated from the TabNet device experiment, and enough, multiplied by the trial change, to push a one-hour dataset past a twelve-hour cap.

TabNet on letter and tamilnadu-electricity (both 10 000 rows) needs 18.12 h and 16.19 h. At a twelve-hour cap and again at twenty they recorded NaN; letter was 36 fits of 804 short the second time. What the 18.12 h buys is 0.9841, between Random Forest (0.9820, 0.06 h) and SVC (0.9882, 0.04 h), while TabPFN-3 takes the dataset at 0.9984 in 0.72 h. On tamilnadu-electricity ten models reach a perfect 1.0000, Random Forest in 3.6 minutes against TabNet's 16.19 h — 270x for the same score.

All three models now cover all 146 datasets.