Skip to content

Commit 7e9f788

Browse files
committed
ci(test-shards): the density cap carries the declaration closure's own displacement when the split carries the reader slice
The reader slice's closure charge (783 s of packing weight) moves that much whole-package work onto the other shards. When the whole-package split is balanced the density cap is its mean, so no split carrying the charge could ever come back within it, at any slice count: on the 2026-11-02 refresh the cap read 1455.81 s and the CLI cut 2/3/4 ways reached 1552.68/1552.77/1552.63 s, so the full list ran the CLI whole as one 1432 s serial task. A candidate split that carries the reader slice is now judged, and repaired, against the cap plus DECLARATION_CLOSURE_CHARGE_SECONDS / (shardCount - 1); a split without it keeps the bare cap. The self-test's hard constraint is scored against the same adjusted cap; the cap-definition pin and the capFixture controls are unchanged. Four new cases pin the allowance on the refresh's frozen package weights: x2 is admitted within 1567.67 s, a split with no reader slice is allowed nothing, and a split past the cap plus its allowance is refused by name. Today's committed dataset splits byte-identically: every shard's stdout and stderr, the items, the bins and the cap; the slice decision record gains one `allowance` key. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ihL2gLghEqeDKTbDTC7Zd
1 parent b0ef25f commit 7e9f788

1 file changed

Lines changed: 162 additions & 13 deletions

File tree

‎scripts/partition-test-shards.mjs‎

Lines changed: 162 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1186,7 +1186,9 @@ export function balanceOf(bins, items = null) {
11861186
//
11871187
// ⛔ Density is never traded for balance. planShards() takes a slicing only
11881188
// when the sliced split keeps every shard within this cap, and the self-test
1189-
// pins the committed dataset's full list, split as CI splits it, within it.
1189+
// pins the committed dataset's full list, split as CI splits it, within it --
1190+
// both judged with the declaration-closure allowance below (closureAllowance())
1191+
// when the split carries the reader slice, and with no allowance otherwise.
11901192
//
11911193
// `packages` is whole-package `{name, weight}`, CI's exclusions already applied.
11921194
export function densityCap(packages, shardCount = SHARD_COUNT) {
@@ -1269,6 +1271,36 @@ export function refineToCap(bins, items, cap) {
12691271
return out;
12701272
}
12711273

1274+
// ── THE CAP'S DECLARATION-CLOSURE ALLOWANCE (ruling record 6596963071) ─────
1275+
//
1276+
// The reader slice's closure charge (DECLARATION_CLOSURE_CHARGE_SECONDS) is
1277+
// packing weight, and LPT spends it by moving that much whole-package work off
1278+
// the reader slice's shard onto the others. The density cap above is the densest
1279+
// bin of a split that carries no such charge, so when that split is balanced --
1280+
// the sliced package sits below the mean, and the cap IS the mean -- no split
1281+
// carrying the charge can come back within it, at any slice count: the charge
1282+
// alone lifts the other shards by about charge / (shards - 1). Measured on the
1283+
// 2026-11-02 refresh at 8 shards (the CLI 1432.04s, the mean 1455.81s): the
1284+
// cap read 1455.81s, and the CLI cut 2, 3 and 4 ways put its densest shard at
1285+
// 1552.68s / 1552.77s / 1552.63s, so the full list ran the CLI whole, a 1432s
1286+
// serial task, against ~500-570s on every other shard. On every dataset before
1287+
// it the cap was the CLI's own whole bin, which no slicing approaches, so the
1288+
// charge and the cap had never met.
1289+
//
1290+
// So a candidate split that CARRIES the reader slice is judged against the cap
1291+
// plus that displacement: DECLARATION_CLOSURE_CHARGE_SECONDS / (shardCount - 1),
1292+
// 111.86s at 8 shards. It is the charge's own arithmetic -- the density the
1293+
// closure pin already chose to move -- and nothing else: a split with no reader
1294+
// slice (the package whole, a run without it, a document that states its own
1295+
// slicing) is judged against the bare cap, and a split denser than the cap plus
1296+
// the charge's share is still refused, by name (renderSliceDecision()). A
1297+
// one-shard run displaces nothing and is allowed nothing.
1298+
export function closureAllowance(items, shardCount) {
1299+
return shardCount > 1 && items.some((i) => i.closure !== undefined)
1300+
? DECLARATION_CLOSURE_CHARGE_SECONDS / (shardCount - 1)
1301+
: 0;
1302+
}
1303+
12721304
// The split a run actually runs: partition(), and -- only when the split
12731305
// carries a slice -- refineToCap() against the density cap. A split with no
12741306
// slice is the plain LPT split it always was, byte for byte.
@@ -1291,7 +1323,9 @@ export function placeItems(items, shardCount, cap = Infinity) {
12911323
// SPREADS its slices land on distinct shards at this run's shard count,
12921324
// which is what makes them run in parallel at all;
12931325
// WITHIN CAP the sliced split keeps every shard within the density cap
1294-
// (densityCap), repaired by refineToCap() where LPT overshoots.
1326+
// (densityCap), plus closureAllowance() when the split carries
1327+
// the reader slice, repaired by refineToCap() where LPT
1328+
// overshoots; the final placement is repaired against the same.
12951329
//
12961330
// All three => sliced at the configured count. Any one fails => whole, and the
12971331
// decision says which. Packages are decided in name order, each against the
@@ -1321,8 +1355,11 @@ export function planShards(
13211355
const needed = own > target;
13221356
let spreads = false;
13231357
let densest = null;
1358+
let allowance = 0;
13241359
if (needed) {
1325-
const bins = placeItems(expandWith(pkg.name, pkg.sliceCount), shardCount, cap);
1360+
const candidate = expandWith(pkg.name, pkg.sliceCount);
1361+
allowance = closureAllowance(candidate, shardCount);
1362+
const bins = placeItems(candidate, shardCount, cap + allowance);
13261363
try {
13271364
assertSlicesSpread(bins);
13281365
spreads = true;
@@ -1331,24 +1368,29 @@ export function planShards(
13311368
}
13321369
densest = Math.max(...bins.map((b) => b.total));
13331370
}
1334-
const withinCap = densest !== null && densest <= cap;
1371+
const withinCap = densest !== null && densest <= cap + allowance;
13351372
const count = needed && spreads && withinCap ? pkg.sliceCount : 1;
13361373
chosen.set(pkg.name, count);
1337-
decisions.push({ name: pkg.name, configured: pkg.sliceCount, count, needed, spreads, withinCap, own, target, densest, cap });
1374+
decisions.push({ name: pkg.name, configured: pkg.sliceCount, count, needed, spreads, withinCap, own, target, densest, cap, allowance });
13381375
}
13391376
const items = expandSlices(weighed.map((p) => ({ ...p, sliceCount: chosen.get(p.name) })), readers);
1340-
return { items, bins: placeItems(items, shardCount, cap), decisions };
1377+
return { items, bins: placeItems(items, shardCount, cap + closureAllowance(items, shardCount)), decisions };
13411378
}
13421379

1343-
// One line per decision, for the shard log: what the run chose and why.
1380+
// One line per decision, for the shard log: what the run chose and why. The
1381+
// declaration-closure allowance is named whenever it was part of the verdict --
1382+
// a refusal past it, or a slicing only it admitted -- and left out when the bare
1383+
// cap already decided, so the line never claims a cap the split did not meet.
13441384
export function renderSliceDecision(d) {
13451385
const floor = `whole ${d.own.toFixed(0)}s against ${d.target.toFixed(0)}s, ${MAX_SHARD_OVER_MEAN}x the heaviest other serial task`;
13461386
const capText = Number.isFinite(d.cap) ? `${d.cap.toFixed(2)}s density cap` : 'no density cap';
1387+
const allowed =
1388+
Number.isFinite(d.cap) && (d.allowance ?? 0) > 0 ? ` plus its ${d.allowance.toFixed(2)}s declaration-closure allowance` : '';
13471389
let why;
13481390
if (!d.needed) why = `not the run's floor: ${floor}`;
13491391
else if (!d.spreads) why = `${floor}, but ${d.configured} slices cannot spread across this run's shards`;
1350-
else if (!d.withinCap) why = `${floor}, but sliced the densest shard would carry ${d.densest.toFixed(2)}s, past the ${capText}`;
1351-
else why = `${floor}; densest shard ${d.densest.toFixed(2)}s, within the ${capText}`;
1392+
else if (!d.withinCap) why = `${floor}, but sliced the densest shard would carry ${d.densest.toFixed(2)}s, past the ${capText}${allowed}`;
1393+
else why = `${floor}; densest shard ${d.densest.toFixed(2)}s, within the ${capText}${d.densest > d.cap ? allowed : ''}`;
13521394
return `${d.name}: ${d.count > 1 ? `sliced x${d.count}` : 'whole'} (${why})`;
13531395
}
13541396

@@ -1842,7 +1884,9 @@ const SELF_TEST_BATTERIES = Object.freeze({
18421884
'drift warning tier under the red (#16465)': 12,
18431885
'file-level slice items (#16173)': 20,
18441886
'file-level slices reach vitest through OS_TEST_SHARD (#19278)': 11,
1845-
'density cap and the per-run slice decision (#22075)': 14,
1887+
// 4 of these 18 are the declaration-closure allowance (ruling record
1888+
// 6596963071): 1 unit, 3 on the frozen 2026-11-02 shape.
1889+
'density cap and the per-run slice decision (#22075)': 18,
18461890
'the drift step reports on every event, `schedule` included, and refuses a missing event': 8,
18471891
// The nightly tiers state their own slicing (`sliceAcrossShards`). 6 of these
18481892
// 17 read the live tree: 3 read test-nightly-tiers.yml back (the matrix, the
@@ -3198,14 +3242,18 @@ function selfTest() {
31983242
// splits it, puts no shard past the cap -- and that split slices, so the cap
31993243
// is graded on the shape it exists to bound (pin 3b proves the slice is
32003244
// there). This is the case that fails when a slicing packs a shard denser
3201-
// than the split the drift step was proven green on.
3245+
// than the split the drift step was proven green on. Scored against the cap
3246+
// planShards() judged that split by: plus closureAllowance() when it carries
3247+
// the reader slice, which the full list does whenever it slices the CLI.
32023248
check(() => {
3203-
const over = real.map((b, i) => ({ shard: i + 1, total: b.total })).filter((b) => b.total > datasetCap);
3249+
const allowance = closureAllowance(datasetItems, SHARD_COUNT);
3250+
const over = real.map((b, i) => ({ shard: i + 1, total: b.total })).filter((b) => b.total > datasetCap + allowance);
32043251
if (over.length > 0 || !datasetItems.some((i) => i.slice)) {
32053252
throw new Error(
32063253
`density cap: the committed dataset's full list, split as CI splits it, puts ` +
32073254
`${over.map((b) => `shard ${b.shard} at ${b.total.toFixed(2)}s`).join(', ') || 'no slice on any shard'} ` +
3208-
`against the ${datasetCap.toFixed(2)}s cap (the whole-package split's densest bin). ` +
3255+
`against the ${datasetCap.toFixed(2)}s cap (the whole-package split's densest bin) ` +
3256+
`plus a ${allowance.toFixed(2)}s declaration-closure allowance. ` +
32093257
`Bins: ${real.map((b) => b.total.toFixed(2)).join('/')}s; ` +
32103258
(datasetPlan.decisions.map(renderSliceDecision).join('; ') || 'no slicing decision') + '.'
32113259
);
@@ -3234,6 +3282,107 @@ function selfTest() {
32343282
}
32353283
});
32363284

3285+
// THE DECLARATION-CLOSURE ALLOWANCE (closureAllowance(), ruling record
3286+
// 6596963071), on the shape that needed it: the 2026-11-02 refresh's package
3287+
// weights, frozen here rather than read from the dataset file so the pin does
3288+
// not move with a refresh, the CLI cut 2 ways, at 8 shards, with the reader
3289+
// slice and its rider fixed. The whole split is balanced, so the bare cap is
3290+
// the mean (1455.81s) and the charge lifts the other shards past it; carried,
3291+
// the reader slice earns 783 / 7 = 111.86s and the split is judged against
3292+
// 1567.67s.
3293+
const refreshedWeights = [
3294+
'cli 1432.04 spec 1008.78 runtime 987.16 objectql 693.87 rest 630.61 metadata-protocol 566.5',
3295+
'service-automation 526.88 plugin-security 515.26 plugin-auth 450.08 driver-sql 267.52',
3296+
'plugin-audit 261.68 plugin-approvals 243.57 cloud-connection 233.46 lint 207.88 verify 189.81',
3297+
'plugin-sharing 183.78 service-storage 183.37 client 164.18 service-messaging 145.67',
3298+
'service-analytics 144 driver-memory 141.87 service-settings 140.81 service-datasource 136.01',
3299+
'driver-turso 108.76 organizations 107.27 core 106.66 plugin-email 101.12 example-showcase 90.51',
3300+
'connector-app-github 82.27 driver-sqlite-wasm 78.42 platform-objects 77 trigger-record-change 75.65',
3301+
'plugin-hono-server 72.94 mcp 70.64 connector-app-stripe 69.8 plugin-dev 60.84 plugin-webhooks 53.92',
3302+
'service-mailbox 53.18 connector-mcp 50.82 http-conformance 49.59 plugin-pinyin-search 49.43',
3303+
'driver-mongodb 49.14 example-todo 47.71 service-package 45.89 metadata 43.72 connector-openapi 43.28',
3304+
'formula 43.23 service-queue 40.79 driver-odata 35.98 hono 35.4 vitest-filter-preflight 31.35',
3305+
'service-print 29.15 example-crm 27.79 trigger-schedule 26.54 types 25.69 metadata-core 25.24',
3306+
'create-objectstack 23.84 service-knowledge 22.56 connector-rest 22.09 service-sms 21.48',
3307+
'metadata-fs 19.4 connector-slack 18.54 downstream-contract 17.77 service-job 14.35',
3308+
'service-cluster 13.58 example-embed-objectql 13.14 service-i18n 13.13 client-react 12.26',
3309+
'trigger-api 10.48 service-cluster-redis 9.22 service-realtime 8.19 eslint-plugin 6.46',
3310+
'sdui-parser 5.34 embedder-openai 4.47 observability 3.47 knowledge-ragflow 3.23 connector-imap 3.2',
3311+
'service-cache 2.77 knowledge-memory 2.69',
3312+
].join(' ').split(' ');
3313+
// The packages that run `test` and `test:repo` as two tasks, as the workspace
3314+
// manifests read on that refresh; without them spec's 1008.78s would be the
3315+
// floor and the CLI would not be NEEDED at all.
3316+
const refreshedTwoTask = new Set(['core', 'objectql', 'rest', 'runtime', 'spec', 'types']);
3317+
const refreshedShape = (cliSlices) => {
3318+
const out = [];
3319+
for (let i = 0; i < refreshedWeights.length; i += 2) {
3320+
const short = refreshedWeights[i];
3321+
out.push({
3322+
name: short === 'create-objectstack' ? short : `@objectstack/${short}`,
3323+
weight: Number(refreshedWeights[i + 1]),
3324+
tasks: refreshedTwoTask.has(short) ? 2 : 1,
3325+
sliceCount: short === 'cli' ? cliSlices : 1,
3326+
});
3327+
}
3328+
return out;
3329+
};
3330+
const refreshedReaders = Object.freeze({
3331+
slices: Object.freeze({ '@objectstack/cli': 1 }),
3332+
packages: Object.freeze(['@objectstack/cli', '@objectstack/metadata']),
3333+
});
3334+
check(() => {
3335+
// The allowance itself: the charge's share of the other shards when a
3336+
// closure is carried, nothing when none is, nothing on one shard.
3337+
const carried = [{ name: 'a 1/2', weight: 1, slice: { index: 1, count: 2 }, closure: DECLARATION_CLOSURE_CHARGE_SECONDS }];
3338+
const plain = [{ name: 'a', weight: 1, slice: null }];
3339+
const got = [closureAllowance(carried, 8), closureAllowance(plain, 8), closureAllowance(carried, 1)];
3340+
if (got[0] !== DECLARATION_CLOSURE_CHARGE_SECONDS / 7 || got[1] !== 0 || got[2] !== 0) {
3341+
throw new Error(`closure allowance: carried at 8 / plain at 8 / carried at 1 read ${got.join(' / ')}s`);
3342+
}
3343+
});
3344+
check(() => {
3345+
// Carried: the refreshed shape slices x2, its densest shard past the bare
3346+
// cap and within the cap plus the allowance, and the log names the allowance.
3347+
const shape = refreshedShape(2);
3348+
const cap = densityCap(shape, 8);
3349+
const plan = planShards(shape, 8, { cap, readers: refreshedReaders });
3350+
const [d] = plan.decisions;
3351+
const judged = cap + d.allowance;
3352+
if (
3353+
shape.length !== 79 || cap.toFixed(2) !== '1455.81' || d.allowance.toFixed(2) !== '111.86' ||
3354+
judged.toFixed(2) !== '1567.67' || d.count !== 2 || !d.withinCap || !(d.densest > cap) ||
3355+
!(d.densest <= judged) || balanceOf(plan.bins).max > judged ||
3356+
!renderSliceDecision(d).includes('plus its 111.86s declaration-closure allowance')
3357+
) {
3358+
throw new Error(
3359+
`closure allowance: the refreshed shape (${shape.length} packages, cap ${cap.toFixed(2)}s, allowance ` +
3360+
`${d.allowance.toFixed(2)}s) was judged ${renderSliceDecision(d)}`
3361+
);
3362+
}
3363+
assertSlicesSpread(plan.bins);
3364+
});
3365+
check(() => {
3366+
// Not carried, not allowed: the same shape with no reader slice in the
3367+
// split is judged against the bare cap.
3368+
const shape = refreshedShape(2);
3369+
const [d] = planShards(shape, 8, { cap: densityCap(shape, 8), readers: NO_DECLARATION_READERS }).decisions;
3370+
if (d.allowance !== 0 || renderSliceDecision(d).includes('allowance')) {
3371+
throw new Error(`closure allowance: a split with no reader slice was allowed ${d.allowance}s: ${renderSliceDecision(d)}`);
3372+
}
3373+
});
3374+
check(() => {
3375+
// The allowance widens the cap and does not lift it: the carried split
3376+
// against a cap one second short of admitting it is refused, by name.
3377+
const shape = refreshedShape(2);
3378+
const [taken] = planShards(shape, 8, { cap: densityCap(shape, 8), readers: refreshedReaders }).decisions;
3379+
const short = taken.densest - taken.allowance - 1;
3380+
const [d] = planShards(shape, 8, { cap: short, readers: refreshedReaders }).decisions;
3381+
if (d.count !== 1 || d.withinCap || !renderSliceDecision(d).includes('declaration-closure allowance')) {
3382+
throw new Error(`closure allowance: a split past the cap plus its allowance was taken: ${renderSliceDecision(d)}`);
3383+
}
3384+
});
3385+
32373386
// The REPAIR: LPT overshoots a cap by where its last small items fall, and
32383387
// refineToCap() moves whole packages until the split is back within it. Two
32393388
// shards, `big` cut in two, and five packages: the whole split's densest bin

0 commit comments

Comments
 (0)