Repository navigation
conformance: coverage semantics, and nine more languages measured - #3092
Merged
Merged
Conversation
…ivergence The runner compares only the sections the reference and the language both traced, counts the rest toward a per-language coverage printed beside its totals, and prints a dash in a cell whose api the language ran none of — a green mark would say it agrees about something it never ran. The five languages measured today are untouched: js still 🔴 25, moon, yue and fennel clean, each now reading coverage 127/127. Ten languages are coming, and two of them (gravity, miniscript) have no error handling: the 19 refused-call probes cannot be written there at all — a refused call ends the script and fails the measurement, exit 2 — so without this they would carry 19 permanent red marks burying whatever they actually disagree about. A section only the language traced stays a divergence: that is the port out of sync with the reference. Verified both ways: with every file complete the marks are unchanged, and with the two rrect sections removed from js.js it reads 119/127 with dashes in those cells instead of two red marks. Co-Authored-By: Claude Code <noreply@anthropic.com>
The first of the ten remaining languages. The port is the js file in squirrel's clothes: the same sections in the same order, every literal double-quoted (squirrel's 'c' is a character constant — 'ab' does not compile, which the first draft learned the hard way), and a prelude that asks the root table whether the function exists and runs each probe under try/catch. It measures 127/127 — every refusal the reference records, squirrel records too — and disagrees three times: - `fset/missing-bool`: lua refuses the call, squirrel records nothing at all. Fourteen wrappers (fget, fset, the vqt family and the fft family) call sq_throwerror without returning it, so the error is set in the VM and the closure still reports success: the refused call silently does nothing. - `pix/string-arg`: lua converts the numeric string "1" to 1 and writes at (1,2); squirrel's number reader converts any non-number to 0 and writes at (0,2). - `ttri/basic`: lua passes no chromakey array; squirrel passes a one-element [0], so color 0 is masked — the same divergence js has. Findings, not fixes: the change that measures does not change the binding. Co-Authored-By: Claude Code <noreply@anthropic.com>
mrb_trace called core->data->trace directly, skipping core->api — the only binding whose trace does (wren's other site is the System.print hook, not TIC.trace). Behavior is identical for a cart, tic_api_trace being the same call with a null guard, but nothing wrapped around the API table sees ruby's trace, the conformance record among them: every one of a ruby probe's markers vanished, and the language could not be measured at all. Found while porting ruby (squirrel landed first): trace is in TIC_API_LIST, so this wrapper was the one place not calling through it. Co-Authored-By: Claude Code <noreply@anthropic.com>
The second of the ten. The port is the js file in ruby's clothes: the same sections, `probe(id) do ... end`, a prelude that asks `respond_to?(name, true)` and runs each probe under begin/rescue. It needed one fix before it could be measured at all: mruby's trace bypassed the API table, so the markers never arrived (e86f941, the commit before this one). What it found, in four kinds: - `rrect/missing-radius` and `rrectb/missing-radius`: ruby takes the documented five arguments ("iiiii|if" with explicit defaults) where lua refuses — the arity split the rrect change recorded, now measured on a second language. - Ruby refuses where lua coerces: cls('x'), pix('1', 2, 4), mget('x', 0) and print() all raise in mruby and are coerced or accepted in lua — lua turns the string "1" into 1, ruby raises. - `spr/basic`, `map/defaults` and `map/size`: `mrb_spr` and `mrb_map` read an uninitialized `mrb_value` when the color argument is absent, so the call arrives with a garbage one-element chroma array (128 and 65 in this run, and not the same values next run). That is undefined behavior, not just a divergence: a garbage type tag that passes mrb_array_p reads a bogus pointer. `mrb_ttri` initializes its chroma (0xff) and passes the same `1:[255]` js passes. - `keyp/no-key`: ruby passes (tic_key)-1 where lua passes tic_key_unknown; the core reads both as "any key", so the behavior matches and only the recorded argument differs. `exit/call`: ruby's exit goes to the data callback, not the API table (miniscript has the same two sites), so the call is invisible to the record. Findings, not fixes: the change that measures does not change the binding. Co-Authored-By: Claude Code <noreply@anthropic.com>
The third of wave 1, and the loudest so far. PocketPy's lambda cannot hold
statements, so every probe body is a def and the call follows it; the prelude
asks globals() and runs each body under try/except. Coverage is complete.
What it found:
- `paint` is not bound at all: both paint probes read `!missing` — the API
simply does not exist for a python cart.
- `poke(addr, value, bits)` ignores `bits`: py_pokebits never reads its third
argument, so `poke(32, 9, 4)` writes a whole byte and the nibble reads back 0
where lua reads 9.
- `pmem(index, value)` does not return the old value: lua's wrapper reads and
then writes, so the call returns the previous value and leaves two records;
python's makes one call and returns None.
- Python refuses what lua coerces or defaults: cls('x'), pix('1', 2, 4),
mget('x', 0), btn() with no arguments, pix(10.5, 10.5) — a float for a
declared int — and rrect/rrectb(…, 4) — an int for the declared
`exponent: float`. PocketPy's typed signatures raise where lua's reader
truncates.
- `clip()` raises: py_clip reads and type-checks four arguments
unconditionally, so the documented reset is not callable at all.
- `keyp()`'s default period is -17, as declared in the binding, where lua
passes -1.
- An explicit `-1` colorkey is `[]` here and `[255]` in lua — the same "no
transparent color", encoded differently (spr/scale, spr/flip-rotate,
map/scale).
- `reset()` is recorded here and not in lua, whose reset sets
`core->state.initialized = false` directly instead of calling the API — a
bypass in the reference this time.
- The arity split arrives as expected: rrect/rrectb take the documented five
arguments.
Findings, not fixes: the change that measures does not change the binding.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The first namespaced language: the api is a module, so every call is
tic80/name and the prelude runs each body under try/catch. Janet resolves
symbols at compile time and sandboxes the introspection the other languages
use, so a body can only name functions that exist — this binding has all 61,
and a language that lacked one would write !missing where the call would go.
What it found:
- `ffts` calls the plain `fft`: janet_ffts returns core->api.fft, so the
smoothed spectrum is not smoothed. The rest of the family — fftr, fftrs and
the eight vqt variants — calls what it names; this one is the wasm-rrect
copy-paste class, caught by the port.
- `music`'s default sustain is true (janet_optboolean(argv, argc, 4, true))
where the documented default, and lua's, is false.
- lua's `music(track)` stops the current track first — music(-1,0,0,false,
false,-1,-1) — before the call the probe asked for, so its sections carry two
records against janet's one. The extra call is the reference's, not a janet
behavior.
- Janet refuses what lua coerces: cls('x'), pix('1', 2, 4), mget('x', 0),
btn(), pix(10.5, 10.5) — the typed argv raises on a string, a float for an
int, an argument that is not there.
- `key()` and `keyp()` with no arguments pass 255 as the unknown-key sentinel
where lua passes tic_key_unknown; the core reads both as "any key", so the
behavior matches and the recorded argument does not (ruby's finding again).
- `vbank()` — the read form lua answers with the current bank — raises.
Findings, not fixes: the change that measures does not change the binding.
Co-Authored-By: Claude Code <noreply@anthropic.com>
mrb_spr and mrb_map declared their optional color as a bare mrb_value and tested it with mrb_array_p/mrb_fixnum_p even when the caller passed none, so the calls arrived with whatever the stack held: the conformance record showed garbage one-element chroma arrays (128 and 65 in one run, other values the next), and a garbage type tag that passes mrb_array_p would read a bogus pointer — undefined behavior, not just a divergence. Both now start from mrb_nil_value() and treat a missing color as none: spr passes count 0, map passes NULL — what lua passes for the same calls, so ruby's three divergences there are gone (12 → 9) and coverage stays 127/127. Recorded, not fixed: mrb_spr's array branch does count++ after a loop that already counted the entries, so an array colorkey masks one palette index the caller never listed. No probe passes an array there; found by reading. Co-Authored-By: Claude Code <noreply@anthropic.com>
scheme_map read x, y, w, h, sx and sy with s7_car/s7_cadr/s7_list_ref before it looked at the argument count, and every one of them is optional: a bare map() dereferenced the empty list and took the conformance runner down with it — SIGSEGV in scheme_map, the whole table lost, not just scheme's column. map(0, 0, 4, 4) had the same hole in sx and sy. Six guarded reads with the documented defaults (w=30, h=17, the rest 0), the shape its neighbours already use. Found by the scheme port: a zero-argument sweep of all 61 APIs showed map is the only one that crashes. Co-Authored-By: Claude Code <noreply@anthropic.com>
The second namespaced language: the api lives under t80::, so every call is (t80::name ...) and the prelude runs each body under (catch #t ...), which takes whatever the binding raises. It needed one fix before it could be measured at all — map() crashed the runner (previous commit). What it found: - Scheme refuses what lua coerces: s7_integer on a boolean raises, so print(..., false, ...), music(..., true, ...), fset(0, 0, true) and sync(..., false) are refusals; on a float it answers 0, so pix(10.5, 11.9, 4) writes at (0,0) where lua writes at (10,11) — and line(0.4, 1.6, ...) draws at (0,0). The string argument lua turns into 1 arrives as 0 here too. - `clip()` — the reset form — is refused, so the clip region cannot be reset: the clip/set probe ends in a refusal where lua passes the full screen. - `pmem(1000)` is performed instead of refused: the index is not checked, and the out-of-range probe records a read where lua raises. - `pmem(index, value)` does not return the old value, like python's. - The arity split (rrect/rrectb take the documented five), the unknown-key sentinel (255 against lua's tic_key_unknown), lua's extra music stop call and lua's own reset bypass — the same shapes ruby, janet and python showed. Findings, not fixes: the change that measures does not change the binding. Co-Authored-By: Claude Code <noreply@anthropic.com>
The first language with no try/catch, and the coverage change's test from the other side — which it turned out not to need: nothing among the reference's 127 calls is refused by this binding. It coerces what lua coerces and defaults what lua defaults, so the file carries every section and the coverage reads 127/127; the prelude answers !missing from the module's own hasIndex for the fourteen APIs the module does not carry. What it found: - `keyp` is `btnp`: the function registered under "keyp" calls core->api.btnp in both of its branches, so a MiniScript cart asking about the keyboard is asking about the gamepad. - `poke1` masks its value with & 0x1 and `poke2` with & 0x3 — the bit count where the field mask belongs (poke4's & 0xF is the nibble mask, which is why only the first two are wrong): poke1(16, 7) stores 1, poke2(24, 44) stores 0. - Fourteen APIs are absent from the module: paint, the four fft variants and the eight vqt ones — their sections read !missing. - The defaults drift from lua's: sfx note and duration default to -1 where lua passes 0, music's loop is false where lua's is true, print() of nothing prints "null" (js prints "undefined"), cls(99) wraps to 3. - `pmem(1000)` is performed instead of refused, and `pmem(index, value)` does not return the old value — python's finding and scheme's again. - The refusal family is the js shape: a call lua refuses has its missing arguments defaulted here, silently. Findings, not fixes: the change that measures does not change the binding. Co-Authored-By: Claude Code <noreply@anthropic.com>
The first structural language: a wren cart is a class, so the probes live in
Game's constructor and the prelude is a method of it. Each body runs in a
fiber whose try() catches what the binding refuses — wren has no reflection,
so !missing has no meaning here: the class declares the whole API and a name
it lacked would not compile.
What it found:
- The deprecated-OVR shim runs after every tick: vbank(1), cls(0), vbank(back)
— bank 1's VRAM is cleared every frame for carts that never asked for OVR.
lua's shim is guarded by lua_isfunction, wren's by if(overline_handle),
which is always true because the prelude defines OVR(){}. The port gives it
a section of its own (#wren-bind/overlay-shim) so the three calls sit in a
named row instead of inside reset/call.
- `mset(x, y)` writes a stale third argument: wren_mset reads slot 3
unconditionally and the dispatcher maps the two-argument overload to it, so
the tile written is whatever the previous call left in the slot — mset(6, 6)
after mset(6, 6, 7) writes 7.
- `key()`, `keyp()`, `fft()` and `ffts()` with no arguments are refused: the
dispatcher carries the no-argument entries, the class's foreign
declarations do not — the dangling-entry shape rrect's five-argument
overload has here too.
- `pix('1', 2, 4)` writes at (0,2): the string coerces to 0, as in squirrel.
- lua's own extra music stop call, the reference's shape again.
Findings, not fixes: the change that measures does not change the binding.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The first language the coverage change was built for. Gravity's binding refuses (RETURN_ERROR) instead of coercing, and there is no try/catch, so the eighteen calls the reference records as !error would end the script and fail the measurement; the port carries the other 109 sections and lets the coverage line say so — 109/127, with no red marks for what it cannot run. What it found, in the six it can report: - The arity split: rrect and rrectb take the documented five arguments, so gravity joins js, python, scheme, miniscript and ruby against lua's refusal. - An explicit -1 colorkey arrives as [] here and [255] in lua — the same "no transparent color", encoded differently (spr/scale, spr/flip-rotate, map/scale); python's shape. - `reset()` is recorded here and not in lua, whose reset sets `core->state.initialized` directly — the reference's own bypass again. Findings, not fixes: the change that measures does not change the binding. Co-Authored-By: Claude Code <noreply@anthropic.com>
The shim — vbank(1), cls(0), vbank(back) — wraps the OVR callback so a cart written for the deprecated API can draw its overlay on bank 1. It was gated on `if(overline_handle)`, and the prelude gives every cart an empty OVR(), so the handle always existed: every wren cart switched to bank 1 and cleared it on every frame, whether or not it had an overlay to draw. lua gates on lua_isfunction, js on JS_IsFunction, squirrel on the root table; wren has no reflection over methods, so the cart's source answers — a method the "OVR()" call handle can resolve cannot be defined without spelling `OVR(`, so a missed match means the shim runs when it need not (the old behavior), never that a cart's overlay goes uncalled. The conformance probe file drops the section it kept for the shim's three calls, and the artificial wren-bind row leaves the table: 61 api rows again, wren 10 → 9 divergences. Co-Authored-By: Claude Code <noreply@anthropic.com>
…s why it has none Both from the review of the nine ports. squirrel's check called getroottable().rawget(name), which *throws* when the key is absent — and the try sat below it, so a name the root table lacked would have killed the cart instead of reading !missing. An `in` test with an index replaces it; verified with a probe of a name that does not exist, which now traces !missing and lets the cart live on. scheme has no existence check at all — a name the binding lacked raises inside the catch and reads as a refusal, not as a gap, which is janet's situation — and its header now says so, as janet's does. Co-Authored-By: Claude Code <noreply@anthropic.com>
mrb_exit called core->data->exit directly, skipping core->api — the same bypass trace had (fixed in e86f941) and the last one in the binding: a ruby cart's exit() never reached the API table, so nothing wrapped around it saw the call, the conformance record among them. tic_api_exit is the same callback call, so behavior is unchanged; ruby's exit/call divergence is gone (9 → 8). miniscript's two data->exit sites stay: they are the binding's own "the interpreter is done" signal after a tick, not the exit() API. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The conformance table watched five of the fifteen script languages. This measures eight more — and prints what each one found.
luais the reference,js,moon,yueandfennelwere already in; added:squirrel,ruby,python,janet,scheme,miniscript,wren,gravity— 13 columns, each a probe file in that language's own form with its own prelude. Still on the not-measured line:wasm(its cart is a compiled module, not text) andforth(moved to its own change once the porting showed it is a hand table of 61 stack shapes, not a translation).The table itself is in the Checks run — Actions → this PR's Checks workflow → the
testsjob → Summary (the runner prints markdown to the step summary). It lists every api × language, the divergences in full, and acoverage n/127per language.The one runner change: a section a language cannot express is coverage
gravityhas no try/catch, so the eighteen calls the reference records as!errorwould end the script and fail the measurement; its file omits them and reads 109/127, with no red marks for what it cannot run. A section a language lacks is coverage now, not a divergence; a section only a language traced stays a divergence. The five languages that existed are untouched (127/127 each, same marks).What the ports found
sq_throwerrorwithout returning it:fset(0,1)silently does nothing instead of refusing.spr/mapread an uninitializedmrb_valuewhen no colour is given: a garbage colorkey, and a garbage type tag that passesmrb_array_pwould read a bogus pointer. Fixed here, as are itstraceandexit, which both bypassed the API table.paintis not bound at all;pokeignores itsbitsargument;clip()cannot reset;pmem(index, value)does not return the old value; typed signatures refuse what lua coerces.fftscalls the plainfft: the smoothed spectrum is not smoothed.map()with fewer than six arguments crashed the process (SIGSEGV, fixed here);clip()is refused; floats become 0.keypisbtnp(both branches callcore->api.btnp);poke1masks with& 0x1andpoke2with& 0x3— the bit count where the field mask belongs;paintand the fft/vqt families are absent.vbank(1),cls(0),vbank(back)), clearing VRAM bank 1 on every frame for carts that never asked for an overlay. Fixed here: the shim is now gated on the cart actually definingOVR. Also found:mset(x, y)writes a stale third argument;key(),keyp(),fft()with no arguments are declared in the dispatcher but not in the class.[255]-against-[]colorkey encoding.Cross-cutting, all re-measured per language: lua refuses a five-argument
rrect/rrectbwhile js, ruby, python, scheme, miniscript and gravity take the documentedradius=-1 exponent=2; lua'smusic(track)makes a stop call first and itsresetbypasses the API table; the "unknown key" sentinel is 255 in ruby, janet and scheme against lua'stic_key_unknown.Fixes included (each named in its commit)
Five binding fixes, all crash- or measurement-class: mruby's
traceandexitdid not go through the API table (the first made ruby unmeasurable); scheme'smap()crashed the runner; mruby'sspr/mapread uninitialized values; wren's overlay shim cleared bank 1 every frame for carts without an overlay. Two probe-file fixes came out of the review of the ports (squirrel's!missingcheck could not fire; scheme's header now says it has no existence check). Everything else is a finding, not a fix — the change measures.Checks
api-conformance tests/conformance/probes: exit 0, 13 languages,js🔴 25 (unchanged),moon/yue/fennelclean.ctest: 15/15.The planning artifacts live in the gitignored
openspec/changes/conformance-all-languages/(coverage semantics, per-language findings, porting notes).🤖 Generated with Claude Code