Symptom
tokenless stats summary --compare <baseline> <active> treats missing session IDs as a successful comparison. A typo or unfinished dual-run prints a 0% savings report and exits 0, so A/B scripts look like "no savings" instead of "this session was never recorded."
The Python TokenlessStats.compare client and tokenless stats diff --session already fail closed on the same empty-session case.
Reproduction
# Isolated stats DB so this does not touch ~/.tokenless/stats.db
export TOKENLESS_STATS_DB="$HOME/.tokenless-repro-compare/stats.db"
mkdir -p "$(dirname "$TOKENLESS_STATS_DB")"
# No records are inserted for these session IDs.
tokenless stats summary --compare missing-a missing-b
echo "exit=$?"
tokenless stats summary --json --compare missing-a missing-b
echo "json_exit=$?"
# Sibling command already fails closed:
tokenless stats diff --session missing-a
echo "diff_exit=$?"
On main at 9f34d035474fef4d134587b46086d13231f306ce, the first two commands succeed.
Actual
--compare with two unknown session IDs exits 0 and prints a comparison whose totals are all zero, including --json (saved_percent: 0.0). One known side plus one missing side still succeeds and reports 0% against the empty side.
tokenless stats diff --session missing-a exits 1 with No records found for session "missing-a".
Expected
--compare should fail closed when either named session has no records, with a non-zero exit and an error that names the missing side(s), matching stats diff --session and TokenlessStats.compare.
Root cause
src/tokenless/crates/tokenless-cli/src/main.rs (Commands::Stats::Summary --compare path) always called format_compare / format_compare_json after records_by_session. Empty slices format as a successful 0% report (query.rs format_compare). warn_mode_mismatch also returns immediately on empty input, so there is no warning either.
Impact
- Dual-run measurement (
TOKENLESS_COMPRESSION_ENABLED=0/1 then --compare) silently reports 0% savings when a --session-id is mistyped or one run failed to record.
--json consumers treat that 0% payload as a real comparison.- CLI and Python stats clients disagree on the same empty-session contract.
Symptom
tokenless stats summary --compare <baseline> <active>treats missing session IDs as a successful comparison. A typo or unfinished dual-run prints a 0% savings report and exits 0, so A/B scripts look like "no savings" instead of "this session was never recorded."The Python
TokenlessStats.compareclient andtokenless stats diff --sessionalready fail closed on the same empty-session case.Reproduction
On
mainat9f34d035474fef4d134587b46086d13231f306ce, the first two commands succeed.Actual
--comparewith two unknown session IDs exits 0 and prints a comparison whose totals are all zero, including--json(saved_percent: 0.0). One known side plus one missing side still succeeds and reports 0% against the empty side.tokenless stats diff --session missing-aexits 1 withNo records found for session "missing-a".Expected
--compareshould fail closed when either named session has no records, with a non-zero exit and an error that names the missing side(s), matchingstats diff --sessionandTokenlessStats.compare.Root cause
src/tokenless/crates/tokenless-cli/src/main.rs(Commands::Stats::Summary--comparepath) always calledformat_compare/format_compare_jsonafterrecords_by_session. Empty slices format as a successful 0% report (query.rsformat_compare).warn_mode_mismatchalso returns immediately on empty input, so there is no warning either.Impact
TOKENLESS_COMPRESSION_ENABLED=0/1then--compare) silently reports 0% savings when a--session-idis mistyped or one run failed to record.--jsonconsumers treat that 0% payload as a real comparison.