Repository navigation
feat: neural card recognition models (0.2.0) - #2
Merged
Merged
Conversation
bdon_vision.scenes lays native card tiles out with the list widget geometry (1920x1080 reference canvas, centred six-member or four-snap rows, top padding and scroll offset) over blurred backgrounds, with a header bar, light UI distractors, out-of-gallery tiles recorded as foreign_cards and card-free screens. Truth files stay bdon-synthetic/2 compatible; added keys are listed in schema_extension. Generation is chunked over worker processes, resumable and deterministic per scene index; verify and stats subcommands check hashes and summarise a set. A separate final acquisition profile uses screen sizes and quality steps that appear in no other profile. List tiles of the BD rarity are rendered, and cropped UI sprites are reused across renderer processes.
A stride-8 CenterNet-style network (MobileNetV3-Large with a light FPN) predicts member and snap tile centres, sizes and offsets in a whole screenshot without a reference gallery. Includes NumPy target encoding and decoding with visibility and aspect filters, scene-set training with scale, crop, mosaic, resampling and compression augmentation, detection metrics with ignore regions, ONNX export with a CPU consistency check, and unit tests on synthetic inputs.
Member and snap tiles get separate retrieval encoders whose inputs follow each kind's artwork window (member 160x128, snap 128x224). Includes per-kind crop banks built from scene truth files, open-set retrieval metrics with leave-true-out rejection, training with optional distillation and cross-kind identities, threshold selection, and ONNX export with gallery vectors bound to the artwork bytes they came from.
Member and snap cards get separate parameter-field readers (106 and 101 classes) and six-class rank-icon classifiers. Crops are defined relative to the card tile rectangle, so the same geometry serves crop banks, training and inference. - kind_fields: crop geometry, labels, acceptance rules, misread sweep and scene crop-bank builder (catalog and foreign-art cards) - kind_hardcrops: rendered acquisition-degraded crops with optional UI-panel occlusion labelled by covered glyph or icon pixels - kind_pose: locator pose error per kind and the induced crop shift - train_kind_fields / train_kind_rank: training with per-checkpoint zero-misread threshold selection, ONNX export (opset 17, variable batch) and ONNX Runtime consistency checks - select_kind_models: checkpoint selection on saved checkpoints - evaluate_kind_fields: stratified evaluation on scene sets, crop banks and detected tiles
Engine.scan now runs the card-frame locator on the whole screenshot, identifies each tile by nearest-neighbour search in a per-kind artwork gallery and reads level, training count and card rank with the per-kind field and rank models. All seven models and their acceptance thresholds are pinned by models/recognition.json and checked by SHA-256 at start-up. - tiles whose identity is not accepted are returned in a new `unidentified` list with the nearest gallery card as a review candidate; they never enter the merged box - galleries are cached per kind in models/<kind>-gallery.npz, keyed by the encoder hash, catalog order and reference artwork bytes, and rebuilt by `prepare`; new cards need no model change - fields and rank icons that are not fully inside the image stay unknown - observations schema v2 and scan-result/2 describe the new card fields, the `unidentified` list and recognition provenance v2 - evaluate_engine scores any engine through the scan contract: one-to-one tile pairing, identity, foreign-art rejection, visible and hidden fields, stratified by kind, locale, view, resolution, card width and occlusion - verify_gallery_update checks that removed cards come back unidentified and are identified again after re-adding them, with unchanged models - the template, RootSIFT and OCR inference paths and their diagnostics are removed, together with training modules kept only in the frozen reproduction stages
- README, model card and contract describe the locator, encoder, field and rank pipeline, models/recognition.json, the gallery caches and the `unidentified` output - training guide lists the scene, crop-bank, training, export, selection and evaluation commands with the seeds used for the released weights - data card and coverage list every full-screen and crop set, resolutions per profile, identity split and field distributions - evaluation card reports the frozen final-profile results and the real regression set; reports/ holds model selection, scene coverage and the stratified acceptance summaries - reproduction guide covers the training chain of the shared initial encoder and field weights - third-party notices add the MobileNetV3-Large backbone of the locator
Bump the package version and regenerate public-manifest.json for the release source tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BoxLens 0.2.0 replaces the scan pipeline with seven ONNX models, all on CPU through ONNX Runtime:
locator.onnx): MobileNetV3-Large, stride-8 centre heatmap with tile size and offset heads; finds member and Snap tiles on the whole screenshot, including tiles cut by the screen edge.member-encoder.onnx,snap-encoder.onnx): 128-d embeddings matched against a local gallery built from the user's own artwork. An identity is accepted only above a similarity and margin threshold; other tiles go to a newunidentifiedlist with the nearest card as a review candidate. New cards only need a gallery rebuild.member-fields.onnx,snap-fields.onnx) and card-rank classifiers (member-rank.onnx,snap-rank.onnx); values are emitted only when the crop lies fully inside the image and passes the acceptance threshold.models/recognition.jsonpins every file by SHA-256 together with its input/output description and thresholds. The observations schema moves to v2 and the HTTP response toscan-result/2.Also included: the list-screen scene generator, training, export and selection modules for every model,
evaluate_enginefor end-to-end scoring through the scan contract, and docs and reports for the released weights.Evaluation
Models and thresholds were frozen on development sets before scoring two final-profile sets (2,000 screenshots each, resolutions and compression settings not used in training):
Real regression set: 129/129 identities, all visible fields correct. Strata and limits are in
docs/EVALUATION.mdandreports/acceptance/.Checks
pytest -q tests: 127 passedvalidate_contract.pyon the scan request and response examples, and on CLIbox.json/observations.jsonoutput