add dflash2 selector metrics

[curnane-lab]

Motivation

Follow-up to #772 (DFlash2). The DFlash2 selector currently reports only selector_loss / selector_accuracy / selector_coverage / selector_target_probability, which is not enough to answer the two questions that come up when tuning the selector:

• How much score mass can reranking ever move - i.e., is selector_top_k too small, or already wasteful?
• How much accepted length could a perfect reranker buy, and how much does the realized greedy path actually buy today?

This PR ports the three validation signals that the merged DFlash2 implementation in vllm-project/speculators reports (unary_candidate_target_mass_at_k, unary_top_k_oracle_accepted_length, and the realized greedy self-conditioned path), adapted to SpecForge's hard-target online pipeline. Since SpecForge captures hidden states only, the teacher-side soft "target mass" becomes the draft's own softmax mass on the same strict unary top-k candidate set.

Modifications

specforge/algorithms/common/dflash_family_model.py:

• SelectorTerms / DFlashObjectiveTerms gain six additive telemetry fields (top-k mass, oracle/path accepted lengths, path accuracy, and their block/position denominators), preserving the flat tensor-only NamedTuple contract of checkpointed_chunk_reduce.
• Inside the existing no_grad block of _selector_chunk_terms, the new metrics are computed:
  • unary_topk_probability_mass - the draft's softmax mass inside its own strict unary top-k, via two logsumexp reductions (no extra full-vocab materialization);
  • unary_topk_oracle_accepted_length - per-block accepted length under a perfect reranker: every supervised slot whose gold token sits in the unary top-k is credited. The verified anchor counts as one accepted token, and a block extends only while every earlier slot is supervised and covered;
  • selector_path_accepted_length / selector_path_accuracy - the realized greedy candidate path: each slot is re-scored with its own selected predecessor (starting from the verified anchor) instead of the gold one, mirroring the serving walk; per-position accuracy is conditioned on the earlier path being correct.
• forward exposes the four new metrics through the existing ratio_metrics (numerator, denominator) contract under the same _selector_objective_enabled gate, so they chunk-reduce and all-reduce exactly like the current selector metrics (zero denominators are clamped by the consumer).
• Loss terms, gradients, and checkpoint contents are unchanged; drafts without a candidate selector pay nothing.

tests/test_modeling/test_dflash2.py:

• test_candidate_mass_and_accepted_length_metrics: a two-block fixture with hand-computed mass, oracle, and realized-path values (rank-1 coverage vs. realized top-1 miss vs. uncovered slot).
• test_forward_reports_candidate_path_ratio_metrics: forward-level check that the four new ratio metrics are reported with the expected values.

Related Issues

• Follow-up to support dflash2 #772 (merged).

Accuracy Test

• tests.test_modeling.test_dflash2: 23 passed (21 existing + 2 new).
• Regression: tests.test_utils.test_dflash_losses, tests.test_modeling.test_draft_registry, tests.test_algorithms.test_builtin_providers all pass.
• No model-side numerics change: loss terms are untouched; all additions are no_grad telemetry.

Benchmark & Profiling

No throughput impact by design. The metrics run under no_grad inside the existing chunk pass; the realized path adds block_size - 1 small codebook-lookup/einsum scorings per chunk (vs. one full-vocab lm_head projection already there), and the mass metric avoids any full-vocab softmax materialization via two logsumexp reductions.

Checklist

• Format your code according to the Code Formatting with Pre-Commit. (ruff check clean; new test code mirrors the file's existing formatting.)
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation. (Inline comments document the metric semantics; no user-facing surface changes.)
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results. (N/A - telemetry only.)
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://sgl-fru7574.slack.com/archives/C09784E3EN6 to discuss your PR.