Repository navigation
fix(sampling): bound top-k before int32 conversion - #790
Merged
Merged
Conversation
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 23, 2026
…int32 conversion
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 23, 2026
FlashML-org#790 FlashML-org#788 adopted, FlashML-org#802 superseded by FlashML-org#804, FlashML-org#809 FlashML-org#792 drafts deferred; the 11 rebased drafts stay deferred at their new heads. Assisted-by: Claude Opus 5.5
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Nov 2, 2026
FlashML-org#850 adopted with fixups, FlashML-org#848/FlashML-org#849 superseded (FlashML-org#790/FlashML-org#788), FlashML-org#845 and FlashML-org#844 deferred (no NVFP4 qwen4exp GGUF; GPU driver down). FlashML-org#316 was closed upstream without merge and is kept for the ROCm branches. Assisted-by: Claude Opus 5.5
Collaborator
|
@wenlive |
Contributor
Author
|
sure |
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Nov 4, 2026
Upstream dacac5f..628b88f. Conflict resolutions: - FlashML-org#814 timings: upstream's final version replaces the PR head next adopted (a4a62a5). `timings` now comes from upstream's scheduler-side ReqStats (queue_ms, unrounded spans, draft_n/draft_n_accepted) on the new DetokenizeMsg/UserReply `timings` field; next's old API-side timings_block / gen_timings_block are gone, so there is one source. Option A per user: --enable-timings-report defaults ON on next (BooleanOptionalAction, --no-enable-timings-report turns it off; upstream defaults off). FlashML-org#504 metrics are unchanged: GenTimer + prefill_ms wire field, now on GenDone/GenResult.gen_timings (GenDone/GenResult.timings is upstream's dict); GenTimings drops its unused prompt_ms/predicted_ms. The ReqStats stamps were added to next's own _drain_single_token and _drain_multi_token paths (_drain_verify/MTP came from upstream). DFlash draft counters come from a new ForwardOutput.draft_len (verify_len - 1). Streams carry timings on the last chunk only (upstream). /v1/completions keeps next's n>1 fan-out, disconnect aborts and error codes, and reports prompt 0 / choice 0 timings. Responses passes both metrics= and state=; in streams timings ride the event data, not the response snapshot. - FlashML-org#790, FlashML-org#651: identical to next's earlier adoptions (e920eb4, 56ed267); no code change. For the FlashML-org#651 test, next's pool-pressure variant (371d985) is kept. - FlashML-org#854 attention: next's fp8/nvfp4 KV branches are kept, and every slot load takes upstream's KV_OFFSET_DTYPE cast. - FlashML-org#764 engine: upstream's _disable_unsafe_cpu_moe_graphs sits next to next's DFlash helpers. - FlashML-org#851/FlashML-org#845/FlashML-org#856/FlashML-org#858/FlashML-org#859: no conflicts beyond docs; cli.md merges next's DFlash rows with FlashML-org#856's GGUF/mtp sidecar wording. - core.py: Req gains `stats`. Upstream's output_token_counts is dropped, since next does penalties through token_pool. - tokenizer/server.py, message/*: both next's prefill_ms/logprobs and upstream's timings are kept. Assisted-by: Claude Opus 5.5
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Nov 4, 2026
Upstream merged FlashML-org#651, FlashML-org#790, FlashML-org#814 (adopted here before), and FlashML-org#764, FlashML-org#845, FlashML-org#851 (deferred here); they now come through the sync. FlashML-org#851 was marked closed by the render before upstream/main had its squash commit. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #789.
Clamp
top_kto the vocabulary size before int32 conversion so oversized values such as2**31do not fail batch preparation. Full-vocabulary values become unfiltered sampling; bounded rows, top-p and greedy rows retain their behavior.Validation before cleanup at
278267e: 16 sampler regression cases and 17 existing fallback regression cases passed (33 total); 10 sampler cases failed on main546c5fe. The sampler test file is no longer included in this PR. Cleanup only removes tests and comments; GPU tests were not rerun for cleanup.Original validation environment: Windows 11, Ryzen 7 9700X, RTX 5070 Ti (driver 616.56), Python 3.12.10, Torch 2.14.1+cu130 and Triton Windows 3.8.0.post29. No checkpoint required. Command run at
278267e:AI-assisted; validation was executed through development tooling.