Skip to content

fix(engine): seed_for hashes a long prompt in slices, so it stops building one string - #51

Open
imD-5 wants to merge 1 commit into
Elevatormusic:deepseek-v41-tp2from
imD-5:fix/seed-chunked-hash
Open

fix(engine): seed_for hashes a long prompt in slices, so it stops building one string#51
imD-5 wants to merge 1 commit into
Elevatormusic:deepseek-v41-tp2from
imD-5:fix/seed-chunked-hash

Conversation

@imD-5

@imD-5 imD-5 commented Oct 28, 2026

Copy link
Copy Markdown

Effect

  • seed_for on a 94K-token prompt stops building a ~700 KB string and ~94K small strings: about 12 MiB allocated per request becomes ~30 KB a slice.
  • The seed is unchanged, so sampled replies are unchanged, and both ranks still agree.

Reproduce

python3 -m pytest -q tests/test_seed_for.py     # the sliced digest equals the one-join form at 0, 1, 4095, 4096, 4097, 8192, 8193 and 50k tokens
  • The reference in the test is the previous implementation, so a boundary that swallows a comma fails at 4096.

Change

  • exact_sampling.seed_for: hashlib.sha256() updated once per 4096-token slice, then the salt; the joined form is gone.

Why

  • A hash does not care how the bytes arrive, and the only byte the slicing adds is the comma a chunk boundary needs; with it the stream is byte-identical, so the digest, the seed and every sampled reply are too.

Type

  • type: fix: a bug fix, or a fix after review findings

Tests

  • New tests/test_seed_for.py: 12 cases, all passing; test_seed_salt.py and test_exact_sampling.py unchanged (the one failure there is in known-test-failures.txt).

Checklist

  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.
AI disclosure

AI: Claude Opus 5.5 via Hermes Agent
Checked: by author

@imD-5
fix(engine): seed_for hashes a long prompt in slices, so it stops bui…
b2981d6 
…lding one string

The prompt key hashed a comma-joined string of every token id: on a 94K-token prompt that is ~700 KB of text plus
94K small strings, about 12 MiB allocated per request, all thrown away once the digest exists. Feed the same bytes to
sha256 in 4096-token slices instead, with the comma the boundary needs, so the digest and the seed are unchanged.
@imD-5
imD-5 requested a review from Elevatormusic as a code owner October 28, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Elevatormusic Elevatormusic Awaiting requested review from Elevatormusic Elevatormusic is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@imD-5