fix(data): skip zero-message conversations before apply_chat_template - #777
Open
zhentaocc wants to merge 1 commit into
Open
fix(data): skip zero-message conversations before apply_chat_template#777zhentaocc wants to merge 1 commit into
zhentaocc wants to merge 1 commit into
Conversation
GeneralParser.parse drops leading non-user turns, so assistant-first rows with no system prompt can produce an empty messages list. On transformers 5.x, tokenizer.apply_chat_template indexes conversation[0] unconditionally and raises IndexError, which is not caught by the existing ValueError/TypeError fallback and aborts the whole dataset.map preprocessing job (observed in build_eagle3_dataset / hidden-states prep on EAGLE3 data with assistant-first conversations). Return an empty (input_ids, loss_mask) sample when messages is empty so the downstream trainable-token / loss-mask eligibility filters drop it, matching the existing None-source skip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GeneralParser.parsedrops leading non-user turns while assembling the messagelist. For assistant-first rows with no system prompt, this can leave
messagesempty. On transformers 5.x,
tokenizer.apply_chat_templateindexesconversation[0]unconditionally and raisesIndexError.That
IndexErroris not caught by the existingexcept (ValueError, TypeError)fallback around
apply_chat_template, so a single bad row aborts the entiredataset.mappreprocessing job (observed duringbuild_eagle3_dataset/hidden-states prep on EAGLE3 data containing assistant-first conversations).
Fix
Before calling
apply_chat_template, detect the empty-messagescase and returnan empty
(input_ids, loss_mask)sample. The downstream trainable-token /loss-mask eligibility filters (
minimum_valid_tokens,loss_mask_filterinpreprocessing.py) already drop zero-length samples, matching the existingNone-source skip path. A warning is emitted so these rows are visible.
Validation
Reproduced the original
IndexErrortraceback during dataset preprocessing onassistant-first EAGLE3 data; with this guard the offending rows are skipped and
preprocessing completes.