Generation crashes with encoder_repetition_penalty when encoder_outputs are provided without input_ids #49695

System Info

• transformers version: 5.19.0.dev0
• Platform: Windows-11-10.0.26300-SP0
• Python version: 3.13.1
• Huggingface_hub version: 1.33.0
• Safetensors version: 0.8.0
• PyTorch version (accelerator?): 2.14.0+cpu (NA)
• Accelerate version: not installed
• DeepSpeed version: not installed
• Using distributed or parallel set-up in script?: No

Who can help?

No response

Reproduction

import torch
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained(
    "hf-internal-testing/tiny-random-t5"
)
model.eval()

encoder_input_ids = torch.tensor([[3, 4, 5]])

with torch.no_grad():
    encoder_outputs = model.get_encoder()(
        input_ids=encoder_input_ids,
        return_dict=True,
    )

# Reported to crash when encoder token IDs are unavailable.
model.generate(
    encoder_outputs=encoder_outputs,
    encoder_repetition_penalty=1.5,
    max_new_tokens=5,
)

Reported exception:

RuntimeError: index -100 is out of bounds for dimension 1 with size 1103

A related test using encoder_no_repeat_ngram_size=2 reportedly completes without the expected warning, and the intended encoder-side constraint appears not to be applied.

Providing the original input_ids alongside encoder_outputs allowed both configurations to complete in the investigated tests.

Expected behavior

When precomputed encoder_outputs are provided without the original encoder token IDs, generation should not pass sentinel values such as -100 to encoder-side logits processors.

The generation process should handle unavailable encoder token IDs gracefully, following the existing warning-and-ignore behavior where appropriate, rather than crashing or silently applying an ineffective constraint.