Skip to content

fix: make TransformersRunnable work on CPU-only machines (#4376) - #5664

Open
jissen706 wants to merge 1 commit into
bentoml:mainfrom
jissen706:fix/transformers-runnable-cpu-default-tensor-type
Open

fix: make TransformersRunnable work on CPU-only machines (#4376)#5664
jissen706 wants to merge 1 commit into
bentoml:mainfrom
jissen706:fix/transformers-runnable-cpu-default-tensor-type

Conversation

@jissen706

Copy link
Copy Markdown

What does this PR address?

Fixes #4376.

Instantiating a runnable for a torch pretrained transformers model (bentoml.transformers.save_model(name, pretrained_model)bento_model.to_runnable()()) crashed in two independent ways:

  1. CUDA default tensor type set on CPU-only machines (the bug reported in bug: TransformersRunnable.__init__ sets torch default tensor type to torch.cuda.FloatTensor even if gpu is not available #4376): TransformersRunnable.__init__ called torch.set_default_tensor_type("torch.cuda.FloatTensor") unconditionally for torch-framework pretrained models — even when CUDA_VISIBLE_DEVICES was unset/-1 and the model had just been placed on CPU. On machines without CUDA this raises TypeError: type torch.cuda.FloatTensor not available. The call is now gated on the same GPU-assignment condition used for device placement, matching how common/pytorch.py and detectron.py gate the identical call on torch.cuda.is_available().

  2. Stale generation methods in default signatures: make_default_signatures records generate, greedy_search, beam_search, etc. for every PreTrainedModel subclass, but on transformers >= 4.50 these methods moved to GenerationMixin (and the individual search methods were removed entirely), so runnable init crashed at getattr(self.model, method_name) for every torch pretrained model, GPU or not. Default signatures are now filtered to methods the class actually provides, and signature methods missing at load time (e.g. models saved with an older transformers version) are skipped with a warning instead of crashing.

Testing

Added two regression tests in tests/integration/frameworks/test_transformers_unit.py using a tiny offline BertForSequenceClassification built from config (same pattern as the existing custom-pipeline test — no network needed):

  • test_pretrained_runnable_init_without_gpu: before this fix it fails with the exact TypeError from the issue; now it verifies the model lands on CPU and the default tensor type stays a CPU type.
  • test_pretrained_runnable_skips_unavailable_signature_methods: saves a model with a stale greedy_search signature and verifies runnable init warns and skips it instead of crashing.

Verified locally on macOS (CPU-only, torch 2.13, transformers 4.57): both new tests pass, tests/unit passes (298 passed, 5 skipped), and the remaining failures in test_transformers_unit.py are pre-existing on main (transformers 4.57 API incompatibilities, unrelated to this change).

Before submitting:

  • Does the Pull Request follow Conventional Commits specification naming?
  • Does the code follow BentoML's code style, both make format and make lint script have passed (documentation)?
  • Did your changes require updates to the documentation? Not needed — behavior fix only.
  • Did you write tests to cover your changes?

🤖 Generated with Claude Code

…ransformers

Instantiating a runnable for a torch pretrained transformers model crashed
in two ways:

- torch.set_default_tensor_type("torch.cuda.FloatTensor") was called
  unconditionally, raising TypeError on machines without CUDA even though
  the model had just been placed on CPU (bentoml#4376). It is now only called
  when a GPU is assigned, matching the gating used by the pytorch and
  detectron runnables.
- default signatures recorded generation methods (generate, greedy_search,
  beam_search, ...) that no longer exist on transformers>=4.50 model
  classes, so runnable init crashed on getattr for every torch pretrained
  model. Default signatures are now filtered to methods the class actually
  has, and signature methods missing at load time (e.g. from models saved
  with an older transformers) are skipped with a warning instead of
  crashing.

Fixes bentoml#4376
@jissen706
jissen706 requested a review from a team as a code owner July 19, 2026 19:59
@jissen706
jissen706 requested review from parano and removed request for a team July 19, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: TransformersRunnable.__init__ sets torch default tensor type to torch.cuda.FloatTensor even if gpu is not available

1 participant