fix: make TransformersRunnable work on CPU-only machines (#4376) - #5664
Open
jissen706 wants to merge 1 commit into
Open
fix: make TransformersRunnable work on CPU-only machines (#4376)#5664jissen706 wants to merge 1 commit into
jissen706 wants to merge 1 commit into
Conversation
…ransformers
Instantiating a runnable for a torch pretrained transformers model crashed
in two ways:
- torch.set_default_tensor_type("torch.cuda.FloatTensor") was called
unconditionally, raising TypeError on machines without CUDA even though
the model had just been placed on CPU (bentoml#4376). It is now only called
when a GPU is assigned, matching the gating used by the pytorch and
detectron runnables.
- default signatures recorded generation methods (generate, greedy_search,
beam_search, ...) that no longer exist on transformers>=4.50 model
classes, so runnable init crashed on getattr for every torch pretrained
model. Default signatures are now filtered to methods the class actually
has, and signature methods missing at load time (e.g. from models saved
with an older transformers) are skipped with a warning instead of
crashing.
Fixes bentoml#4376
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR address?
Fixes #4376.
Instantiating a runnable for a torch pretrained transformers model (
bentoml.transformers.save_model(name, pretrained_model)→bento_model.to_runnable()()) crashed in two independent ways:CUDA default tensor type set on CPU-only machines (the bug reported in bug: TransformersRunnable.__init__ sets torch default tensor type to torch.cuda.FloatTensor even if gpu is not available #4376):
TransformersRunnable.__init__calledtorch.set_default_tensor_type("torch.cuda.FloatTensor")unconditionally for torch-framework pretrained models — even whenCUDA_VISIBLE_DEVICESwas unset/-1and the model had just been placed on CPU. On machines without CUDA this raisesTypeError: type torch.cuda.FloatTensor not available. The call is now gated on the same GPU-assignment condition used for device placement, matching howcommon/pytorch.pyanddetectron.pygate the identical call ontorch.cuda.is_available().Stale generation methods in default signatures:
make_default_signaturesrecordsgenerate,greedy_search,beam_search, etc. for everyPreTrainedModelsubclass, but on transformers >= 4.50 these methods moved toGenerationMixin(and the individual search methods were removed entirely), so runnable init crashed atgetattr(self.model, method_name)for every torch pretrained model, GPU or not. Default signatures are now filtered to methods the class actually provides, and signature methods missing at load time (e.g. models saved with an older transformers version) are skipped with a warning instead of crashing.Testing
Added two regression tests in
tests/integration/frameworks/test_transformers_unit.pyusing a tiny offlineBertForSequenceClassificationbuilt from config (same pattern as the existing custom-pipeline test — no network needed):test_pretrained_runnable_init_without_gpu: before this fix it fails with the exactTypeErrorfrom the issue; now it verifies the model lands on CPU and the default tensor type stays a CPU type.test_pretrained_runnable_skips_unavailable_signature_methods: saves a model with a stalegreedy_searchsignature and verifies runnable init warns and skips it instead of crashing.Verified locally on macOS (CPU-only, torch 2.13, transformers 4.57): both new tests pass,
tests/unitpasses (298 passed, 5 skipped), and the remaining failures intest_transformers_unit.pyare pre-existing onmain(transformers 4.57 API incompatibilities, unrelated to this change).Before submitting:
make formatandmake lintscript have passed (documentation)?🤖 Generated with Claude Code