fix: recognize missing snowpark submodule in Snowpark pandas apply() error - #4336
fix: recognize missing snowpark submodule in Snowpark pandas apply() error#4336sfc-gh-kmehta wants to merge 1 commit into
Conversation
Referencing modin.pandas inside apply() makes the generated UDTF handler import a package the sandbox does not have, and we translate that server error into a message telling the user to use native pandas instead. The guard only matched "No module named 'snowflake'", so a sandbox reporting the missing submodule as 'snowflake.snowpark' fell through to the retry branch and surfaced the raw ModuleNotFoundError, naming an internal temp function. Match the prefix so both forms are recognized. Co-authored-by: Cursor <cursoragent@cursor.com>
63f5a89 to
3932cd2
Compare
|
Rebased onto Context: this answers @sfc-gh-yuwang's question on #4335 about the one pandas failure in the merge gate. It is not related to #4335 — the responsible code dates to dc4deee (2026-06-03) — but it is a real bug rather than the environment issue I first assumed. Full reasoning in this comment. |
Verified end-to-end: the fix worksBoth jobs that were failing this test now pass, with zero failures in either:
These are the two accounts whose sandbox reports the missing package as
Required checks: 6 of 8 passing, the other 2 still running, nothing failing. Ready for review — @sfc-gh-yuwang, this closes out your question on #4335. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4336 +/- ##
==========================================
+ Coverage 95.24% 95.47% +0.22%
==========================================
Files 171 171
Lines 44756 44758 +2
Branches 7685 7685
==========================================
+ Hits 42630 42731 +101
+ Misses 1339 1253 -86
+ Partials 787 774 -13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Which Jira issue is this PR addressing?
SNOW-1345395 area (Snowpark pandas
apply()); no dedicated ticket yet.Pre-review checklist:
Please describe how your code solves the related issue.
Problem
Referencing
modin.pandasinside a Snowpark pandasapply()is a documented limitation. We detect it and raise a message pointing the user at native pandas. That detection was a substring match on the server error:The generated UDTF handler imports a package the sandbox does not have, and which name the sandbox reports depends on the image. A sandbox carrying no
snowflakepackage reportssnowflake, matching the guard. One carrying some othersnowflake.*distribution instead reports the missing submodule:The character after
snowflakeis., not', so the guard misses. Control falls to theelsebranch, which re-runscache_result, fails identically, and propagates the rawModuleNotFoundError.So on those accounts a user hitting this limitation gets a
ModuleNotFoundErrornaming an internal temp function, instead of the guidance we wrote for them.This is also why
tests/integ/modin/frame/test_apply.py::test_snowpandas_in_apply_negativefails on some accounts and not others — currentlymodin-ubuntu-latest-64-cores-3.12-awsandmodin-ubuntu-latest-64-cores-3.10-azure. It looked like flakiness or a sandbox packaging problem; it is neither. The sandbox behavior is expected, and the test is correctly reporting that our error translation misses.Fix
Match on the prefix so both forms are recognized, extracted into
_is_missing_snowpark_or_modin_errorso it can be tested directly:modinis included for the same reason — the handler can fail on either import depending on the image.Behavior is otherwise unchanged. Errors that are genuinely something else still take the existing path.
Test plan
test_is_missing_snowpark_or_modin_error/_negativeintests/unit/modin/test_snowflake_query_compiler.py, covering the baresnowflakeform, thesnowflake.snowparkandsnowflake.snowpark.modinsubmodule forms,modin/modin.pandas,Modin is not installed, and negatives including a different missing package (scipy) so the prefix match can't over-trigger.False, new guardTrue.tests/unit/modinpasses locally: 801 passed, 17 skipped, 4 xfailed. (test_internal_frame.pyfails to collect in my local env both with and without this change — a pre-existing local artifact.)test_snowpandas_in_apply_negativeshould go green on the two modin jobs above, which is the real end-to-end confirmation.Note on scope
I deliberately left the
elsebranch alone, but flagging it for a follow-up: on a guard miss it blindly re-runscache_result(udtf_dataframe), so the whole UDTF-plus-cache path executes twice before failing. That doubles the cost of any unrelated failure in this code path.Made with Cursor