Skip to content

fix: recognize missing snowpark submodule in Snowpark pandas apply() error - #4336

Open
sfc-gh-kmehta wants to merge 1 commit into
mainfrom
kmehta-fix-modin-apply-missing-module-guard
Open

fix: recognize missing snowpark submodule in Snowpark pandas apply() error#4336
sfc-gh-kmehta wants to merge 1 commit into
mainfrom
kmehta-fix-modin-apply-missing-module-guard

Conversation

@sfc-gh-kmehta

Copy link
Copy Markdown
Collaborator
  1. Which Jira issue is this PR addressing?

    SNOW-1345395 area (Snowpark pandas apply()); no dedicated ticket yet.

  2. Pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
    • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe.
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support.
  3. Please describe how your code solves the related issue.

Problem

Referencing modin.pandas inside a Snowpark pandas apply() is a documented limitation. We detect it and raise a message pointing the user at native pandas. That detection was a substring match on the server error:

if "No module named 'snowflake'" in str(e) or "Modin is not installed" in str(e):

The generated UDTF handler imports a package the sandbox does not have, and which name the sandbox reports depends on the image. A sandbox carrying no snowflake package reports snowflake, matching the guard. One carrying some other snowflake.* distribution instead reports the missing submodule:

100357 (P0000): Python Interpreter Error:
ModuleNotFoundError: No module named 'snowflake.snowpark' in function SNOWPARK_TEMP_TABLE_FUNCTION_GK4GB6JDMA with handler udf_py_480608373.compute

The character after snowflake is ., not ', so the guard misses. Control falls to the else branch, which re-runs cache_result, fails identically, and propagates the raw ModuleNotFoundError.

So on those accounts a user hitting this limitation gets a ModuleNotFoundError naming an internal temp function, instead of the guidance we wrote for them.

This is also why tests/integ/modin/frame/test_apply.py::test_snowpandas_in_apply_negative fails on some accounts and not others — currently modin-ubuntu-latest-64-cores-3.12-aws and modin-ubuntu-latest-64-cores-3.10-azure. It looked like flakiness or a sandbox packaging problem; it is neither. The sandbox behavior is expected, and the test is correctly reporting that our error translation misses.

Fix

Match on the prefix so both forms are recognized, extracted into _is_missing_snowpark_or_modin_error so it can be tested directly:

return (
    "No module named 'snowflake" in message
    or "No module named 'modin" in message
    or "Modin is not installed" in message
)

modin is included for the same reason — the handler can fail on either import depending on the image.

Behavior is otherwise unchanged. Errors that are genuinely something else still take the existing path.

Test plan

  • New test_is_missing_snowpark_or_modin_error / _negative in tests/unit/modin/test_snowflake_query_compiler.py, covering the bare snowflake form, the snowflake.snowpark and snowflake.snowpark.modin submodule forms, modin / modin.pandas, Modin is not installed, and negatives including a different missing package (scipy) so the prefix match can't over-trigger.
  • Verified the predicate against the verbatim message from the failing CI job: old guard False, new guard True.
  • tests/unit/modin passes locally: 801 passed, 17 skipped, 4 xfailed. (test_internal_frame.py fails to collect in my local env both with and without this change — a pre-existing local artifact.)
  • test_snowpandas_in_apply_negative should go green on the two modin jobs above, which is the real end-to-end confirmation.

Note on scope

I deliberately left the else branch alone, but flagging it for a follow-up: on a guard miss it blindly re-runs cache_result(udtf_dataframe), so the whole UDTF-plus-cache path executes twice before failing. That doubles the cost of any unrelated failure in this code path.

Made with Cursor

Referencing modin.pandas inside apply() makes the generated UDTF handler
import a package the sandbox does not have, and we translate that server
error into a message telling the user to use native pandas instead. The
guard only matched "No module named 'snowflake'", so a sandbox reporting
the missing submodule as 'snowflake.snowpark' fell through to the retry
branch and surfaced the raw ModuleNotFoundError, naming an internal temp
function. Match the prefix so both forms are recognized.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sfc-gh-kmehta

Copy link
Copy Markdown
Collaborator Author

Rebased onto fb4d77b (post-#4335 main) so the required py-windows-latest-64-cores-3.14-gcp check can pass — before the rebase this branch predated both fixes in #4335 and would have failed that gate on bugs unrelated to this change. Only the CHANGELOG.md entry needed resolving; the code change is unchanged.

Context: this answers @sfc-gh-yuwang's question on #4335 about the one pandas failure in the merge gate. It is not related to #4335 — the responsible code dates to dc4deee (2026-06-03) — but it is a real bug rather than the environment issue I first assumed. Full reasoning in this comment.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed

This PR was classified as LOW risk by the automated pre-screen.

@sfc-gh-kmehta

Copy link
Copy Markdown
Collaborator Author

Verified end-to-end: the fix works

Both jobs that were failing this test now pass, with zero failures in either:

job test_snowpandas_in_apply_negative
Test modin-ubuntu-latest-64-cores-3.12-aws pass
Test modin-ubuntu-latest-64-cores-3.10-azure pass

These are the two accounts whose sandbox reports the missing package as snowflake.snowpark rather than snowflake, so they are exactly the case the old guard missed. The friendly "only native pandas inside apply()" message is now raised as intended.

Test py-windows-latest-64-cores-3.14-gcp is also green after the rebase onto fb4d77b.

Required checks: 6 of 8 passing, the other 2 still running, nothing failing. Ready for review — @sfc-gh-yuwang, this closes out your question on #4335.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.47%. Comparing base (fb4d77b) to head (3932cd2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4336      +/-   ##
==========================================
+ Coverage   95.24%   95.47%   +0.22%     
==========================================
  Files         171      171              
  Lines       44756    44758       +2     
  Branches     7685     7685              
==========================================
+ Hits        42630    42731     +101     
+ Misses       1339     1253      -86     
+ Partials      787      774      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sfc-gh-jzeng sfc-gh-jzeng added the NO-PANDAS-CHANGEDOC-UPDATES This PR does not update Snowpark pandas docs label Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

NO-PANDAS-CHANGEDOC-UPDATES This PR does not update Snowpark pandas docs snowpark-pandas

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants