Skip to content

fix: strip path root correctly in partition extraction - #3418

Open
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/partition-extraction-root-path
Open

fix: strip path root correctly in partition extraction#3418
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/partition-extraction-root-path

Conversation

@hsusul

@hsusul hsusul commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Feature or Bugfix

  • Bugfix

Detail

  • Fix _extract_partitions_from_path in awswrangler/_arrow.py where f"{path_root}/" was evaluated on a path_root variable that was already normalized to end with a trailing slash (/).
  • Previously, f"{path_root}/" resulted in a double trailing slash (//), causing path_wo_filename.replace(f"{path_root}/", "") to fail silently and return path_wo_filename unstripped.
  • Consequently, if path_root contained any directory or bucket segment with an = (e.g. s3://bucket/env=dev/table/), _extract_partitions_from_path incorrectly parsed env=dev as a dataset partition column ({'env': 'dev', 'year': '2023'} instead of {'year': '2023'}).
  • Replaced .replace(...) with removeprefix(...) across partition and copy path resolution functions (_arrow.py, s3/_read.py, s3/_copy.py) to safely strip prefixes without side effects.
  • Added comprehensive unit tests in test_utils.py and a moto test in test_moto.py.

Relates

  • N/A

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant