Skip to content

fix: prevent pandas_kwargs mutation during text file compression resolution - #3428

Open
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/s3-text-read-write-mixed-compression
Open

fix: prevent pandas_kwargs mutation during text file compression resolution#3428
hsusul wants to merge 1 commit into
aws:mainfrom
hsusul:fix/s3-text-read-write-mixed-compression

Conversation

@hsusul

@hsusul hsusul commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes a bug where _get_read_details and _get_write_details mutated the passed pandas_kwargs dictionary in place when resolving file compression (e.g. setting pandas_kwargs["compression"] = infer_compression(...)).

Symptom & Root Cause

When reading a list of S3 paths or a dataset containing files with different compression extensions (such as ["s3://bucket/file1.csv.gz", "s3://bucket/file2.csv"] or ["s3://bucket/file1.json.gz", "s3://bucket/file2.json"]), _get_read_details mutated the shared pandas_kwargs dictionary during the processing of the first file (e.g., setting "compression": "gzip").

Because pandas_kwargs was mutated in place:

  1. pandas_kwargs.get("compression", "infer") returned "gzip" for subsequent files rather than "infer".
  2. Compression inference was skipped for all remaining files in paths.
  3. Reading mixed compressed/uncompressed files failed with errors such as gzip.BadGzipFile: Not a gzipped file, UnicodeDecodeError, or produced corrupted data.

Solution

  • Updated _get_read_details and _get_write_details to operate on a copy of pandas_kwargs per-file, preserving pandas_kwargs across multiple files so each file infers its own compression setting independently.
  • Added unit regression tests in tests/unit/test_moto.py covering read_csv and read_json with mixed compression paths and chunked reads.

Validation Results

  • pytest tests/unit/test_moto.py: All 47 tests passed.
  • git diff --check: Clean (0 errors).
  • ruff check: All checks passed.
  • ruff format --check: All files formatted.
  • mypy: Success (0 errors).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant