Fix reading gzipped files downloaded over HTTP

[handecelikkanat]

Refs #635

Problem

Gzipped CSV, TSV, JSON and JSON Lines files cannot be read when they are downloaded over HTTP.

• mlcroissant saves HTTP downloads in its cache as croissant-<sha256 of URL>, without a file extension.
• But Read in read.py uses filepath.suffix == ".gz" to detect gzipped files. (Introduced in git lfs download fileObject and read gzipped files #636.)
• So downloaded files cannot be detected as gzipped, and sent directly to pandas:
  • Which fails with: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
• This also affects the latest release (1.1.0).
• Gzipped files read with unArchive (for example in manifest files for Common Crawl) are not affected. They go through extract.py, which already detects gzip by content.

We found this when we added a gzipped CSV file over HTTP to the Common Crawl croissants.

Fix

Use is_gzip from extract.py to detect gzipped files by magic bytes instead, which doesnt depend on the exact file name. (Introduced in #1001).

What it affects

• This patch covers formats read through the gzip wrapper: {CSV, TSV, JSON and JSONL}.
• As for Parquet files:
  • No change for regular Parquet files
  • Gzipped Parquet files (ie., .parquet.gz) did not work, they still dont work after this change. (The wrapper opens files in text mode.)
• Gzipped files named .gz work as before.
• Gzipped files without .gz now work, eg. HTTP downloads, local files without an extension, etc.
• A file named .gz that is not gzipped is now correctly detected as a plain file, so it will not fail.
• Plain files are not affected.

Tests

• New test added that reads two gzipped CSVs, named file.csv.gz and file.
  • First test passes before and after this patch.
  • Second test passes after patch.
• Also tested with a local HTTP server, loading a file.csv.gz URL, a URL without an extension (file), and a plain CSV (file.csv) all worked.

Note

This patch fixes the loader only. #635 introduces a broader spec question regarding how to best declare compression in the metadata, which is still an open question.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[handecelikkanat]

@benjelloun If you have time to look at this in the next two days, it would be very helpful, we want to publish gzipped metadata files in October crawl, to be cited from Croissants in November. Ty!

[benjelloun]

Thanks for the fix!