Remove legacy native HDFS client crates and standardize on OpenDAL HDFS

[peterxcli]

Background

Comet currently has two HDFS implementations:

• native/core default feature hdfs-opendal, which uses OpenDAL's HDFS backend through object_store_opendal
• legacy Comet-maintained crates:
  • native/hdfs
  • native/fs-hdfs

The default native build already uses hdfs-opendal, and the legacy implementation is only selected when the hdfs feature is enabled without hdfs-opendal.

Proposal

Remove the legacy Comet-maintained HDFS client/object_store implementation and standardize HDFS support on OpenDAL.

This should include:

• Remove native/hdfs and native/fs-hdfs from the Rust workspace.
• Remove the legacy hdfs implementation path from native/core/src/parquet/parquet_support.rs.
• Either remove the hdfs feature or keep it as a compatibility alias to hdfs-opendal.
• Update user docs that still recommend COMET_FEATURES=hdfs.
• Keep spark.hadoop.fs.comet.libhdfs.schemes behavior so configured schemes still route through the native HDFS backend.
• Verify HDFS read/write behavior through the existing OpenDAL path.

Motivation

Maintaining a separate libhdfs wrapper and object_store implementation increases Comet's maintenance burden. OpenDAL already provides HDFS support and is the default path in Comet, so keeping the legacy crates creates duplicate code paths and feature confusion.

Notes

OpenDAL HDFS still depends on the Hadoop/JDK/libhdfs runtime environment, so this is not intended to remove those runtime requirements. The goal is to stop maintaining Comet's own HDFS client code.

Acceptance Criteria

• Default HDFS read path continues to work through OpenDAL.
• Native HDFS write path continues to work through OpenDAL.
• Existing fake-HDFS/native scan tests remain valid or are updated to reflect the OpenDAL-only path.
• Documentation no longer points users at the legacy hdfs feature or native/hdfs README.
• cargo build and relevant JVM tests pass with the final feature layout.