[morristai]

s3:// Parquet becomes a first-class native source, and Iceberg tables from REST and AWS Glue catalogs read through it. Two commits: the implementation (811084980) and fixes from a follow-up review (68ef7e3fb).

Remote Parquet as a native source

• Remote footers: a physical optimizer rule cannot wait on the network, so planning reads s3:// footers only from the session's DataFusion file metadata cache. The planning invocation carries that cache (AlgeonAttempt, Flight SQL statements, plan_with_native_diagnostics, validate_query). Multi-object and byte-range remote scans now lower natively.
• Schema contract: footers must agree on column names, order and types. A column may be required in one file and optional in another when the table declares it nullable; other differences are rejected with native_source_file_schema_mismatch.
• Rejections name their cause: source rejections carry a source blocking subject, so native_required errors expose x-algeon-reason (for example native_source_remote_footer_unavailable). Diagnostics of rejected remote scans report their real read strategy instead of local_kvikio.
• Chunk planning: local_parquet_throughput is renamed parquet_throughput (no alias) and also widens chunks of remote files that carry footer byte metadata. TPC-DS SF100 store_sales (one 17.2 GB file, same-region S3, g7e.8xlarge): SUM(ss_ext_sales_price) 1.1 s; before, the scan was rejected, or took 270 s with file-scan repartitioning off.
• Server: the [s3] table returns (endpoint, region, path_style, unsigned, default credential chain), and the s3-parquet-e2e tool covers one object, a multi-object prefix and a byte-range split.

Read-only Iceberg

• New unpublished crate algeon-iceberg, on iceberg-rust pinned to 5b82ab99 (the revision apache/datafusion-iceberg pins). It does not depend on iceberg-datafusion.
  • REST (no SigV4) and Glue catalog providers.
  • A table provider that plans the current snapshot with manifest pruning into a standard Parquet DataSourceExec, and fills the footer cache for native lowering.
• Rejections: snapshots the Parquet path would read incorrectly fail with the new unsupported_table_feature code and a reason fact, in every execution mode, with no CPU fallback. This covers delete files, field-id or name-mapping mismatches, absent required or identity-partition columns, encrypted or non-Parquet files, several buckets, other location schemes, FileIO storage properties that disagree with [s3], and promoted column types.
• Server: [[iceberg.catalogs]] config, an algeon-server iceberg feature shipped in the CPU image, and the iceberg-local-e2e tool (RustFS, iceberg-rest-fixture, moto as Glue).
• TLS: the AWS SDK adds rustls's aws-lc-rs backend next to ring, and rustls then selects no default provider. The server now installs ring as the process provider before tonic builds its TLS acceptor.

Review fixes (68ef7e3fb)

• Row-group pruning read statistics by DataFusion file-schema position. When a table orders columns differently from its files, this read another column's statistics and dropped matching row groups. Pruning now resolves columns against the footer schema. The bug already existed for local tables with a declared schema.
• Type promotion: Iceberg files that store a column with a type other than the one the table declares fail with iceberg_column_type_mismatch.

Validation

• Focused nextest sets pass: native lowering, source and footers; server config; query-planning; query-engine chunk planning; algeon-iceberg (CPU and GPU); server_tests_gpu; bench config tests.
• Workspace clippy (--all-features -D warnings), nightly fmt, the public API ledger check, the workspace publication check, the prose check and the website build pass. The full pre-merge gate was not run.
• AWS (same region):
  • s3-parquet-e2e passes for library native_required and functions_only sessions and a server [s3] session.
  • Iceberg tables registered in Glue over 62 Parquet files list over Flight SQL and query natively, with results equal to a ListingTable over the same files.
  • A year filter prunes 62 files to 12; TPC-DS Q3 takes 6.5 s.
• Local: iceberg-local-e2e passes for REST and Glue (moto), including the delete-file rejection.

Not in this PR:

• Many-file remote layouts are latency-bound: a chunk never spans files and a join leaf prefetches one chunk, so Q3 over 62 files takes 6.5 s against 0.15 s locally.
• The footer cache limit has no server setting: Footer cache limit caps the size of remote and Iceberg tables the server can plan natively #10.
• Not covered: HTTPS REST endpoints, SigV4 REST catalogs, delete files, field-id binding, vended credentials.

🤖 Generated with Claude Code