fix: WIP collapse Arrow record batches above configurable cap

[igorDykhta]

Summary

• Collapse Arrow/Parquet record batches into one when a table exceeds maxArrowBatches (default 255).
• Threshold is set from application config: initApplicationConfig({ maxArrowBatches }).

Fewer batches avoid several problems that show up on large parquet/DuckDB tables:

• Picking: deck.gl encodes at most 255 pickable leaf layers. Kepler creates one Deck layer per Arrow record batch (GeoJSON adds fill/stroke/point sublayers), so 255+ batches block hover/click on most of the data.
• Polygon layer: polygon filters and GeoJSON rendering are unreliable when geometry is split across many batches.
• Performance: many small batches produce many Deck layers and extra CPU/GPU overhead.

JS loaders do not expose a way to control the record-batch size they produce, so collapsing after load is the option to alleviate these issues.

Memory: compaction copies the table, so peak memory is higher during processing. If that is a problem for a given dataset, raise maxArrowBatches (a very large value skips collapse):

initApplicationConfig({ maxArrowBatches: Number.MAX_SAFE_INTEGER });

Test plan

• Load a parquet/Arrow file with more than 255 record batches and confirm hover/click work (no “Too many pickable layers” warning).
• Confirm tables at or below the cap are left unchanged.
• Confirm maxArrowBatches: 0 always collapses, and a very large value never collapses.
• Doesn't break anything

[Copilot]

Pull request overview

This PR addresses deck.gl picking limitations and GeoArrow coordinate robustness by collapsing excessive Arrow record batches into fewer chunks and by making polygon filtering tolerant of non-finite altitude values (z) while still validating lng/lat.

Changes:

• Added compactArrowTable and integrated it into Arrow ingestion paths (processors, DuckDB table, ArrowDataContainer) to reduce record-batch explosion that can break picking.
• Updated polygon filtering to only require finite lng/lat (ignoring non-finite altitude) and added tests for that behavior.
• Improved GeoArrow point coordinate extraction and reduced GeoArrow-derived layer proliferation by building single-batch vectors / binary collections.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/node/utils/filter-utils-test.js	Adds regression coverage for polygon filtering with null/undefined altitude.
test/node/utils/data-container-test.js	Adds coverage for Arrow batch compaction and ArrowDataContainer chunk behavior.
src/utils/src/index.ts	Exposes compactArrowTable from @kepler.gl/utils.
src/utils/src/filter-utils.ts	Relaxes polygon filtering to only require finite lng/lat; refactors coordinate checks.
src/utils/src/arrow-data-container.ts	Implements compactArrowTable and applies it in ArrowDataContainer construction/update.
src/utils/src/application-config.ts	Introduces maxArrowBatches application config and default value.
src/processors/src/data-processor.ts	Compacts Arrow tables during batch processing.
src/layers/src/point-layer/point-layer.ts	Uses shared GeoArrow point coordinate reader for robustness.
src/layers/src/layer-utils.ts	Adds getGeoArrowPointCoords; collapses WKB conversions and point vector batching.
src/layers/src/heatmap-layer/heatmap-layer.ts	Uses shared GeoArrow point coordinate reader for robustness.
src/layers/src/geojson-layer/geojson-layer.ts	Adjusts incremental update handling to account for compacted tables / row-growth.
src/duckdb/src/table/duckdb-table.ts	Compacts DuckDB Arrow results before deriving fields/columns.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.