Skip to content

BigQuery reconcile: handle decimal/timestamp hash parity via DataType_transform_mapping #2537

Description

@take60

Follow-up from review of #2527 (requested by @m-abulazm).

#2527 reverted the special-cased BigQuery decimal transform (bigquery_decimal_transform and the
conditional branch in query_builder/base.py::_get_transform) to avoid a second place that
conditionally rewrites types. The row/data hash transforms for BigQuery should instead be handled
properly through DataType_transform_mapping.

Problem

For row/data reconciliation, columns are cast to STRING and hashed. BigQuery CAST(NUMERIC AS STRING)
strips trailing zeros (23.7989) while Spark CAST(DECIMAL(p,s) AS STRING) pads to the column scale
(23.798900000), so identical decimal values hash differently → false mismatches. TIMESTAMP/FLOAT64
string formatting parity needs similar handling.

Proposed

Extend the "bigquery" entry in DataType_transform_mapping
(reconcile/query_builder/expression_generator.py) to emit scale-aware decimal formatting (and
timestamp/number handling) — no conditional special-casing in _get_transform. @m-abulazm offered to
pair on this.

Marker left in code: # TODO: add timestamps and numbers handling.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions