Follow-up from review of #2527 (requested by @m-abulazm).
#2527 reverted the special-cased BigQuery decimal transform (bigquery_decimal_transform and the
conditional branch in query_builder/base.py::_get_transform) to avoid a second place that
conditionally rewrites types. The row/data hash transforms for BigQuery should instead be handled
properly through DataType_transform_mapping.
Problem
For row/data reconciliation, columns are cast to STRING and hashed. BigQuery CAST(NUMERIC AS STRING)
strips trailing zeros (23.7989) while Spark CAST(DECIMAL(p,s) AS STRING) pads to the column scale
(23.798900000), so identical decimal values hash differently → false mismatches. TIMESTAMP/FLOAT64
string formatting parity needs similar handling.
Proposed
Extend the "bigquery" entry in DataType_transform_mapping
(reconcile/query_builder/expression_generator.py) to emit scale-aware decimal formatting (and
timestamp/number handling) — no conditional special-casing in _get_transform. @m-abulazm offered to
pair on this.
Marker left in code: # TODO: add timestamps and numbers handling.
Follow-up from review of #2527 (requested by @m-abulazm).
#2527 reverted the special-cased BigQuery decimal transform (
bigquery_decimal_transformand theconditional branch in
query_builder/base.py::_get_transform) to avoid a second place thatconditionally rewrites types. The row/data hash transforms for BigQuery should instead be handled
properly through
DataType_transform_mapping.Problem
For row/data reconciliation, columns are cast to STRING and hashed. BigQuery
CAST(NUMERIC AS STRING)strips trailing zeros (
23.7989) while SparkCAST(DECIMAL(p,s) AS STRING)pads to the column scale(
23.798900000), so identical decimal values hash differently → false mismatches. TIMESTAMP/FLOAT64string formatting parity needs similar handling.
Proposed
Extend the
"bigquery"entry inDataType_transform_mapping(
reconcile/query_builder/expression_generator.py) to emit scale-aware decimal formatting (andtimestamp/number handling) — no conditional special-casing in
_get_transform. @m-abulazm offered topair on this.
Marker left in code:
# TODO: add timestamps and numbers handling.