Skip to content

[BUG] Multi-column prejoin conditions are paired by column name, not by definition order #188

Description

@obause

Summary

PrejoinDefinition stores the two sides of a join condition as two independent m2m fields, so
their pairing is lost. The export builder re-reads each side separately and the stage template zips
them positionally, which means a multi-column condition is only correct by coincidence.

Where

engine/models/prejoin.py

prejoin_condition_source_column = models.ManyToManyField(SourceColumn, related_name="prejoin_conditions_as_source", ...)
prejoin_condition_target_column = models.ManyToManyField(SourceColumn, related_name="prejoin_conditions_as_target", ...)

engine/services/export/builder.py_get_prejoins_for_source_table

source_columns = [c.source_column_physical_name for c in prejoin.prejoin_condition_source_column.all()]
target_columns = [c.source_column_physical_name for c in prejoin.prejoin_condition_target_column.all()]
join_conditions = PrejoinCondition(source_columns=source_columns, target_columns=target_columns, ...)

Both .all() calls inherit SourceColumn.Meta.ordering = ["source_column_physical_name"], i.e. each
side comes back alphabetically by its own column name.

engine/services/generation/templates/sql/stage.sql.j2 then emits the two lists independently:

this_column_name:
  [% for source_column in prejoin.join_conditions.source_columns %]
    - '[[ source_column ]]'
  [% endfor %]
ref_column_name:
  [% for target_column in prejoin.join_conditions.target_columns %]
    - '[[ target_column ]]'
  [% endfor %]

datavault4dbt.stage pairs this_column_name[i] with ref_column_name[i], so the pairing that
survives generation is "i-th alphabetical base column = i-th alphabetical target column".

Reproduction

Prejoin orders → customer with the intended two-column condition:

base (orders) target (customer)
customer_id id
valid_from from_date

Sorted per side: base → customer_id, valid_from; target → from_date, id.

Generated:

this_column_name:
  - 'customer_id'
  - 'valid_from'
ref_column_name:
  - 'from_date'
  - 'id'

orders.customer_id = customer.from_date AND orders.valid_from = customer.id. Both pairs are
wrong, and nothing in validation or generation flags it. Single-column conditions are unaffected
(one element per list), which is why this has not surfaced so far.

Suggested fix

Give the condition an explicit, ordered pairing rather than two parallel sets — either

  1. an ordered through-model, e.g. PrejoinJoinCondition(prejoin, source_column, target_column, sort_order)
    with Meta.ordering = ["sort_order"] and a unique constraint on (prejoin, sort_order); or
  2. minimally, a sort_order on both m2m through-tables and explicit .order_by("sort_order") in the
    builder — this keeps the model shape but still leaves two lists that can drift in length.

Option 1 also lets PrejoinDefinition.clean()'s "equal counts" rule be replaced by something
structural. The export type PrejoinCondition can keep its two-list shape (that's what the
datavault4dbt macro wants); only the derivation needs to become order-stable.

Impact / context

Turbovault Studio is adding a prejoin editor. Because of the above it deliberately allows exactly
one condition pair
and tells the user "One condition per prejoin. Multi-column joins need engine
support for ordered conditions." Lifting that restriction is blocked on this issue.

Related: #187 (reference-table ref_satellites empty when a table has no assignments).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions