Summary
PrejoinDefinition stores the two sides of a join condition as two independent m2m fields, so
their pairing is lost. The export builder re-reads each side separately and the stage template zips
them positionally, which means a multi-column condition is only correct by coincidence.
Where
engine/models/prejoin.py
prejoin_condition_source_column = models.ManyToManyField(SourceColumn, related_name="prejoin_conditions_as_source", ...)
prejoin_condition_target_column = models.ManyToManyField(SourceColumn, related_name="prejoin_conditions_as_target", ...)
engine/services/export/builder.py → _get_prejoins_for_source_table
source_columns = [c.source_column_physical_name for c in prejoin.prejoin_condition_source_column.all()]
target_columns = [c.source_column_physical_name for c in prejoin.prejoin_condition_target_column.all()]
join_conditions = PrejoinCondition(source_columns=source_columns, target_columns=target_columns, ...)
Both .all() calls inherit SourceColumn.Meta.ordering = ["source_column_physical_name"], i.e. each
side comes back alphabetically by its own column name.
engine/services/generation/templates/sql/stage.sql.j2 then emits the two lists independently:
this_column_name:
[% for source_column in prejoin.join_conditions.source_columns %]
- '[[ source_column ]]'
[% endfor %]
ref_column_name:
[% for target_column in prejoin.join_conditions.target_columns %]
- '[[ target_column ]]'
[% endfor %]
datavault4dbt.stage pairs this_column_name[i] with ref_column_name[i], so the pairing that
survives generation is "i-th alphabetical base column = i-th alphabetical target column".
Reproduction
Prejoin orders → customer with the intended two-column condition:
base (orders) |
target (customer) |
customer_id |
id |
valid_from |
from_date |
Sorted per side: base → customer_id, valid_from; target → from_date, id.
Generated:
this_column_name:
- 'customer_id'
- 'valid_from'
ref_column_name:
- 'from_date'
- 'id'
→ orders.customer_id = customer.from_date AND orders.valid_from = customer.id. Both pairs are
wrong, and nothing in validation or generation flags it. Single-column conditions are unaffected
(one element per list), which is why this has not surfaced so far.
Suggested fix
Give the condition an explicit, ordered pairing rather than two parallel sets — either
- an ordered through-model, e.g.
PrejoinJoinCondition(prejoin, source_column, target_column, sort_order)
with Meta.ordering = ["sort_order"] and a unique constraint on (prejoin, sort_order); or
- minimally, a
sort_order on both m2m through-tables and explicit .order_by("sort_order") in the
builder — this keeps the model shape but still leaves two lists that can drift in length.
Option 1 also lets PrejoinDefinition.clean()'s "equal counts" rule be replaced by something
structural. The export type PrejoinCondition can keep its two-list shape (that's what the
datavault4dbt macro wants); only the derivation needs to become order-stable.
Impact / context
Turbovault Studio is adding a prejoin editor. Because of the above it deliberately allows exactly
one condition pair and tells the user "One condition per prejoin. Multi-column joins need engine
support for ordered conditions." Lifting that restriction is blocked on this issue.
Related: #187 (reference-table ref_satellites empty when a table has no assignments).
Summary
PrejoinDefinitionstores the two sides of a join condition as two independent m2m fields, sotheir pairing is lost. The export builder re-reads each side separately and the stage template zips
them positionally, which means a multi-column condition is only correct by coincidence.
Where
engine/models/prejoin.pyengine/services/export/builder.py→_get_prejoins_for_source_tableBoth
.all()calls inheritSourceColumn.Meta.ordering = ["source_column_physical_name"], i.e. eachside comes back alphabetically by its own column name.
engine/services/generation/templates/sql/stage.sql.j2then emits the two lists independently:this_column_name: [% for source_column in prejoin.join_conditions.source_columns %] - '[[ source_column ]]' [% endfor %] ref_column_name: [% for target_column in prejoin.join_conditions.target_columns %] - '[[ target_column ]]' [% endfor %]datavault4dbt.stagepairsthis_column_name[i]withref_column_name[i], so the pairing thatsurvives generation is "i-th alphabetical base column = i-th alphabetical target column".
Reproduction
Prejoin
orders → customerwith the intended two-column condition:orders)customer)customer_ididvalid_fromfrom_dateSorted per side: base →
customer_id,valid_from; target →from_date,id.Generated:
→
orders.customer_id = customer.from_date AND orders.valid_from = customer.id. Both pairs arewrong, and nothing in validation or generation flags it. Single-column conditions are unaffected
(one element per list), which is why this has not surfaced so far.
Suggested fix
Give the condition an explicit, ordered pairing rather than two parallel sets — either
PrejoinJoinCondition(prejoin, source_column, target_column, sort_order)with
Meta.ordering = ["sort_order"]and a unique constraint on(prejoin, sort_order); orsort_orderon both m2m through-tables and explicit.order_by("sort_order")in thebuilder — this keeps the model shape but still leaves two lists that can drift in length.
Option 1 also lets
PrejoinDefinition.clean()'s "equal counts" rule be replaced by somethingstructural. The export type
PrejoinConditioncan keep its two-list shape (that's what thedatavault4dbt macro wants); only the derivation needs to become order-stable.
Impact / context
Turbovault Studio is adding a prejoin editor. Because of the above it deliberately allows exactly
one condition pair and tells the user "One condition per prejoin. Multi-column joins need engine
support for ordered conditions." Lifting that restriction is blocked on this issue.
Related: #187 (reference-table
ref_satellitesempty when a table has no assignments).