diff --git a/macros/hooks/clean_up_pit.md b/macros/hooks/clean_up_pit.md new file mode 100644 index 00000000..8d7069ca --- /dev/null +++ b/macros/hooks/clean_up_pit.md @@ -0,0 +1,33 @@ +{% docs clean_up_pit %} + +## Clean Up PIT (Post-Hook) + +This macro should be used as a post-hook for each PIT table whenever a logarithmic snapshot logic is used. +The macro deletes all records in a PIT table that are no longer active according to the snapshot view. +Deletion is safe here because no actual data is deleted — only pointers to satellite entries. + +### Usage as a post-hook + +```jinja +{% raw %} +{{ config( + post_hook="{{ datavault4dbt.clean_up_pit('control_snap_v1') }}" +) }} +{% endraw %} +``` + +### With custom column names + +```jinja +{% raw %} +{{ config( + post_hook="{{ datavault4dbt.clean_up_pit( + snapshot_relation='control_snap_v1', + snapshot_trigger_column='is_active', + sdts='sdts' + ) }}" +) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/hooks/clean_up_pit.yml b/macros/hooks/clean_up_pit.yml new file mode 100644 index 00000000..713a0ef3 --- /dev/null +++ b/macros/hooks/clean_up_pit.yml @@ -0,0 +1,21 @@ +version: 2 + +macros: + - name: clean_up_pit + description: '{{ doc("clean_up_pit") }}' + arguments: + - name: snapshot_relation + type: string + description: > + The name of the dbt model that creates the snapshot table / view that has the logarithmic snapshot + logic applied. + - name: snapshot_trigger_column + type: string + description: > + The name of the boolean column inside the snapshot tables that activates/deactivates single snapshots. + If not set, the name defined inside the global variable 'datavault4dbt.snapshot_trigger_column' is used. + - name: sdts + type: string + description: > + The name of the snapshot date timestamp column inside the snapshot table. If not set, the name defined + inside the global variable 'datavault4dbt.sdts_alias' is used. diff --git a/macros/internal/helpers/helpers.yml b/macros/internal/helpers/helpers.yml new file mode 100644 index 00000000..e34a4e7e --- /dev/null +++ b/macros/internal/helpers/helpers.yml @@ -0,0 +1,82 @@ +version: 2 + +macros: + - name: is_list + description: > + Internal helper. Returns true if the given object is a list (iterable, not a string, + not a mapping). Used throughout the package to distinguish list inputs from scalars. + arguments: + - name: obj + type: any + description: The object to check. + - name: empty_is_false + type: bool + description: If true, returns false when obj is None/undefined (empty lists still return true). Default is false. + + - name: is_nothing + description: > + Internal helper. Returns true if the given object is None, undefined, or falsy. + arguments: + - name: obj + type: any + description: The object to check. + + - name: is_something + description: > + Internal helper. Returns true if the given object is not None, is defined, and is truthy. + Inverse of is_nothing. + arguments: + - name: obj + type: any + description: The object to check. + + - name: is_expression + description: > + Internal helper. Returns true if the given string looks like a SQL expression — i.e. it is + single-quoted, contains parentheses, uses :: (cast syntax) or || (concatenation), or is a + recognized SQL keyword on Trino. + arguments: + - name: obj + type: any + description: The object to check. + + - name: is_attribute + description: > + Internal helper. Returns true if the given string is a plain column name reference — + i.e. not a static string literal (no leading '!') and not a SQL expression. + Raises a compiler error if obj is null or not a string. + arguments: + - name: obj + type: any + description: The object to check. Must be a non-null string. + + - name: is_something_or_false + description: > + Internal helper. Returns true if the given object is not None and is defined, + even if it evaluates to false (e.g. 0 or empty string). + arguments: + - name: obj + type: any + description: The object to check. + + - name: prepend_generated_by + description: > + Internal helper. Outputs an ASCII art banner identifying SQL as generated by datavault4dbt + by Scalefree International GmbH. Prepended as a comment block in generated SQL files. + + - name: replace_standard + description: > + Internal helper. Returns the input_variable if it is not None and not empty. Otherwise falls + back to the value of the given dbt project variable (global_variable), using default_value + if the variable is not set. + arguments: + - name: input_variable + type: any + description: The value passed in by the caller. If None or empty string, the fallback is used. + - name: global_variable + type: string + description: > + The name of the dbt project variable to use as fallback (resolved via var()). + - name: default_value + type: any + description: The value returned if both input_variable and global_variable are unset. diff --git a/macros/internal/metadata_processing/metadata_processing.yml b/macros/internal/metadata_processing/metadata_processing.yml index 3e2a46d2..80a47a7f 100644 --- a/macros/internal/metadata_processing/metadata_processing.yml +++ b/macros/internal/metadata_processing/metadata_processing.yml @@ -14,8 +14,185 @@ macros: type: string description: The forwarded parameter of the top-level macro. This is used, if the yaml-metadata is none. - name: required - type: boolean + type: bool description: Whether this parameter is required for the top-level macro. Default is False. - name: documentation type: string - description: A string that holds documentation of this parameter. + description: A string that holds documentation of this parameter. + + - name: alias + description: > + Internal helper. Renders a single column expression as either 'source_column AS alias' or + a plain column reference with an optional table prefix. Dispatches to default__alias. + arguments: + - name: alias_config + type: any + description: > + Either a dict with 'source_column' and 'alias' keys (produces 'source_column AS alias'), + or a plain column name string (produces the column with optional prefix). + - name: prefix + type: string + description: Optional table prefix prepended to the column reference (e.g. 'src'). + + - name: default__alias + description: Default implementation of alias. Accepts the same arguments as the parent alias macro. + + - name: alias_all + description: > + Internal helper. Iterates over a list of column definitions and calls alias() on each one, + returning a comma-separated list of column expressions. Also accepts a single column definition + directly, in which case it is forwarded to alias() as-is. Dispatches to default__alias_all. + arguments: + - name: columns + type: any + description: > + A list of column definitions, or a single column definition. Each definition is either a plain + column name string or a dict with 'source_column' and 'alias' keys, passed individually to alias(). + - name: prefix + type: string + description: Optional table prefix forwarded to each alias() call. + + - name: default__alias_all + description: Default implementation of alias_all. Accepts the same arguments as the parent alias_all macro. + + - name: as_constant + description: > + Internal helper. Renders a column reference as a SQL literal or expression. + A leading '!' prefix strips the '!' and returns the value as a SQL string literal. + A SQL expression (quoted, contains parentheses, cast or concatenation operators) is passed + through unchanged. A plain identifier is wrapped in the adapter-appropriate escape characters. + Dispatches to default__as_constant. + arguments: + - name: column_str + type: string + description: The column string to evaluate. Leading '!' indicates a static literal. + + - name: default__as_constant + description: Default implementation of as_constant. Accepts the same arguments as the parent as_constant macro. + + - name: check_required_parameters + description: > + Internal helper. Inspects the kwargs dict of a calling macro and raises a dbt compiler + error listing every parameter whose value is None. Used in front-end macros to enforce + that all required arguments have been supplied before code generation begins. + + - name: concat_ws + description: > + Internal helper. Generates a SQL concatenation expression for a list of column strings + using an adapter-appropriate syntax. The default implementation uses the SQL CONCAT + function with the separator interleaved between arguments. Adapter overrides produce + platform-specific syntax (|| operator, CAST to VARCHAR, etc.). + arguments: + - name: string_list + type: list[string] + description: List of column names or expressions to concatenate. + - name: separator + type: string + description: The string placed between each element. Default is '||' (Redshift/Oracle default to '|'). + + - name: default__concat_ws + description: Default implementation of concat_ws using CONCAT with interleaved separator. Accepts the same arguments as the parent concat_ws macro. + + - name: exasol__concat_ws + description: Exasol-specific implementation of concat_ws. Accepts the same arguments as the parent concat_ws macro. + + - name: redshift__concat_ws + description: Amazon Redshift-specific implementation of concat_ws using || operator. Accepts the same arguments as the parent concat_ws macro. + + - name: oracle__concat_ws + description: Oracle-specific implementation of concat_ws using || operator. Accepts the same arguments as the parent concat_ws macro. + + - name: trino__concat_ws + description: Trino-specific implementation of concat_ws with CAST to VARCHAR. Accepts the same arguments as the parent concat_ws macro. + + - name: escape_column_names + description: > + Internal helper. Applies adapter-specific column name escaping to a single column string, + a list of column strings, or a dict with 'source_column' and 'alias' keys. Returns the + same structure as the input with each column name escaped. + arguments: + - name: columns + type: any + description: > + A column name string, a list of column name strings, or a dict with + 'source_column' and 'alias' keys. + + - name: escape_column_name + description: > + Internal helper. Escapes a single column name using the adapter-appropriate escape characters. + The default implementation uppercases the name and wraps it in double quotes. + Dispatches per adapter. + arguments: + - name: column + type: string + description: The raw column name to escape. + + - name: default__escape_column_name + description: Default implementation of escape_column_name. Uppercases the name and wraps it in double quotes. Accepts the same arguments as the parent escape_column_name macro. + + - name: synapse__escape_column_name + description: Azure Synapse Analytics-specific implementation of escape_column_name. Accepts the same arguments as the parent escape_column_name macro. + + - name: bigquery__escape_column_name + description: BigQuery-specific implementation of escape_column_name using backtick escape characters. Accepts the same arguments as the parent escape_column_name macro. + + - name: postgres__escape_column_name + description: PostgreSQL-specific implementation of escape_column_name. Lowercases the name. Accepts the same arguments as the parent escape_column_name macro. + + - name: redshift__escape_column_name + description: Amazon Redshift-specific implementation of escape_column_name. Lowercases the name and wraps it in double quotes. Accepts the same arguments as the parent escape_column_name macro. + + - name: exasol__escape_column_name + description: Exasol-specific implementation of escape_column_name. Accepts the same arguments as the parent escape_column_name macro. + + - name: fabric__escape_column_name + description: Microsoft Fabric-specific implementation of escape_column_name. Respects the datavault4dbt.set_casing variable. Accepts the same arguments as the parent escape_column_name macro. + + - name: databricks__escape_column_name + description: Databricks-specific implementation of escape_column_name. Respects the datavault4dbt.set_casing variable. Accepts the same arguments as the parent escape_column_name macro. + + - name: sqlserver__escape_column_name + description: SQL Server-specific implementation of escape_column_name. Accepts the same arguments as the parent escape_column_name macro. + + - name: trino__escape_column_name + description: Trino-specific implementation of escape_column_name. Accepts the same arguments as the parent escape_column_name macro. + + - name: expand_column_list + description: > + Internal helper. Flattens a list that may contain nested sub-lists and filters out None + values, returning a single flat list of column names. Used to normalize column list inputs + before further processing. + arguments: + - name: columns + type: list[any] + description: A list of column names, possibly containing nested lists or None values. + + - name: multikey + description: > + Internal helper. Generates a multi-column SQL condition string for JOIN ON clauses or + WHERE filters. Pairs each column from 'columns' with the corresponding column from + 'right_columns' (if provided) or with itself, applying an optional table prefix and + comparison operator. Dispatches to default__multikey. + arguments: + - name: columns + type: list[string] + description: List of column names on the left side of the condition. + - name: prefix + type: string + description: Optional table alias prefix prepended to each left-side column. + - name: condition + type: string + description: > + The SQL logical operator used to join multiple conditions. Typically 'AND' or 'OR'. + - name: operator + type: string + description: > + The SQL comparison operator between left and right columns. Typically '=' or 'IS NOT DISTINCT FROM'. + - name: right_columns + type: list[string] + description: > + Optional list of column names for the right side. If not provided, each left column + is compared to itself (useful for self-joins or deduplication conditions). + + - name: default__multikey + description: Default implementation of multikey. Accepts the same arguments as the parent multikey macro. diff --git a/macros/rehashing/internal_overwrites/internal_overwrites.yml b/macros/rehashing/internal_overwrites/internal_overwrites.yml new file mode 100644 index 00000000..4459f22f --- /dev/null +++ b/macros/rehashing/internal_overwrites/internal_overwrites.yml @@ -0,0 +1,78 @@ +version: 2 + +macros: + - name: custom_alter_relation_add_remove_columns + description: > + Internal override of dbt's alter_relation_add_remove_columns. Adds or removes columns + on an existing relation using adapter-appropriate DDL. The BigQuery implementation uses + ADD COLUMN IF NOT EXISTS / DROP COLUMN IF EXISTS to make the operation idempotent. + Other adapters fall back to dbt's built-in alter_relation_add_remove_columns. + Used by the rehash macros to synchronise the target relation's schema after a rehash. + arguments: + - name: relation + type: any + description: The dbt relation object representing the table to alter. + - name: add_columns + type: list[column] + description: List of dbt Column objects to add to the relation. + - name: remove_columns + type: list[column] + description: List of dbt Column objects to remove from the relation. + + - name: default__custom_alter_relation_add_remove_columns + description: Default implementation of custom_alter_relation_add_remove_columns delegating to dbt's built-in. Accepts the same arguments as the parent macro. + + - name: bigquery__custom_alter_relation_add_remove_columns + description: BigQuery-specific implementation of custom_alter_relation_add_remove_columns using ADD COLUMN IF NOT EXISTS / DROP COLUMN IF EXISTS. Accepts the same arguments as the parent macro. + + - name: custom_get_rename_column_sql + description: > + Internal override of dbt's get_rename_column_sql. Returns the DDL statement for renaming + a column on an existing relation. Adapter overrides handle platforms where dbt's default + does not produce correct syntax (e.g. Databricks ALTER TABLE ... RENAME COLUMN, + Snowflake dynamic-table-aware rename). Used by the rehash macros when renaming columns + after a rehash operation. + arguments: + - name: relation + type: any + description: The dbt relation object representing the table containing the column. + - name: old_col_name + type: string + description: The current name of the column to rename. + - name: new_col_name + type: string + description: The new name for the column. + + - name: default__custom_get_rename_column_sql + description: Default implementation of custom_get_rename_column_sql delegating to dbt's built-in get_rename_column_sql. Accepts the same arguments as the parent macro. + + - name: databricks__custom_get_rename_column_sql + description: Databricks-specific implementation of custom_get_rename_column_sql using ALTER TABLE ... RENAME COLUMN. Accepts the same arguments as the parent macro. + + - name: snowflake__custom_get_rename_column_sql + description: Snowflake-specific implementation of custom_get_rename_column_sql. Handles dynamic tables via the DDL prefix for ALTER. Accepts the same arguments as the parent macro. + + - name: rehash_deprecated_relation + description: > + Returns a dbt relation object representing the '_deprecated' copy of the given relation, + built deterministically by appending '_deprecated' to the identifier. Unlike + make_temp_relation, this never adds uniqueness suffixes, ensuring CREATE, RENAME, and + DROP operations all reference the same identifier even across interrupted runs. + arguments: + - name: relation + type: any + description: The source dbt relation object to derive the deprecated copy from. + + - name: rehash_prepare_rename + description: > + Renames the given relation to its '_deprecated' copy in preparation for a rehash operation. + Idempotent — if the deprecated copy already exists (from an interrupted prior run) it is + dropped first, so a stale leftover never blocks a fresh run. Logs the steps taken when + output_logs is true. + arguments: + - name: relation + type: any + description: The dbt relation object to rename. + - name: output_logs + type: bool + description: Whether to emit log messages for each step. Default is true. diff --git a/macros/rehashing/multiple_entities/rehash_multiple_entities.yml b/macros/rehashing/multiple_entities/rehash_multiple_entities.yml index 88789ea3..358f961e 100644 --- a/macros/rehashing/multiple_entities/rehash_multiple_entities.yml +++ b/macros/rehashing/multiple_entities/rehash_multiple_entities.yml @@ -22,7 +22,7 @@ macros: - c_custkey ``` - name: drop_old_values - type: boolean + type: bool description: > Whether to drop the old hashkey columns after rehashing. If true, the old hashkeys are dropped. If false, they are renamed with a "_deprecated" suffix. @@ -67,7 +67,7 @@ macros: business_keys: - c_custkey - name: drop_old_values - type: boolean + type: bool description: > Whether to drop the old hashkey columns after rehashing. If true, the old hashkeys are dropped. If false, they are renamed with a "_deprecated" suffix. @@ -96,7 +96,7 @@ macros: business_keys: C_CUSTKEY ``` - name: drop_old_values - type: boolean + type: bool description: > Whether to drop the old hashkey and hashdiff columns after rehashing. If true, the old columns are dropped. If false, they are renamed with a "_deprecated" suffix. @@ -121,7 +121,7 @@ macros: parent_entity: order_customer_nl ``` - name: drop_old_values - type: boolean + type: bool description: > Whether to drop the old hashkey column after rehashing. If true, the old column is dropped. If false, it is renamed with a "_deprecated" suffix. @@ -153,7 +153,7 @@ macros: - C_CUSTKEY ``` - name: drop_old_values - type: boolean + type: bool description: > Whether to drop the old hashkey and hashdiff columns after rehashing. If true, the old columns are dropped. If false, they are renamed with a "_deprecated" suffix. \ No newline at end of file diff --git a/macros/rehashing/rehash_all_rdv_entities.yml b/macros/rehashing/rehash_all_rdv_entities.yml index 1b0b9025..fbdfbd86 100644 --- a/macros/rehashing/rehash_all_rdv_entities.yml +++ b/macros/rehashing/rehash_all_rdv_entities.yml @@ -11,7 +11,7 @@ macros: satellites, multi-active satellites, and non-historized satellites. This YAML is used to drive the rehash process for all specified entities. - name: drop_old_values - type: boolean + type: bool description: > A boolean flag that determines whether to drop the old hashkey columns after the rehash operation. If set to `true`, the old hashkey columns are dropped. If set to `false`, they are retained with a diff --git a/macros/rehashing/single_entities/rehash_single_entities.yml b/macros/rehashing/single_entities/rehash_single_entities.yml index 087214d4..384f76d0 100644 --- a/macros/rehashing/single_entities/rehash_single_entities.yml +++ b/macros/rehashing/single_entities/rehash_single_entities.yml @@ -16,16 +16,16 @@ macros: type: string description: The name of the existing hashkey column. - name: business_keys - type: string|list - description: The business key(s) used to calculate the new hash. + type: any + description: The business key(s) used to calculate the new hash. Accepts either a single string or a list of strings. - name: overwrite_hash_values - type: boolean + type: bool description: Whether to overwrite the old hashkey with the new one. - name: output_logs - type: boolean + type: bool description: Whether to output logs during the macro execution. - name: drop_old_values - type: boolean + type: bool description: Whether to drop the old hashkey column after renaming. - name: hub_update_statement @@ -43,7 +43,7 @@ macros: type: string description: The name of the existing hashkey column. - name: hash_config_dict - type: dict + type: dict[string, any] description: A dictionary containing the hash configuration. - name: rehash_single_link @@ -62,19 +62,19 @@ macros: type: string description: The name of the existing link hashkey column. - name: hub_config - type: list + type: list[dict[string, any]] description: A list of dictionaries, where each dictionary contains hub_hashkey, hub_name, and business_keys for each hub involved in the link. - name: additional_hash_input_cols - type: list + type: list[string] description: A list of additional columns to include in the link hashkey calculation. - name: overwrite_hash_values - type: boolean + type: bool description: Whether to overwrite the old hashkeys with the new ones. - name: output_logs - type: boolean + type: bool description: Whether to output logs during the macro execution. - name: drop_old_values - type: boolean + type: bool description: Whether to drop the old hashkey columns after renaming. - name: link_update_statement @@ -86,7 +86,7 @@ macros: type: relation description: The link table relation. - name: hub_hashkeys - type: list + type: list[dict[string, any]] description: A list of dictionaries containing hub hashkey information. - name: link_hashkey type: string @@ -95,7 +95,7 @@ macros: type: string description: The name of the new link hashkey column. - name: additional_hash_input_cols - type: list + type: list[string] description: Additional columns used in hash calculation. - name: rehash_single_ma_satellite @@ -116,25 +116,25 @@ macros: type: string description: The name of the existing hashdiff column. - name: ma_keys - type: list + type: list[string] description: The multi-active key(s) of the satellite. - name: payload - type: list + type: list[string] description: The payload columns of the satellite. - name: parent_entity type: string description: The name of the parent entity (hub or link). - name: business_keys - type: string|list - description: The business key(s) of the parent entity. + type: any + description: The business key(s) of the parent entity. Accepts either a single string or a list of strings. - name: overwrite_hash_values - type: boolean + type: bool description: Whether to overwrite the old hashkey and hashdiff with the new ones. - name: output_logs - type: boolean + type: bool description: Whether to output logs during the macro execution. - name: drop_old_values - type: boolean + type: bool description: Whether to drop the old hashkey and hashdiff columns after renaming. - name: ma_satellite_update_statement @@ -155,16 +155,16 @@ macros: type: string description: The name of the existing hashkey column. - name: business_key_list - type: list + type: list[string] description: A list of business keys. - name: ma_keys - type: list + type: list[string] description: A list of multi-active keys. - name: ldts_col type: string description: The load date timestamp column name. - name: hash_config_dict - type: dict + type: dict[string, any] description: A dictionary containing the hash configuration. - name: parent_relation type: relation @@ -195,16 +195,16 @@ macros: type: string description: The name of the parent entity (hub or link). - name: business_keys - type: string|list - description: The business key(s) of the parent entity. Optional, only needed if parent is hub. + type: any + description: The business key(s) of the parent entity. Optional, only needed if parent is hub. Accepts either a single string or a list of strings. - name: overwrite_hash_values - type: boolean + type: bool description: Whether to overwrite the old hashkey with the new one. - name: output_logs - type: boolean + type: bool description: Whether to output logs during the macro execution. - name: drop_old_values - type: boolean + type: bool description: Whether to drop the old hashkey column after renaming. - name: nh_satellite_update_statement @@ -228,7 +228,7 @@ macros: type: relation description: The parent entity relation. - name: hash_config_dict - type: dict + type: dict[string, any] description: A dictionary containing the hash configuration. - name: rehash_single_satellite @@ -249,22 +249,22 @@ macros: type: string description: The name of the existing hashdiff column. - name: payload - type: list + type: list[string] description: The payload columns of the satellite. - name: parent_entity type: string description: The name of the parent entity (hub or link). - name: business_keys - type: string|list - description: The business key(s) of the parent entity. Optional, only needed if parent is hub. + type: any + description: The business key(s) of the parent entity. Optional, only needed if parent is hub. Accepts either a single string or a list of strings. - name: overwrite_hash_values - type: boolean + type: bool description: Whether to overwrite the old hashkey and hashdiff with the new ones. - name: output_logs - type: boolean + type: bool description: Whether to output logs during the macro execution. - name: drop_old_values - type: boolean + type: bool description: Whether to drop the old hashkey and hashdiff columns after renaming. - name: satellite_update_statement @@ -288,7 +288,7 @@ macros: type: string description: The load date timestamp column name. - name: hash_config_dict - type: dict + type: dict[string, any] description: A dictionary containing the hash configuration. - name: parent_relation type: relation diff --git a/macros/staging/stage.md b/macros/staging/stage.md new file mode 100644 index 00000000..071aea3c --- /dev/null +++ b/macros/staging/stage.md @@ -0,0 +1,88 @@ +{% docs stage %} + +## Stage + +Creates the staging layer for the Data Vault model. This layer is mainly for hashing, and additionally gives the +option to create derived columns, conduct prejoins and add NULL values for missing columns. Always create one stage +per source table that you want to add to the Data Vault model. The staging layer is not to harmonize data — that +will be done in the later layers. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.stage( + source_model='source_account', + ldts='edwLoadDate', + rsrc='!SAP.Accounts', + hashed_columns={ + 'hk_account_h': ['account_number', 'account_key'], + 'hd_account_s': { + 'is_hashdiff': true, + 'columns': ['name', 'address', 'country', 'phone', 'email'] + } + }, + derived_columns={ + 'country_isocode': {'value': '!GER', 'datatype': 'STRING'} + } +) }} +{% endraw %} +``` + +### Multi-active stage + +```jinja +{% raw %} +{{ datavault4dbt.stage( + source_model='source_contact_phones', + ldts='edwLoadDate', + rsrc='!CRM.ContactPhones', + hashed_columns={ + 'hk_contact_h': ['contact_id'], + 'hd_contact_phonenumber_s': { + 'is_hashdiff': true, + 'columns': ['phone_type', 'phone_number'] + } + }, + multi_active_config={ + 'multi_active_key': 'phone_type', + 'main_hashkey_column': 'hk_contact_h' + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +source_model: 'source_account' +ldts: 'edwLoadDate' +rsrc: '!SAP.Accounts' +hashed_columns: + hk_account_h: + - account_number + - account_key + hd_account_s: + is_hashdiff: true + columns: + - name + - address + - country + - phone + - email +derived_columns: + country_isocode: + value: '!GER' + datatype: STRING +{%- endset -%} + +{{ datavault4dbt.stage(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/staging/stage.yml b/macros/staging/stage.yml new file mode 100644 index 00000000..b5b84a99 --- /dev/null +++ b/macros/staging/stage.yml @@ -0,0 +1,118 @@ +version: 2 + +macros: + - name: stage + description: '{{ doc("stage") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: ldts + type: string + description: > + Name of the column inside the source data, that holds information about the Load Date Timestamp. + Can also be a SQL expression. + Examples: 'edwLoadDate' (column name as-is), + "PARSE_TIMESTAMP('%Y-%m-%dT%H-%M-%S', edwLoadDate)" (applying a SQL function). + - name: rsrc + type: string + description: > + Name of the column inside the source data, that holds information about the Record Source. Can also + be a SQL expression or a static string. A static string must begin with a '!'. + Examples: 'edwRecordSource' (column), '!SAP.Accounts' (static string), + "CONCAT(source_system, '||', source_object)" (expression). + - name: source_model + type: any + description: > + Can be just a string holding the name of the referred dbt model to use as a source. But if the 'source' + functionality inside the .yml file is used, it must be a dictionary with 'source_name': 'source_table'. + Examples: 'source_account' (model name), {'source_data': 'source_account'} (dbt source reference). + - name: include_source_columns + type: bool + description: > + Defines if all columns from the referred source table should be included in the result table, or if only + the added columns should be part of the result table. By default the source columns should be included. + - name: hashed_columns + type: dict[string, any] + description: > + Defines the names and input for all hashkeys and hashdiffs to create. The key of each hash column is + the name of the hash column. The value for Hashkeys is a list of input Business Keys; for Hashdiffs + another dictionary with the pairs 'is_hashdiff:true' and 'columns: '. + Example: {'hk_account_h': ['account_number', 'account_key'], 'hd_account_s': {'is_hashdiff': true, 'columns': ['name', 'address']}}. + - name: derived_columns + type: dict[string, dict[string, string]] + description: > + Defines values and datatypes for derived ('added' or 'calculated') columns. The values of this dictionary + are the desired column names; the value is another dictionary with the keys 'value' (holding a column name, + a SQL expression, or a static string beginning with '!') and 'datatype' (holding a valid SQL datatype). + Example: {'country_isocode': {'value': '!GER', 'datatype': 'STRING'}}. + - name: sequence + type: string + description: > + Name of the column inside the source data, that holds a sequence number generated during data source + extraction. Optional and not required. + Example: 'edwSequence'. + - name: prejoined_columns + type: any + description: > + Defines information about columns that need to be prejoined. Most commonly used to create links when + the source data does not hold the Business Key but the technical key of the referred object. Accepts + either a dictionary of dictionaries (old syntax) or a list of dictionaries (new syntax); see + process_prejoined_columns. For each prejoined column define: 'src_name', 'src_table' (or 'ref_model'), + 'bk', 'this_column_name', 'ref_column_name'. + - name: missing_columns + type: dict[string, string] + description: > + If the schema of the source changes over time and columns are disappearing, this parameter gives you + the option to create additional columns holding NULL values. The dictionary holds the column names as + keys and the SQL datatypes as values. + Example: {'legacy_account_uuid': 'INT64', 'shipping_address': 'STRING'}. + - name: multi_active_config + type: dict[string, string] + description: > + If the source data holds multi-active data, define here the column(s) holding the multi-active key and + the main hashkey column. The combination of multi-active key(s), the main hashkey and the ldts column + should be unique in the final result satellite. If not set, the stage will be treated as a single-active stage. + Example: {'multi_active_key': 'phonetype', 'main_hashkey_column': 'hk_contact_h'}. + - name: enable_ghost_records + type: bool + description: > + If set to true, the stage will be created with ghost records. By default, ghost records are enabled. + Optional Parameter. + + - name: snowflake__stage + description: Snowflake-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: databricks__stage + description: Databricks-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: exasol__stage + description: Exasol-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: fabric__stage + description: Microsoft Fabric-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: oracle__stage + description: Oracle-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: postgres__stage + description: PostgreSQL-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: redshift__stage + description: Amazon Redshift-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: sqlserver__stage + description: SQL Server-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: synapse__stage + description: Azure Synapse Analytics-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: trino__stage + description: Trino-specific implementation of stage. Accepts the same arguments as the parent stage macro. + + - name: default__stage + description: > + Default implementation of stage, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent stage macro. diff --git a/macros/staging/staging.yml b/macros/staging/staging.yml index aa97fdbd..1538fe1e 100644 --- a/macros/staging/staging.yml +++ b/macros/staging/staging.yml @@ -3,13 +3,15 @@ version: 2 macros: - name: process_prejoined_columns description: > - A macro to process prejoined columns. If a list of dictioniaries(new syntax) is provided it will do nothing and return the list. - If a dictionary of dictionaries if provided(old syntax) it will be transformed to the new syntax. - When multiple columns are to be extracted from the same prejoin-target and with the same conditions(columns and operator) they will be combined into one item. + A macro to process prejoined columns. If a list of dictionaries (new syntax) is provided, it will do nothing and return the list. + If a dictionary of dictionaries is provided (old syntax), it will be transformed to the new syntax. + When multiple columns are to be extracted from the same prejoin target and with the same conditions (columns and operator), they will be combined into one item. arguments: - name: prejoined_columns - type: list|dict - description: The value of the prejoined_columns as defined in the yaml_metadata of the stage-model. + type: any + description: > + The value of the prejoined_columns as defined in the yaml_metadata of the stage-model. Accepts either + a list of dictionaries (new syntax) or a dictionary of dictionaries (old syntax). - name: extract_prejoin_column_names description: > @@ -19,5 +21,142 @@ macros: Returns an empty list if the passed parameter is empty. arguments: - name: prejoined_columns - type: list - description: The prejoined_columns as process by the process_prejoined_columns-macro \ No newline at end of file + type: list[dict[string, any]] + description: The prejoined_columns as processed by the process_prejoined_columns-macro + + - name: derive_columns + description: > + Renders the SELECT-list of a stage model's derived ('added' or 'calculated') columns together with the + remaining source columns, in source-columns-then-derived-columns order. Dispatches per adapter. + arguments: + - name: source_relation + type: relation + description: The dbt relation of the source model, used to look up its existing columns. + - name: columns + type: dict[string, dict[string, any]] + description: > + Dictionary of derived column definitions, keyed by the desired output column name. Same shape as the + 'derived_columns' parameter of the stage macro. + + - name: default__derive_columns + description: Default implementation of derive_columns. Accepts the same arguments as the parent macro. + + - name: derived_columns_datatypes + description: > + Enriches a derived_columns definition with the detected 'datatype', 'value' and 'col_size' for each entry + that does not already define them explicitly, by looking up the referenced column in the source relation. + Returns the enriched dictionary as a JSON string. Dispatches per adapter. + arguments: + - name: columns + type: dict[string, dict[string, any]] + description: Dictionary of derived column definitions, keyed by the desired output column name. + - name: source_relation + type: relation + description: The dbt relation of the source model, used to detect datatypes and column sizes. + + - name: default__derived_columns_datatypes + description: Default implementation of derived_columns_datatypes. Accepts the same arguments as the parent macro. + + - name: hash_columns + description: > + Renders the SELECT-list of hashkey and hashdiff expressions for a stage model's 'hashed_columns' + definition, delegating to the hash macro for each entry and applying multi-active satellite hashing + rules when a multi-active key is configured. Dispatches per adapter. + arguments: + - name: columns + type: dict[string, any] + description: Dictionary of hashkey/hashdiff definitions. Same shape as the 'hashed_columns' parameter of the stage macro. + - name: multi_active_key + type: any + description: Optional multi-active key column(s), used to apply multi-active hashing rules. + - name: main_hashkey_column + type: string + description: Optional name of the parent hashkey column, used in multi-active hashdiff calculations. + + - name: default__hash_columns + description: Default implementation of hash_columns. Accepts the same arguments as the parent macro. + + - name: redshift__hash_columns + description: Redshift-specific implementation of hash_columns. Accepts the same arguments as the parent macro. + + - name: source_columns + description: > + Returns a plain list of column name strings for the given relation, wrapping + adapter.get_columns_in_relation(). + arguments: + - name: source_relation + type: relation + description: The dbt relation to look up columns for. + + - name: process_columns_to_select + description: > + Filters a list of column names down to only those not present in an exclude list, normalising casing + per the adapter's 'datavault4dbt.set_casing' configuration. Dispatches per adapter. + arguments: + - name: columns_list + type: list[string] + description: The full list of column names to filter. + - name: exclude_columns_list + type: list[string] + description: The list of column names to exclude from the result. + + - name: default__process_columns_to_select + description: Default implementation of process_columns_to_select. Accepts the same arguments as the parent macro. + + - name: fabric__process_columns_to_select + description: Microsoft Fabric-specific implementation of process_columns_to_select. Accepts the same arguments as the parent macro. + + - name: databricks__process_columns_to_select + description: Databricks-specific implementation of process_columns_to_select. Accepts the same arguments as the parent macro. + + - name: synapse__process_columns_to_select + description: Azure Synapse Analytics-specific implementation of process_columns_to_select. Delegates to default__process_columns_to_select. + + - name: sqlserver__process_columns_to_select + description: SQL Server-specific implementation of process_columns_to_select. Delegates to default__process_columns_to_select. + + - name: extract_column_names + description: > + Returns the list of keys of a columns dictionary (e.g. a 'hashed_columns' or 'derived_columns' + definition), or an empty list if the input is not a mapping. + arguments: + - name: columns_dict + type: dict[string, any] + description: The columns dictionary to extract keys from. + + - name: extract_input_columns + description: > + Extracts the list of underlying source column names referenced by a columns or prejoin definition, + handling the 'src_cols_required', hashdiff 'columns', and prejoin 'this_column_name' shapes. + arguments: + - name: columns_dict + type: any + description: A dictionary of hash/derived column configs, or a list of processed prejoin dictionaries. + + - name: process_hash_column_excludes + description: > + For each entry in a 'hashed_columns' definition that specifies 'exclude_columns' instead of an explicit + 'columns' list, resolves the actual list of columns to hash by subtracting the excludes from the full + list of source columns. + arguments: + - name: hash_columns + type: dict[string, any] + description: Dictionary of hashkey/hashdiff definitions, potentially using the 'exclude_columns' shorthand. + - name: source_columns + type: list[string] + description: The full list of available source column names. + + - name: print_list + description: > + Renders a comma-separated, indented list of column names for use inside a SELECT statement, optionally + prefixed with a source alias. + arguments: + - name: list_to_print + type: list[string] + description: The list of column names to render. + - name: indent + type: integer + description: Number of spaces to indent each rendered line. Defaults to 4. + - name: src_alias + type: string + description: Optional table alias prepended to each column name (e.g. 'src.column_name'). \ No newline at end of file diff --git a/macros/supporting/supporting.yml b/macros/supporting/supporting.yml new file mode 100644 index 00000000..f9d37c9e --- /dev/null +++ b/macros/supporting/supporting.yml @@ -0,0 +1,406 @@ +version: 2 + +macros: + - name: beginning_of_all_times + description: > + Returns the configured beginning-of-all-times timestamp string used as the load date for + ghost records (unknown records). Reads the global variable 'datavault4dbt.beginning_of_all_times'. + Default value is '0001-01-01T00-00-01'. Dispatches per adapter. + + - name: default__beginning_of_all_times + description: Default implementation of beginning_of_all_times. No arguments. + + - name: beginning_of_all_times_date + description: > + Returns the configured beginning-of-all-times date string (date only, no time component) + used for ghost records. Reads the global variable 'datavault4dbt.beginning_of_all_times_date'. + Default value is '0001-01-01'. Dispatches per adapter. + + - name: default__beginning_of_all_times_date + description: Default implementation of beginning_of_all_times_date. No arguments. + + - name: current_timestamp + description: > + Returns the current timestamp expression for the target adapter. Wraps dbt.current_timestamp() + by default. Overrides exist for adapters that need a different syntax (e.g. Synapse, Fabric). + Dispatches per adapter. + + - name: default__current_timestamp + description: Default implementation of current_timestamp using dbt.current_timestamp(). No arguments. + + - name: synapse__current_timestamp + description: Synapse-specific implementation of current_timestamp using sysdatetime(). No arguments. + + - name: fabric__current_timestamp + description: Fabric-specific implementation of current_timestamp returning CAST(SYSDATETIME() AS DATETIME2(6)). No arguments. + + - name: current_timestamp_in_utc + description: > + Returns the current UTC timestamp expression for the target adapter. Overrides exist for + adapters that distinguish local time from UTC (e.g. Synapse, Fabric). Dispatches per adapter. + + - name: default__current_timestamp_in_utc + description: Default implementation of current_timestamp_in_utc using dbt.current_timestamp(). No arguments. + + - name: synapse__current_timestamp_in_utc + description: Synapse-specific implementation of current_timestamp_in_utc using sysutcdatetime(). No arguments. + + - name: fabric__current_timestamp_in_utc + description: Fabric-specific implementation of current_timestamp_in_utc using sysutcdatetime(). No arguments. + + - name: type_timestamp + description: > + Returns the adapter-appropriate SQL data type keyword for timestamp columns. Default is + dbt.type_timestamp(). Overrides exist for adapters that use non-standard keyword spellings + (e.g. Synapse, Fabric, SQL Server). Dispatches per adapter. + + - name: default__type_timestamp + description: Default implementation of type_timestamp using dbt.type_timestamp(). No arguments. + + - name: synapse__type_timestamp + description: Synapse-specific implementation of type_timestamp returning 'datetime2'. No arguments. + + - name: fabric__type_timestamp + description: Fabric-specific implementation of type_timestamp returning 'datetime2(6)'. No arguments. + + - name: sqlserver__type_timestamp + description: SQL Server-specific implementation of type_timestamp returning 'datetime2(7)'. No arguments. + + - name: date_format + description: > + Returns the configured date format string used when parsing date values. Reads the global + variable 'datavault4dbt.date_format'. Default value is '%Y-%m-%d'. Dispatches per adapter. + + - name: default__date_format + description: Default implementation of date_format. No arguments. + + - name: end_of_all_times + description: > + Returns the configured end-of-all-times timestamp string used as the load end date for + active satellite records. Reads the global variable 'datavault4dbt.end_of_all_times'. + Default value is '8888-12-31T23-59-59'. Dispatches per adapter. + + - name: default__end_of_all_times + description: Default implementation of end_of_all_times. No arguments. + + - name: end_of_all_times_date + description: > + Returns the configured end-of-all-times date string (date only, no time component). + Reads the global variable 'datavault4dbt.end_of_all_times_date'. + Default value is '8888-12-31'. Dispatches per adapter. + + - name: default__end_of_all_times_date + description: Default implementation of end_of_all_times_date. No arguments. + + - name: first_day_of_week + description: > + Returns the configured first day of the week as an integer. Reads the global variable + 'datavault4dbt.first_day_of_week'. Default value is 1 (Monday). Dispatches per adapter. + + - name: default__first_day_of_week + description: Default implementation of first_day_of_week. No arguments. + + - name: get_static_analysis_config + description: > + Returns a static analysis configuration override ('off' or none) for a given macro name + and target adapter. Used internally to suppress Snowpark static analysis warnings on + macros that generate dynamic SQL. Reads the global variable + 'datavault4dbt.enable_static_analysis_overwrite'. + arguments: + - name: macro_name + type: string + description: The name of the macro to look up in the static analysis configuration. + + - name: generate_schema_name + description: > + Overrides the dbt built-in generate_schema_name behaviour. If a custom_schema_name is + provided the schema name is used exactly as-is (no target schema prefix), otherwise the + target schema is used. This matches standard single-schema project conventions. + arguments: + - name: custom_schema_name + type: string + description: The custom schema name defined in the model config, or none. + - name: node + type: any + description: The dbt node object (passed by dbt internals, not set by users). + + - name: get_distinct_value + description: > + Queries the given source relation and returns a single distinct value from the specified + column, optionally excluding a list of values. Used internally to resolve dynamic column + values at compile time. Dispatches per adapter. + arguments: + - name: source_relation + type: any + description: A dbt relation object pointing to the source table or view to query. + - name: column_name + type: string + description: The column from which to retrieve a distinct value. + - name: exclude_values + type: list[any] + description: Optional list of values to exclude from the result. + + - name: default__get_distinct_value + description: Default implementation of get_distinct_value. Accepts the same arguments as the parent macro. + + - name: get_field_hash_by_datatype + description: > + Compares hashed column names against the list of all columns in a relation to determine + each column's data type, then substitutes special type-aware expressions where needed + (e.g. DECODE for BOOLEAN, FNV_HASH(ST_AsBinary(...)) for GEOMETRY on Snowflake). + Returns the corrected list of hashed column expressions. + arguments: + - name: hashed_columns + type: list[string] + description: List of column name strings that will be included in a hash calculation. + - name: all_datatype_columns + type: list[column] + description: List of dbt Column objects representing all columns of the source relation. + - name: derived_columns + type: dict[string, dict[string, string]] + description: Optional dict of derived column definitions that may override the detected datatype. + + - name: get_standard_string + description: > + Generates a BigQuery-specific SQL expression that concatenates a list of columns into a + standardised string, replacing NULL values with '^^' placeholders and trimming each value. + arguments: + - name: string_list + type: list[string] + description: List of column names to concatenate into the standardised string. + + - name: ghost_record_per_datatype + description: > + Generates the SQL expression for a single ghost record column, rendering either the + unknown or the error ghost value appropriate for the column's data type. Adapts the + expression to the target platform via adapter dispatch. + arguments: + - name: column_name + type: string + description: The name of the column for which the ghost record value is generated. + - name: datatype + type: string + description: The SQL data type of the column (e.g. 'TIMESTAMP', 'VARCHAR', 'NUMBER'). + - name: ghost_record_type + type: string + description: Whether to render the 'unknown' or the 'error' ghost record value. + - name: col_size + type: string + description: Optional column size, used for fixed-length character types. + - name: alias + type: string + description: Optional column alias in the generated expression. Defaults to column_name. + + - name: default__ghost_record_per_datatype + description: Default implementation of ghost_record_per_datatype. Accepts the same arguments as the parent macro. + + - name: hash + description: > + Generates the complete hash (hashkey or hashdiff) SQL expression for one or more columns. + Handles string standardisation, null replacement, concatenation, and the chosen hash + algorithm. Adapter-specific implementations handle platform differences in hash function + syntax. Dispatches per adapter. + arguments: + - name: columns + type: list[string] + description: List of column names or expressions to hash together. + - name: alias + type: string + description: The output column alias for the hash expression. + - name: is_hashdiff + type: bool + description: If true, generates a hashdiff expression. If false, generates a hashkey. Default is false. + - name: multi_active_key + type: any + description: Optional multi-active key configuration used in multi-active satellite hashing. + - name: main_hashkey_column + type: string + description: Optional name of the parent hashkey column, used in hashdiff calculations. + - name: use_trim + type: bool + description: > + Whether to trim string values before hashing. Defaults to the global variable + 'datavault4dbt.hashdiff_use_trim' for hashdiffs, or true for hashkeys. + - name: rtrim_hashdiff + type: bool + description: If true, applies RTRIM to the final hashdiff concatenated string. Default is false. + + - name: default__hash + description: Default implementation of hash. Accepts the same arguments as the parent hash macro. + + - name: hash_default_values + description: > + Returns a JSON string containing the hash algorithm name and the pre-computed unknown + and error key values for the given hash function. Used to populate ghost record hash columns. + Dispatches per adapter. + arguments: + - name: hash_function + type: string + description: The hash function name to look up (e.g. 'MD5', 'SHA256'). + - name: hash_datatype + type: string + description: Optional data type override for the hash column. + + - name: default__hash_default_values + description: Default implementation of hash_default_values. Accepts the same arguments as the parent macro. + + - name: hash_method + description: > + Returns the configured hash algorithm name for the target adapter. Reads the global + variable 'datavault4dbt.hash'. Default value is 'MD5'. Dispatches per adapter. + + - name: default__hash_method + description: Default implementation of hash_method. No arguments. + + - name: attribute_standardise + description: > + Generates the SQL expression that casts and standardises a single column value before + it is included in a hash calculation. Handles quoting, NULL replacement, and + platform-specific cast syntax. Dispatches per adapter. + arguments: + - name: hash_type + type: string + description: Optional hash type context passed through to the adapter implementation. + - name: use_trim + type: bool + description: Whether to apply TRIM to the cast expression. Default is true. + + - name: default__attribute_standardise + description: Default implementation of attribute_standardise. Accepts the same arguments as the parent macro. + + - name: limit_rows + description: > + Returns the adapter-appropriate SQL clause for limiting result rows in development + queries (e.g. 'TOP 100'). On Synapse and Fabric, returns an empty string when + target.schema == 'prod'. Dispatches per adapter. + + - name: default__limit_rows + description: Default implementation of limit_rows returning 'TOP 100'. No arguments. + + - name: max_datetime + description: > + Returns the maximum representable datetime literal string for the target adapter. + Used as a sentinel upper bound in date range calculations. Dispatches per adapter. + + - name: default__max_datetime + description: Default implementation of max_datetime returning '9999-12-31 23:59:59.999999'. No arguments. + + - name: filter_latest_entries_in_sat + description: > + Returns an optional SQL WHERE clause fragment that filters satellite queries to only + the latest entries for the current hashkeys. An empty string is returned on adapters + where this optimisation is not applicable. Dispatches per adapter. + + - name: default__filter_latest_entries_in_sat + description: Default implementation of filter_latest_entries_in_sat returning an empty string. No arguments. + + - name: prefix + description: > + Applies a table alias prefix to a list of column names or column alias mappings, + generating 'prefix.column' references for use in JOIN conditions or SELECT lists. + Handles plain column strings, list-of-lists, and alias dicts. Dispatches per adapter. + arguments: + - name: columns + type: list[any] + description: List of column names, nested lists, or alias dicts to prefix. + - name: prefix_str + type: string + description: The table alias to prepend (e.g. 'src', 'tgt'). + - name: alias_target + type: string + description: > + For alias dicts, which side to prefix — 'source' uses the source_column, + 'target' uses the alias. Default is 'source'. + + - name: default__prefix + description: Default implementation of prefix. Accepts the same arguments as the parent prefix macro. + + - name: source_model_should_be_selected + description: > + Returns true if the given source model name is present in the current dbt run's selected + resources. Used to implement execution-aware loading — skipping source models that were + not selected in the current run to avoid empty-join errors. + arguments: + - name: source_model_name + type: string + description: The dbt model name to check against the selected_resources list. + + - name: source_model_processing + description: > + Normalises and enriches the source_models input into a standardised list of source + model dicts, resolving rsrc_static values, business keys, reference keys, and other + per-source metadata. Called internally by hub, link, satellite and reference macros + before rendering their SQL. + arguments: + - name: source_models + type: any + description: > + The raw source_models parameter as passed by the user — may be a string (single + source), a list of strings, or a dict mapping model names to per-source config dicts. + - name: set_rsrc_static + type: bool + description: Whether to resolve and validate rsrc_static values. Default is true. + - name: parameters + type: dict[string, any] + description: Additional per-source parameters merged into a single-source dict when source_models is a string. + - name: business_keys + type: any + description: Business key columns forwarded from the calling macro. + - name: reference_keys + type: any + description: Reference key columns forwarded from the calling macro. + - name: foreign_hashkeys + type: any + description: Foreign hashkey columns forwarded from the calling macro. + - name: payload + type: any + description: Payload columns forwarded from the calling macro. + + - name: string_default_dtype + description: > + Returns the configured default string SQL data type for a given column category. + Reads the appropriate global variable (datavault4dbt.rsrc_default_dtype, + datavault4dbt.stg_default_dtype, or datavault4dbt.derived_columns_default_dtype) + based on the type argument. Default value is 'STRING'. Dispatches per adapter. + arguments: + - name: type + type: string + description: > + The column category to look up. Accepts 'rsrc', 'stg', or 'derived_columns'. + + - name: default__string_default_dtype + description: Default implementation of string_default_dtype. Accepts the same arguments as the parent macro. + + - name: string_to_timestamp + description: > + Converts a string literal to a timestamp using the adapter-appropriate SQL function and + the given format string. Used to render beginning_of_all_times and end_of_all_times + ghost record values as typed timestamp columns. Dispatches per adapter. + arguments: + - name: format + type: string + description: The timestamp format string (e.g. '%Y-%m-%dT%H-%M-%S'). + - name: timestamp + type: string + description: The timestamp string literal to convert. + + - name: default__string_to_timestamp + description: Default (BigQuery) implementation of string_to_timestamp using PARSE_TIMESTAMP. Accepts the same arguments as the parent macro. + + - name: timestamp_default_dtype + description: > + Returns the configured default SQL data type for timestamp columns. Reads the global + variable 'datavault4dbt.timestamp_default_dtype'. Default value is 'TIMESTAMP'. + Dispatches per adapter. + + - name: default__timestamp_default_dtype + description: Default implementation of timestamp_default_dtype. No arguments. + + - name: timestamp_format + description: > + Returns the configured timestamp format string used when parsing timestamp literals. + Reads the global variable 'datavault4dbt.timestamp_format'. Default value is + '%Y-%m-%dT%H-%M-%S'. Dispatches per adapter. + + - name: default__timestamp_format + description: Default implementation of timestamp_format. No arguments. diff --git a/macros/tables/control_snap_v0.md b/macros/tables/control_snap_v0.md new file mode 100644 index 00000000..3ede1bcd --- /dev/null +++ b/macros/tables/control_snap_v0.md @@ -0,0 +1,42 @@ +{% docs control_snap_v0 %} + +## Control Snapshot v0 + +Creates a snapshot table to control snapshot-based tables like PITs and Bridges. The snapshot table will hold +daily snapshots starting at a specific start_date with a configurable daytime. Usually one snapshot table per +Data Vault environment is created. The model needs to be scheduled daily at the time matching the desired +snapshot time. + +In addition to the actual snapshot datetimestamp (sdts), the macro generates the following metadata columns: +`replacement_sdts`, `caption`, `is_hourly`, `is_daily`, `is_beginning_of_week`, `is_end_of_week`, +`is_beginning_of_month`, `is_end_of_month`, `is_beginning_of_quarter`, `is_end_of_quarter`, +`is_beginning_of_year`, `is_end_of_year`, `comment`, `force_active`. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.control_snap_v0( + start_date='2020-01-01T00-00-00', + daily_snapshot_time='07:00:00' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +start_date: '2020-01-01T00-00-00' +daily_snapshot_time: '07:00:00' +{%- endset -%} + +{{ datavault4dbt.control_snap_v0(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/control_snap_v0.yml b/macros/tables/control_snap_v0.yml new file mode 100644 index 00000000..ecadc49f --- /dev/null +++ b/macros/tables/control_snap_v0.yml @@ -0,0 +1,68 @@ +version: 2 + +macros: + - name: control_snap_v0 + description: '{{ doc("control_snap_v0") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: start_date + type: string + description: > + Defines the earliest timestamp that should be available inside the snapshot_table. The time part of this + timestamp needs to be set to '00:00:00'. The format of this timestamp must equal to the timestamp format + defined in the global variable 'datavault4dbt.timestamp_format'. + Examples: '2015-01-01T00-00-00' — This snapshot table would hold daily snapshots beginning at 2015. + - name: daily_snapshot_time + type: string + description: > + Defines the time that your daily snapshots should have. Usually this is either something right before + daily business starts, or after daily business is over. + Examples: '07:30:00', '23:00:00'. + - name: sdts_alias + type: string + description: > + Defines the name of the snapshot date timestamp column inside the snapshot_table. It is optional, + if not set will use the global variable 'datavault4dbt.sdts_alias' set inside dbt_project.yml. + - name: end_date + type: string + description: > + Defines the latest timestamp that should be available inside the snapshot_table. + + - name: snowflake__control_snap_v0 + description: Snowflake-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: databricks__control_snap_v0 + description: Databricks-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: exasol__control_snap_v0 + description: Exasol-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: fabric__control_snap_v0 + description: Microsoft Fabric-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: oracle__control_snap_v0 + description: Oracle-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: postgres__control_snap_v0 + description: PostgreSQL-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: redshift__control_snap_v0 + description: Amazon Redshift-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: sqlserver__control_snap_v0 + description: SQL Server-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: synapse__control_snap_v0 + description: Azure Synapse Analytics-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: trino__control_snap_v0 + description: Trino-specific implementation of control_snap_v0. Accepts the same arguments as the parent control_snap_v0 macro. + + - name: default__control_snap_v0 + description: > + Default implementation of control_snap_v0, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent control_snap_v0 macro. diff --git a/macros/tables/control_snap_v1.md b/macros/tables/control_snap_v1.md new file mode 100644 index 00000000..7893334b --- /dev/null +++ b/macros/tables/control_snap_v1.md @@ -0,0 +1,62 @@ +{% docs control_snap_v1 %} + +## Control Snapshot v1 + +Creates a view that extends an existing control_snap_v0 table by dynamically applying logarithmic snapshot logic. +The further you look into the past, the more coarsely the snapshots are granulated — for example, keeping daily +snapshots for the past 30 days but only weekly snapshots for the past 6 months and monthly snapshots for the +past 3 years. + +This procedure strongly reduces the number of active snapshots and therefore the computation inside all PITs +and Bridges. + +In addition to the logarithmic `is_active` column, the following dynamic columns are generated: +`is_latest`, `is_current_year`, `is_last_year`, `is_rolling_year`, `is_last_rolling_year`. + +Note: Whenever a logarithmic snapshot logic is used with PIT tables, the `clean_up_pit` post-hook must be +applied to each PIT table to remove inactive records. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.control_snap_v1( + control_snap_v0='control_snap_v0', + log_logic={ + 'daily': {'duration': 3, 'unit': 'MONTH'}, + 'weekly': {'duration': 1, 'unit': 'YEAR'}, + 'monthly': {'duration': 5, 'unit': 'YEAR'}, + 'yearly': {'forever': 'TRUE'} + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +control_snap_v0: 'control_snap_v0' +log_logic: + daily: + duration: 3 + unit: MONTH + weekly: + duration: 1 + unit: YEAR + monthly: + duration: 5 + unit: YEAR + yearly: + forever: 'TRUE' +{%- endset -%} + +{{ datavault4dbt.control_snap_v1(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/control_snap_v1.yml b/macros/tables/control_snap_v1.yml new file mode 100644 index 00000000..c7e63bb9 --- /dev/null +++ b/macros/tables/control_snap_v1.yml @@ -0,0 +1,64 @@ +version: 2 + +macros: + - name: control_snap_v1 + description: '{{ doc("control_snap_v1") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: control_snap_v0 + type: string + description: > + The name of the underlying version 0 control snapshot table. Needs to be available as a dbt model. + - name: log_logic + type: dict[string, any] + description: > + Defining the desired durations of each granularity. Available granularities are 'daily', 'weekly', + 'monthly', and 'yearly'. For each granularity the duration can be defined as an integer and the time + unit for that duration (e.g. DAY, WEEK, MONTH, QUARTER, YEAR in BigQuery). A granularity can also be + set to 'forever'. If log_logic is not set, no logic will be applied and all snapshots will stay active. + The duration is always counted from the current date. + EXASOL: Due to a missing 'DAY OF WEEK' function, 'weekly' is not supported and must be left out. + Example: {'daily': {'duration': 3, 'unit': 'MONTH'}, 'weekly': {'duration': 1, 'unit': 'YEAR'}, + 'monthly': {'duration': 5, 'unit': 'YEAR'}, 'yearly': {'forever': 'TRUE'}}. + - name: sdts_alias + type: string + description: > + Defines the name of the snapshot date timestamp column inside the snapshot_table. It is optional, + if not set will use the global variable 'datavault4dbt.sdts_alias' set inside dbt_project.yml. + + - name: snowflake__control_snap_v1 + description: Snowflake-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: databricks__control_snap_v1 + description: Databricks-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: exasol__control_snap_v1 + description: Exasol-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: fabric__control_snap_v1 + description: Microsoft Fabric-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: oracle__control_snap_v1 + description: Oracle-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: postgres__control_snap_v1 + description: PostgreSQL-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: redshift__control_snap_v1 + description: Amazon Redshift-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: sqlserver__control_snap_v1 + description: SQL Server-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: synapse__control_snap_v1 + description: Azure Synapse Analytics-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: trino__control_snap_v1 + description: Trino-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. + + - name: bigquery__control_snap_v1 + description: BigQuery-specific implementation of control_snap_v1. Accepts the same arguments as the parent control_snap_v1 macro. diff --git a/macros/tables/eff_sat_v0.md b/macros/tables/eff_sat_v0.md new file mode 100644 index 00000000..efc53bd5 --- /dev/null +++ b/macros/tables/eff_sat_v0.md @@ -0,0 +1,53 @@ +{% docs eff_sat_v0 %} + +## Effectivity Satellite v0 + +Creates an Effectivity Satellite that tracks the active/inactive status of a link relationship over time. +An effectivity satellite contains a boolean flag ('is_active') that indicates whether a given relationship +was active at each load timestamp. + +Note: `source_is_single_batch` is a required parameter — it must be explicitly set to `true` or `false`. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.eff_sat_v0( + source_model='stg_account_contact', + tracked_hashkey='hk_account_contact_l', + source_is_single_batch=false +) }} +{% endraw %} +``` + +### With custom alias for the active flag + +```jinja +{% raw %} +{{ datavault4dbt.eff_sat_v0( + source_model='stg_account_contact', + tracked_hashkey='hk_account_contact_l', + is_active_alias='is_active', + source_is_single_batch=false +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +source_model: 'stg_account_contact' +tracked_hashkey: 'hk_account_contact_l' +source_is_single_batch: false +{%- endset -%} + +{{ datavault4dbt.eff_sat_v0(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/eff_sat_v0.yml b/macros/tables/eff_sat_v0.yml new file mode 100644 index 00000000..02f3d99b --- /dev/null +++ b/macros/tables/eff_sat_v0.yml @@ -0,0 +1,79 @@ +version: 2 + +macros: + - name: eff_sat_v0 + description: '{{ doc("eff_sat_v0") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: source_model + type: string + description: > + Name of the source model. + - name: tracked_hashkey + type: string + description: > + Name of the hashkey column to be tracked. + - name: src_ldts + type: string + description: > + Name of the loaddate column in the source model. Optional. + - name: src_rsrc + type: string + description: > + Name of the record source column in the source model. Optional. + - name: is_active_alias + type: string + description: > + Name of the new active flag column. Optional. + - name: source_is_single_batch + type: bool + description: > + REQUIRED. Whether the source contains only one batch. Must be defined explicitly (true/false). + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be disabled or not. Optional. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the eff_sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__eff_sat_v0 + description: Snowflake-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: databricks__eff_sat_v0 + description: Databricks-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: exasol__eff_sat_v0 + description: Exasol-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: fabric__eff_sat_v0 + description: Microsoft Fabric-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: oracle__eff_sat_v0 + description: Oracle-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: postgres__eff_sat_v0 + description: PostgreSQL-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: redshift__eff_sat_v0 + description: Amazon Redshift-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: sqlserver__eff_sat_v0 + description: SQL Server-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: synapse__eff_sat_v0 + description: Azure Synapse Analytics-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: trino__eff_sat_v0 + description: Trino-specific implementation of eff_sat_v0. Accepts the same arguments as the parent eff_sat_v0 macro. + + - name: default__eff_sat_v0 + description: > + Default implementation of eff_sat_v0, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent eff_sat_v0 macro. diff --git a/macros/tables/hub.md b/macros/tables/hub.md new file mode 100644 index 00000000..d698f533 --- /dev/null +++ b/macros/tables/hub.md @@ -0,0 +1,58 @@ +{% docs hub %} + +## Hub + +Creates a Hub entity in the Raw Data Vault. A Hub captures and stores the unique business keys for a business concept, +along with the first load date and record source. Each business key combination is stored only once. + +Features: +- Loadable by multiple sources +- Supports multiple updates per batch and therefore initial loading +- Can use a dynamic high-water-mark to optimize loading performance of multiple loads +- Allows source mappings for deviations between source column names and hub column names + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.hub( + hashkey='hk_account_h', + business_keys='account_id', + source_models={ + 'stg_account_crm': { + 'bk_columns': 'account_id', + 'rsrc_static': '*/CRM/Accounts/*' + }, + 'stg_account_erp': { + 'bk_columns': 'account_id', + 'rsrc_static': '*/ERP/Accounts/*' + } + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +hashkey: 'hk_account_h' +business_keys: account_id +source_models: + stg_account_crm: + bk_columns: account_id + rsrc_static: '*/CRM/Accounts/*' + stg_account_erp: + bk_columns: account_id + rsrc_static: '*/ERP/Accounts/*' +{%- endset -%} + +{{ datavault4dbt.hub(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/hub.yml b/macros/tables/hub.yml new file mode 100644 index 00000000..84816df5 --- /dev/null +++ b/macros/tables/hub.yml @@ -0,0 +1,88 @@ +version: 2 + +macros: + - name: hub + description: '{{ doc("hub") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: hashkey + type: string + description: > + Name of the hashkey column inside the stage, that should be used as PK of the Hub. + Examples: 'hk_account_h' — This hashkey column was created before inside the corresponding staging area, using the stage macro. + - name: business_keys + type: any + description: > + Name(s) of the business key columns that should be loaded into the hub and are the input of the hashkey column. + Needs to be available inside the stage model. If the names differ between multiple sources, define here how the business + keys should be called inside the final hub model. The actual input column names need to be defined inside the + 'source_model' parameter then. + Examples: 'account_key' (single), ['account_key', 'account_number'] (composite). + - name: source_models + type: dict[string, dict[string, any]] + description: > + Dictionary with information about the source models. The keys of the dict are the names of the source models, + and the value of each source model is another dictionary. This inner dictionary requires the key 'bk_columns' + (which contains the name of the business keys of that source model), and can have the optional keys 'hk_column' + and 'rsrc_static'. The 'rsrc_static' attribute defines a STRING or list of strings containing all the patterns + of the record_source field that remain the same over all loads of one source. Wildcards (e.g. '*') can be used + to match variable parts. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of Hub. Useful when you have to + deviate from the default Hub structure. The columns need to be available in all source models which are used + for the Hub. Optional parameter, defaults to empty list. + + - name: snowflake__hub + description: Snowflake-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: databricks__hub + description: Databricks-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: exasol__hub + description: Exasol-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: fabric__hub + description: Microsoft Fabric-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: oracle__hub + description: Oracle-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: postgres__hub + description: PostgreSQL-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: redshift__hub + description: Amazon Redshift-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: sqlserver__hub + description: SQL Server-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: synapse__hub + description: Azure Synapse Analytics-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: trino__hub + description: Trino-specific implementation of hub. Accepts the same arguments as the parent hub macro. + + - name: default__hub + description: > + Default implementation of hub, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent hub macro. diff --git a/macros/tables/link.md b/macros/tables/link.md new file mode 100644 index 00000000..001ab2aa --- /dev/null +++ b/macros/tables/link.md @@ -0,0 +1,46 @@ +{% docs link %} + +## Link + +Creates a Link entity in the Raw Data Vault. A Link connects two or more entities, or an entity with itself. +It can be loaded by one or more source staging tables if multiple sources share the same business definitions. +If multiple sources are used, they must all have the same number of foreign keys. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.link( + link_hashkey='hk_account_contact_l', + foreign_hashkeys=['hk_account_h', 'hk_contact_h'], + source_models={ + 'stg_account_contact': { + 'rsrc_static': '*/CRM/AccountContact/*' + } + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +link_hashkey: 'hk_account_contact_l' +foreign_hashkeys: + - hk_account_h + - hk_contact_h +source_models: + stg_account_contact: + rsrc_static: '*/CRM/AccountContact/*' +{%- endset -%} + +{{ datavault4dbt.link(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/link.yml b/macros/tables/link.yml new file mode 100644 index 00000000..1549b525 --- /dev/null +++ b/macros/tables/link.yml @@ -0,0 +1,87 @@ +version: 2 + +macros: + - name: link + description: '{{ doc("link") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: link_hashkey + type: string + description: > + Name of the link hashkey column inside the stage. Should get calculated out of all business keys inside the link. + Examples: 'hk_account_contact_l' — This hashkey column belongs to the link between account and contact, and + was created at the staging layer by the stage macro. + - name: foreign_hashkeys + type: list[string] + description: > + List of all hashkey columns inside the link, that refer to other hub entities. All hashkey columns must + be available inside the stage area. + Examples: ['hk_account_h', 'hk_contact_h'] — The link between account and contact needs to contain both + the hashkey of account and contact to enable joins to the corresponding hub entities. + - name: source_models + type: dict[string, dict[string, any]] + description: > + Dictionary with information about the source models. The keys of the dict are the names of the source models, + and the value of each source model is another dictionary. This inner dictionary requires the key 'rsrc_static', + and optionally the keys 'hk_column' and 'fk_columns'. The 'rsrc_static' attribute defines a STRING or list of + strings containing all the patterns of the record_source field that remain the same over all loads of one source. + Wildcards (e.g. '*') can be used to match variable parts. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of Link. Useful when you have to + deviate from the default Link structure. The columns need to be available in all source models which are used + for the Link. Optional parameter, defaults to empty list. + + - name: snowflake__link + description: Snowflake-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: databricks__link + description: Databricks-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: exasol__link + description: Exasol-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: fabric__link + description: Microsoft Fabric-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: oracle__link + description: Oracle-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: postgres__link + description: PostgreSQL-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: redshift__link + description: Amazon Redshift-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: sqlserver__link + description: SQL Server-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: synapse__link + description: Azure Synapse Analytics-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: trino__link + description: Trino-specific implementation of link. Accepts the same arguments as the parent link macro. + + - name: default__link + description: > + Default implementation of link, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent link macro. diff --git a/macros/tables/ma_sat_v0.md b/macros/tables/ma_sat_v0.md new file mode 100644 index 00000000..ca10573f --- /dev/null +++ b/macros/tables/ma_sat_v0.md @@ -0,0 +1,66 @@ +{% docs ma_sat_v0 %} + +## Multi-Active Satellite v0 + +Creates a multi-active satellite version 0, materialized as an incremental table. Applied on top of the staging layer, +connected to either a Hub or a Link. A multi-active satellite stores multiple concurrently valid records per parent +hashkey, distinguished by a multi-active key (e.g. multiple phone numbers per contact). + +On top of each version 0 multi-active satellite, a version 1 should be created using the ma_sat_v1 macro. +If a stage model is defined as multi-active, all satellites out of that stage model must be implemented as +multi-active satellites. + +Features: +- Can handle multiple updates per batch, without losing intermediate changes — initial loading is supported. +- Uses a dynamic high-water-mark to optimize loading performance of multiple loads. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.ma_sat_v0( + parent_hashkey='hk_contact_h', + src_hashdiff='hd_contact_phonenumber_s', + src_ma_key='phonetype', + src_payload=['phone_number', 'is_primary'], + source_model='stage_contact' +) }} +{% endraw %} +``` + +### With composite multi-active key + +```jinja +{% raw %} +{{ datavault4dbt.ma_sat_v0( + parent_hashkey='hk_contact_h', + src_hashdiff='hd_contact_phonenumber_s', + src_ma_key=['phonetype', 'iid'], + src_payload=['phone_number'], + source_model='stage_contact' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +parent_hashkey: 'hk_contact_h' +src_hashdiff: 'hd_contact_phonenumber_s' +src_ma_key: 'phonetype' +src_payload: + - phone_number + - is_primary +source_model: 'stage_contact' +{%- endset -%} + +{{ datavault4dbt.ma_sat_v0(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ma_sat_v0.yml b/macros/tables/ma_sat_v0.yml new file mode 100644 index 00000000..f748ff1b --- /dev/null +++ b/macros/tables/ma_sat_v0.yml @@ -0,0 +1,99 @@ +version: 2 + +macros: + - name: ma_sat_v0 + description: '{{ doc("ma_sat_v0") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: parent_hashkey + type: string + description: > + Name of the hashkey column inside the stage of the object that this satellite is attached to. + Examples: 'hk_account_h' (attached to hub account), 'hk_account_contact_l' (attached to a link). + - name: src_hashdiff + type: string + description: > + Name of the hashdiff column of this satellite, that was created inside the staging area and is + calculated out of the entire payload of this satellite. The stage must hold one hashdiff per + satellite entity. Examples: 'hd_account_data_sfdc_s'. + - name: src_ma_key + type: any + description: > + Name(s) of the multi-active keys inside the staging area. Need to be the same ones as defined in the stage model. + Examples: 'phonetype' (single), ['phonetype', 'company'] (composite — the combination of both is treated as the key). + - name: src_payload + type: list[string] + description: > + A list of all the descriptive attributes that should be included in this satellite. Needs to be the columns that + are fed into the hashdiff calculation of this satellite. Do not include the multi-active key in the payload of a + multi-active satellite, it is included automatically! + Examples: ['name', 'address', 'country', 'phone', 'email']. + - name: source_model + type: string + description: > + Name of the underlying staging model, must be available inside dbt as a model. + Examples: 'stage_account' — This satellite is loaded out of the stage for account. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source model. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source model. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the ma_sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + - name: disable_hwm + type: bool + description: > + Whether the automatic application of a High-Water Mark (HWM) should be disabled or not. + Optional parameter, defaults to False. + - name: source_is_single_batch + type: bool + description: > + Boosts performance by disabling QUALIFY statement. Only activate this if you made sure that the underlying + staging model only holds one row per entry. Optional parameter, defaults to False. + + - name: snowflake__ma_sat_v0 + description: Snowflake-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: databricks__ma_sat_v0 + description: Databricks-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: exasol__ma_sat_v0 + description: Exasol-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: fabric__ma_sat_v0 + description: Microsoft Fabric-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: oracle__ma_sat_v0 + description: Oracle-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: postgres__ma_sat_v0 + description: PostgreSQL-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: redshift__ma_sat_v0 + description: Amazon Redshift-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: sqlserver__ma_sat_v0 + description: SQL Server-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: synapse__ma_sat_v0 + description: Azure Synapse Analytics-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: trino__ma_sat_v0 + description: Trino-specific implementation of ma_sat_v0. Accepts the same arguments as the parent ma_sat_v0 macro. + + - name: default__ma_sat_v0 + description: > + Default implementation of ma_sat_v0, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ma_sat_v0 macro. diff --git a/macros/tables/ma_sat_v1.md b/macros/tables/ma_sat_v1.md new file mode 100644 index 00000000..95f9c813 --- /dev/null +++ b/macros/tables/ma_sat_v1.md @@ -0,0 +1,59 @@ +{% docs ma_sat_v1 %} + +## Multi-Active Satellite v1 + +Calculates the load end dates for multi-active data, based on a multi-active attribute. Must be based on a version 0 +multi-active satellite, that would then hold multiple records per hashkey+ldts combination. + +A version 1 multi-active satellite should be materialized as a view by default. + +Features: +- Calculates virtualized load-end-dates to correctly identify multiple active records per batch. +- Enforces insert-only approach by view materialization. +- Allows multiple attributes to be used as the multi-active-attribute. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.ma_sat_v1( + sat_v0='contact_phonenumber_0_s', + hashkey='hk_contact_h', + hashdiff='hd_contact_phonenumber_s', + ma_attribute='phone_type' +) }} +{% endraw %} +``` + +### With composite multi-active attribute + +```jinja +{% raw %} +{{ datavault4dbt.ma_sat_v1( + sat_v0='contact_phonenumber_0_s', + hashkey='hk_contact_h', + hashdiff='hd_contact_phonenumber_s', + ma_attribute=['phone_type', 'iid'] +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +sat_v0: 'contact_phonenumber_0_s' +hashkey: 'hk_contact_h' +hashdiff: 'hd_contact_phonenumber_s' +ma_attribute: 'phone_type' +{%- endset -%} + +{{ datavault4dbt.ma_sat_v1(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ma_sat_v1.yml b/macros/tables/ma_sat_v1.yml new file mode 100644 index 00000000..3f80ce71 --- /dev/null +++ b/macros/tables/ma_sat_v1.yml @@ -0,0 +1,91 @@ +version: 2 + +macros: + - name: ma_sat_v1 + description: '{{ doc("ma_sat_v1") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: sat_v0 + type: string + description: > + Name of the underlying version 0 multi-active satellite. + Examples: 'contact_phonenumber_0_s' — This satellite would be the version 1 satellite of the underlying + version 0 phone number satellite for contacts. + - name: hashkey + type: string + description: > + Name of the parent hashkey column inside the version 0 satellite. Would either be the hashkey of a + hub or a link. Needs to be similar to the 'parent_hashkey' parameter inside the sat_v0 model. + Examples: 'hk_contact_h' (hub), 'hk_order_contact_l' (link). + - name: hashdiff + type: string + description: > + Name of the hashdiff column inside the underlying version 0 satellite. Needs to be similar to the + 'src_hashdiff' parameter inside the sat_v0 model. Must not include the ma_attribute in calculation. + Examples: 'hd_contact_phonenumber_s'. + - name: ma_attribute + type: any + description: > + Name of the multi active attribute inside the v0 satellite. This needs to be identified under the + requirement that the combination of hashkey + ldts + ma_attribute is unique over the entire stage/satellite. + Examples: 'phone_type' (single), ['phone_type', 'iid'] (composite — needed when multiple records share the same phone_type). + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: ledts_alias + type: string + description: > + Desired alias for the load end date column. Is optional, will use the global variable 'datavault4dbt.ledts_alias' + if not set here. + - name: add_is_current_flag + type: bool + description: > + Optional parameter to add a new column to the v1 sat based on the load end date timestamp (ledts). Default is false. + If set to true it will add this is_current flag to the v1 sat. For each record this column will be set to true if + the load end date timestamp is equal to the variable end of all times. If not, the record is not current. + + - name: snowflake__ma_sat_v1 + description: Snowflake-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: databricks__ma_sat_v1 + description: Databricks-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: exasol__ma_sat_v1 + description: Exasol-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: fabric__ma_sat_v1 + description: Microsoft Fabric-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: oracle__ma_sat_v1 + description: Oracle-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: postgres__ma_sat_v1 + description: PostgreSQL-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: redshift__ma_sat_v1 + description: Amazon Redshift-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: sqlserver__ma_sat_v1 + description: SQL Server-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: synapse__ma_sat_v1 + description: Azure Synapse Analytics-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: trino__ma_sat_v1 + description: Trino-specific implementation of ma_sat_v1. Accepts the same arguments as the parent ma_sat_v1 macro. + + - name: default__ma_sat_v1 + description: > + Default implementation of ma_sat_v1, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ma_sat_v1 macro. diff --git a/macros/tables/nh_link.md b/macros/tables/nh_link.md new file mode 100644 index 00000000..66f31424 --- /dev/null +++ b/macros/tables/nh_link.md @@ -0,0 +1,55 @@ +{% docs nh_link %} + +## Non-Historized Link + +Creates a non-historized (formerly transactional) link entity, connecting two or more entities, or a transactional +fact of one entity. It can be loaded by one or more source staging tables if multiple sources share the same +business definitions. + +In the background a non-historized link uses exactly the same loading logic as a regular link, but adds the +descriptive attributes as additional payload. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.nh_link( + link_hashkey='hk_transaction_account_nl', + foreign_hashkeys=['hk_transaction_h', 'hk_account_h'], + payload=['currency_isocode', 'amount', 'purpose', 'transaction_date'], + source_models={ + 'stg_transaction': { + 'rsrc_static': '*/ERP/Transactions/*' + } + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +link_hashkey: 'hk_transaction_account_nl' +foreign_hashkeys: + - hk_transaction_h + - hk_account_h +payload: + - currency_isocode + - amount + - purpose + - transaction_date +source_models: + stg_transaction: + rsrc_static: '*/ERP/Transactions/*' +{%- endset -%} + +{{ datavault4dbt.nh_link(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/nh_link.yml b/macros/tables/nh_link.yml new file mode 100644 index 00000000..d70f2ae0 --- /dev/null +++ b/macros/tables/nh_link.yml @@ -0,0 +1,105 @@ +version: 2 + +macros: + - name: nh_link + description: '{{ doc("nh_link") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: link_hashkey + type: string + description: > + Name of the non-historized link hashkey column inside the stage. Should get calculated out of all + business keys inside the link. + Examples: 'hk_transaction_account_nl' — This hashkey column belongs to the non-historized link + between transaction and account, and was created at the staging layer by the stage macro. + - name: payload + type: list[string] + description: > + A list of all the descriptive attributes that should be the payload of this non-historized link. + If the names differ between source models, this list defines how the columns are named inside the + result non-historized link. The mapping which columns to use from which source model must then be + defined inside the 'payload' key in 'source_models'. + Examples: ['currency_isocode', 'amount', 'purpose', 'transaction_date']. + - name: source_models + type: dict[string, dict[string, any]] + description: > + Dictionary with information about the source models. The keys of the dict are the names of the source + models, and the value of each source model is another dictionary. This inner dictionary optionally has + the keys 'hk_column', 'fk_columns', 'payload' and 'rsrc_static'. The 'rsrc_static' attribute defines + a STRING or list of strings containing all the patterns of the record_source field that remain the same + over all loads of one source. Wildcards (e.g. '*') can be used to match variable parts. + - name: foreign_hashkeys + type: list[string] + description: > + List of all hashkey columns inside the non-historized link, that refer to other hub entities. + All hashkey columns must be available inside the stage area. + Examples: ['hk_transaction_h', 'hk_account_h']. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: source_is_single_batch + type: bool + description: > + Whether the source contains only one batch. Optional, default False. + - name: union_strategy + type: string + description: > + Defines how multiple sources should be unioned. 'all' will result in a UNION ALL and represents the + default value. Should only be changed if you have duplicates across source systems and do not want + to deduplicate them upfront. Accepted values: 'all', 'distinct'. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of NH-Link. Useful when + you have to deviate from the default NH-Link structure. The columns need to be available in all source + models which are used for the NH-Link. Optional parameter, defaults to empty list. + + - name: snowflake__nh_link + description: Snowflake-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: databricks__nh_link + description: Databricks-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: exasol__nh_link + description: Exasol-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: fabric__nh_link + description: Microsoft Fabric-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: oracle__nh_link + description: Oracle-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: postgres__nh_link + description: PostgreSQL-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: redshift__nh_link + description: Amazon Redshift-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: sqlserver__nh_link + description: SQL Server-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: synapse__nh_link + description: Azure Synapse Analytics-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: trino__nh_link + description: Trino-specific implementation of nh_link. Accepts the same arguments as the parent nh_link macro. + + - name: default__nh_link + description: > + Default implementation of nh_link, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent nh_link macro. diff --git a/macros/tables/nh_sat.md b/macros/tables/nh_sat.md new file mode 100644 index 00000000..41ec761e --- /dev/null +++ b/macros/tables/nh_sat.md @@ -0,0 +1,46 @@ +{% docs nh_sat %} + +## Non-Historized Satellite + +Creates a non-historized satellite, materialized as an incremental table. Applied on top of the staging layer, +connected to either a Hub or a Link. Besides the missing hashdiff, a non-historized satellite applies the same +loading logic as a regular version 0 satellite. Each satellite can only be loaded by one source model. + +Features: +- High-performance loading of non-historized satellite data. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.nh_sat( + parent_hashkey='hk_account_h', + src_payload=['name', 'address', 'country', 'phone', 'email'], + source_model='stage_account' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +parent_hashkey: 'hk_account_h' +src_payload: + - name + - address + - country + - phone + - email +source_model: 'stage_account' +{%- endset -%} + +{{ datavault4dbt.nh_sat(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/nh_sat.yml b/macros/tables/nh_sat.yml new file mode 100644 index 00000000..eb062d77 --- /dev/null +++ b/macros/tables/nh_sat.yml @@ -0,0 +1,80 @@ +version: 2 + +macros: + - name: nh_sat + description: '{{ doc("nh_sat") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: parent_hashkey + type: string + description: > + Name of the hashkey column inside the stage of the object that this satellite is attached to. + Examples: 'hk_account_h' (attached to hub account), 'hk_account_contact_l' (attached to a link). + - name: src_payload + type: list[string] + description: > + A list of all the descriptive attributes that should be included in this satellite. + Examples: ['name', 'address', 'country', 'phone', 'email']. + - name: source_model + type: string + description: > + Name of the underlying staging model, must be available inside dbt as a model. + Examples: 'stage_account' — This satellite is loaded out of the stage for account. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source model. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source model. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: source_is_single_batch + type: bool + description: > + Whether the source contains only one batch. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the nh_sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__nh_sat + description: Snowflake-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: databricks__nh_sat + description: Databricks-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: exasol__nh_sat + description: Exasol-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: fabric__nh_sat + description: Microsoft Fabric-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: oracle__nh_sat + description: Oracle-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: postgres__nh_sat + description: PostgreSQL-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: redshift__nh_sat + description: Amazon Redshift-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: sqlserver__nh_sat + description: SQL Server-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: synapse__nh_sat + description: Azure Synapse Analytics-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: trino__nh_sat + description: Trino-specific implementation of nh_sat. Accepts the same arguments as the parent nh_sat macro. + + - name: default__nh_sat + description: > + Default implementation of nh_sat, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent nh_sat macro. diff --git a/macros/tables/pit.md b/macros/tables/pit.md new file mode 100644 index 00000000..4220aeac --- /dev/null +++ b/macros/tables/pit.md @@ -0,0 +1,54 @@ +{% docs pit %} + +## Point-in-Time (PIT) Table + +Creates a PIT table to gather snapshot-based information about one hub and its surrounding satellites. +For this macro to work, a snapshot table is required that has a trigger column to identify which snapshots +to include in the PIT table. The easiest way to create such a snapshot table is to use the control_snap macros +provided by this package. + +Features: +- Tracks the active satellite entries for each hub entry at each snapshot. +- Strongly improves performance if upstream queries require many JOIN operations. +- Creates a unique dimension key to optimize loading performance of incremental loads. +- Allows inserting a static string as record source column, matching business vault definition. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.pit( + tracked_entity='account_h', + hashkey='hk_account_h', + sat_names=[ + 'account_data_sfdc_1_s', + 'account_financials_erp_1_s' + ], + snapshot_relation='control_snap_v1', + dimension_key='hk_account_h_d' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +tracked_entity: 'account_h' +hashkey: 'hk_account_h' +sat_names: + - account_data_sfdc_1_s + - account_financials_erp_1_s +snapshot_relation: 'control_snap_v1' +dimension_key: 'hk_account_h_d' +{%- endset -%} + +{{ datavault4dbt.pit(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/pit.yml b/macros/tables/pit.yml new file mode 100644 index 00000000..bc50aaaa --- /dev/null +++ b/macros/tables/pit.yml @@ -0,0 +1,114 @@ +version: 2 + +macros: + - name: pit + description: '{{ doc("pit") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: tracked_entity + type: string + description: > + Name of the tracked Hub entity. Must be available as a model inside the dbt project. + - name: hashkey + type: string + description: > + The name of the hashkey column inside the previously referred Hub entity. + - name: sat_names + type: list[string] + description: > + A list of all the satellites that should be included in this PIT table. Can only be satellites that + are attached to the tracked Hub, and should typically include all those satellites. You should always + refer here to the version 1 satellites, since those hold the load-end-date. The macro currently + supports regular satellites and nh-satellites. + - name: snapshot_relation + type: string + description: > + The name of the snapshot relation. It needs to be available as a model inside this dbt project. + - name: dimension_key + type: string + description: > + The desired name of the dimension key inside the PIT table. Should follow some naming conventions. + Recommended is the name of the hashkey with a '_d' suffix. + - name: snapshot_trigger_column + type: string + description: > + The name of the column inside the previously mentioned snapshot relation, that is boolean and identifies + the snapshots that should be included in the PIT table. + - name: ldts + type: string + description: > + Name of the ldts column inside all source models. Is optional, will use the global variable + 'datavault4dbt.ldts_alias'. Needs to use the same column name as defined as alias inside the staging model. + - name: custom_rsrc + type: string + description: > + A custom string that should be inserted into the 'rsrc' column inside the PIT table. Since a PIT table + is a business vault entity, the technical record source is no longer used here. Is optional, if not + defined, no column is added. + - name: ledts + type: string + description: > + Name of the load-end-date column inside the satellites. Is optional, will use the global variable + 'datavault4dbt.ledts_alias' if not set here. + - name: sdts + type: string + description: > + Name of the snapshot date timestamp column inside the snapshot table. It is optional, will use the + global variable 'datavault4dbt.sdts_alias' if not set here. + - name: pit_type + type: string + description: > + String to insert into the 'pit_type' column. Has to be prefixed by a !. Can be set freely, something + like 'PIT' could be the default. Is optional, if not set, no column will be added. + - name: refer_to_ghost_records + type: bool + description: > + Value to define if a NULL satellite hashkey should be replaced by the unknown key. + Optional parameter, default is True. + - name: snapshot_optimization + type: bool + description: > + If set to True, and if the model is run in incremental mode, only the relevant snapshots + (i.e. those that are newer than or equal to the max sdts in the existing PIT table) will be considered. + This can significantly improve performance of incremental loads on large snapshot tables. If set to True, + the model needs to be configured with a unique_key constraint as there may be updates due to late arriving data. + Affected Adapters: Snowflake only! Optional parameter, default is False. + + - name: snowflake__pit + description: Snowflake-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: databricks__pit + description: Databricks-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: exasol__pit + description: Exasol-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: fabric__pit + description: Microsoft Fabric-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: oracle__pit + description: Oracle-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: postgres__pit + description: PostgreSQL-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: redshift__pit + description: Amazon Redshift-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: sqlserver__pit + description: SQL Server-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: synapse__pit + description: Azure Synapse Analytics-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: trino__pit + description: Trino-specific implementation of pit. Accepts the same arguments as the parent pit macro. + + - name: default__pit + description: > + Default implementation of pit, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent pit macro. diff --git a/macros/tables/rec_track_sat.md b/macros/tables/rec_track_sat.md new file mode 100644 index 00000000..7a0f7d52 --- /dev/null +++ b/macros/tables/rec_track_sat.md @@ -0,0 +1,57 @@ +{% docs rec_track_sat %} + +## Record Tracking Satellite + +Creates a Record Tracking Satellite, most commonly used to track the appearances of hashkeys (calculated out of +business keys) inside one or multiple source systems. This can either be the hashkey of a hub, or the hashkey +of a link. Typically if a hub is loaded from three sources, the corresponding Record Tracking Satellite would +track the same three sources. + +Features: +- Tracks the appearance of a specific hashkey in one or more staging areas. +- Allows source mappings for deviations between the hashkey name inside the stages and the target. +- Supports multiple updates per batch and therefore initial loading. +- Uses a dynamic high-water-mark to optimize loading performance of multiple loads. +- Can track either link or hub hashkeys. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.rec_track_sat( + tracked_hashkey='hk_contact_h', + source_models={ + 'stg_contact_crm': { + 'rsrc_static': '*/CRM/Contact/*' + }, + 'stg_contact_erp': { + 'hk_column': 'hk_contact_erp_h', + 'rsrc_static': '*/ERP/Contact/*' + } + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +tracked_hashkey: 'hk_contact_h' +source_models: + stg_contact_crm: + rsrc_static: '*/CRM/Contact/*' + stg_contact_erp: + hk_column: hk_contact_erp_h + rsrc_static: '*/ERP/Contact/*' +{%- endset -%} + +{{ datavault4dbt.rec_track_sat(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/rec_track_sat.yml b/macros/tables/rec_track_sat.yml new file mode 100644 index 00000000..a570eefe --- /dev/null +++ b/macros/tables/rec_track_sat.yml @@ -0,0 +1,85 @@ +version: 2 + +macros: + - name: rec_track_sat + description: '{{ doc("rec_track_sat") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: tracked_hashkey + type: string + description: > + The name of the hashkey column you want to track. Needs to be available in the underlying staging layer. + If you want to track multiple hashkeys out of one stage, you need to create one record tracking satellite + for each hashkey. + Examples: 'hk_contact_h' (tracks appearance of the contact hub hashkey), + 'hk_contact_account_l' (tracks appearance of the link hashkey). + - name: source_models + type: dict[string, dict[string, any]] + description: > + Dictionary with information about the source model. The key of the dict is the name of the source model, + and the value is another dictionary. This inner dictionary requires to have the key 'rsrc_static', and + optionally the key 'hk_column'. The 'rsrc_static' attribute defines a STRING or list of strings that will + always be the same over all loads of one source. Wildcards (e.g. '*') can be used to match variable parts. + If rsrc_static is solely '*', the performance lookup will not be executed. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_stg + type: string + description: > + Name of the source stage model. Is optional, will use the global variable 'datavault4dbt.stg_alias'. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the rec_track_sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__rec_track_sat + description: Snowflake-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: databricks__rec_track_sat + description: Databricks-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: exasol__rec_track_sat + description: Exasol-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: fabric__rec_track_sat + description: Microsoft Fabric-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: oracle__rec_track_sat + description: Oracle-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: postgres__rec_track_sat + description: PostgreSQL-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: redshift__rec_track_sat + description: Amazon Redshift-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: sqlserver__rec_track_sat + description: SQL Server-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: synapse__rec_track_sat + description: Azure Synapse Analytics-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: trino__rec_track_sat + description: Trino-specific implementation of rec_track_sat. Accepts the same arguments as the parent rec_track_sat macro. + + - name: default__rec_track_sat + description: > + Default implementation of rec_track_sat, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent rec_track_sat macro. diff --git a/macros/tables/ref_hub.md b/macros/tables/ref_hub.md new file mode 100644 index 00000000..c53616d2 --- /dev/null +++ b/macros/tables/ref_hub.md @@ -0,0 +1,44 @@ +{% docs ref_hub %} + +## Reference Hub + +Creates a Reference Hub entity in the Raw Data Vault. A Reference Hub stores the unique reference keys for +reference data (e.g. country codes, product categories) that does not originate from a business entity +but is used to classify and enrich other entities. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.ref_hub( + ref_keys='country_code', + source_models={ + 'stg_country': { + 'ref_keys': 'country_code', + 'rsrc_static': '*/REF/Country/*' + } + } +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +ref_keys: 'country_code' +source_models: + stg_country: + ref_keys: country_code + rsrc_static: '*/REF/Country/*' +{%- endset -%} + +{{ datavault4dbt.ref_hub(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ref_hub.yml b/macros/tables/ref_hub.yml new file mode 100644 index 00000000..8d995e26 --- /dev/null +++ b/macros/tables/ref_hub.yml @@ -0,0 +1,69 @@ +version: 2 + +macros: + - name: ref_hub + description: '{{ doc("ref_hub") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: ref_keys + type: any + description: > + Name of the reference key(s) available in the source model(s). + - name: source_models + type: dict[string, dict[string, any]] + description: > + Similar to other source_models parameters, e.g. in Hubs or Links. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the ref_hub. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__ref_hub + description: Snowflake-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: databricks__ref_hub + description: Databricks-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: exasol__ref_hub + description: Exasol-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: fabric__ref_hub + description: Microsoft Fabric-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: oracle__ref_hub + description: Oracle-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: postgres__ref_hub + description: PostgreSQL-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: redshift__ref_hub + description: Amazon Redshift-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: sqlserver__ref_hub + description: SQL Server-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: synapse__ref_hub + description: Azure Synapse Analytics-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: trino__ref_hub + description: Trino-specific implementation of ref_hub. Accepts the same arguments as the parent ref_hub macro. + + - name: default__ref_hub + description: > + Default implementation of ref_hub, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ref_hub macro. diff --git a/macros/tables/ref_sat_v0.md b/macros/tables/ref_sat_v0.md new file mode 100644 index 00000000..f853c568 --- /dev/null +++ b/macros/tables/ref_sat_v0.md @@ -0,0 +1,43 @@ +{% docs ref_sat_v0 %} + +## Reference Satellite v0 + +Creates a version 0 Reference Satellite in the Raw Data Vault. A Reference Satellite stores the descriptive +attributes (payload) of a reference entity over time, tracking every change. It is linked to its parent +Reference Hub via the reference key. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.ref_sat_v0( + parent_ref_keys='country_code', + src_hashdiff='hd_country_rs', + src_payload=['country_name', 'continent', 'region'], + source_model='stg_country' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +parent_ref_keys: 'country_code' +src_hashdiff: 'hd_country_rs' +src_payload: + - country_name + - continent + - region +source_model: 'stg_country' +{%- endset -%} + +{{ datavault4dbt.ref_sat_v0(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ref_sat_v0.yml b/macros/tables/ref_sat_v0.yml new file mode 100644 index 00000000..b51b8d53 --- /dev/null +++ b/macros/tables/ref_sat_v0.yml @@ -0,0 +1,91 @@ +version: 2 + +macros: + - name: ref_sat_v0 + description: '{{ doc("ref_sat_v0") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: parent_ref_keys + type: any + description: > + Name of the reference key(s) of the parent ref_hub. + - name: src_hashdiff + type: string + description: > + Name of the hashdiff column of this ref satellite, that was created inside the staging area and is + calculated out of the entire payload of this ref satellite. The stage must hold one hashdiff per + ref satellite entity. + Examples: 'hd_nation_sfdc_rs' — hashdiff column of the ref satellite for nation. + - name: src_payload + type: list[string] + description: > + A list of all the descriptive attributes that should be included in this ref satellite. Needs to be the + columns that are fed into the hashdiff calculation of this ref satellite. + Examples: ['name', 'continent', 'area']. + - name: source_model + type: string + description: > + Name of the underlying staging model, must be available inside dbt as a model. + Examples: 'stage_nation' — This ref satellite is loaded out of the stage for account. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: source_is_single_batch + type: bool + description: > + Whether the source contains only one batch. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the ref_sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__ref_sat_v0 + description: Snowflake-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: databricks__ref_sat_v0 + description: Databricks-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: exasol__ref_sat_v0 + description: Exasol-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: fabric__ref_sat_v0 + description: Microsoft Fabric-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: oracle__ref_sat_v0 + description: Oracle-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: postgres__ref_sat_v0 + description: PostgreSQL-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: redshift__ref_sat_v0 + description: Amazon Redshift-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: sqlserver__ref_sat_v0 + description: SQL Server-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: synapse__ref_sat_v0 + description: Azure Synapse Analytics-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: trino__ref_sat_v0 + description: Trino-specific implementation of ref_sat_v0. Accepts the same arguments as the parent ref_sat_v0 macro. + + - name: default__ref_sat_v0 + description: > + Default implementation of ref_sat_v0, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ref_sat_v0 macro. diff --git a/macros/tables/ref_sat_v1.md b/macros/tables/ref_sat_v1.md new file mode 100644 index 00000000..0787a5a3 --- /dev/null +++ b/macros/tables/ref_sat_v1.md @@ -0,0 +1,38 @@ +{% docs ref_sat_v1 %} + +## Reference Satellite v1 + +Creates a version 1 Reference Satellite view derived from an existing ref_sat_v0. Adds a load end date timestamp +(ledts) column, enabling point-in-time queries without window functions. A version 1 ref satellite should be +materialized as a view. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.ref_sat_v1( + ref_sat_v0='country_0_rs', + ref_keys='country_code', + hashdiff='hd_country_rs' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +ref_sat_v0: 'country_0_rs' +ref_keys: 'country_code' +hashdiff: 'hd_country_rs' +{%- endset -%} + +{{ datavault4dbt.ref_sat_v1(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ref_sat_v1.yml b/macros/tables/ref_sat_v1.yml new file mode 100644 index 00000000..bf82df05 --- /dev/null +++ b/macros/tables/ref_sat_v1.yml @@ -0,0 +1,79 @@ +version: 2 + +macros: + - name: ref_sat_v1 + description: '{{ doc("ref_sat_v1") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: ref_sat_v0 + type: string + description: > + Name of the underlying ref_sat_v0 dbt model. + - name: ref_keys + type: any + description: > + Name(s) of the reference key(s) in the underlying reference sat v0. + - name: hashdiff + type: string + description: > + Name of the Hashdiff column in the underlying reference sat v0. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: ledts_alias + type: string + description: > + Desired alias for the load end date column. Is optional, will use the global variable 'datavault4dbt.ledts_alias' + if not set here. + - name: add_is_current_flag + type: bool + description: > + Optional parameter to add a new column to the v1 sat based on the load end date timestamp (ledts). Default is false. + If set to true it will add this is_current flag to the v1 sat. For each record this column will be set to true if + the load end date timestamp is equal to the variable end of all times. If not, the record is not current. + + - name: snowflake__ref_sat_v1 + description: Snowflake-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: databricks__ref_sat_v1 + description: Databricks-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: exasol__ref_sat_v1 + description: Exasol-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: fabric__ref_sat_v1 + description: Microsoft Fabric-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: oracle__ref_sat_v1 + description: Oracle-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: postgres__ref_sat_v1 + description: PostgreSQL-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: redshift__ref_sat_v1 + description: Amazon Redshift-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: sqlserver__ref_sat_v1 + description: SQL Server-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: synapse__ref_sat_v1 + description: Azure Synapse Analytics-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: trino__ref_sat_v1 + description: Trino-specific implementation of ref_sat_v1. Accepts the same arguments as the parent ref_sat_v1 macro. + + - name: default__ref_sat_v1 + description: > + Default implementation of ref_sat_v1, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ref_sat_v1 macro. diff --git a/macros/tables/ref_table.md b/macros/tables/ref_table.md new file mode 100644 index 00000000..683579f3 --- /dev/null +++ b/macros/tables/ref_table.md @@ -0,0 +1,52 @@ +{% docs ref_table %} + +## Reference Table + +Creates a Reference Table that combines a ref_hub with one or more ref_satellites into a single queryable entity. +The historization mode controls how much history the table retains: 'latest' keeps only the current record, +'full' keeps all historical records, and 'snapshot' aligns with a snapshot table. + +### Usage (latest — current values only) + +```jinja +{% raw %} +{{ datavault4dbt.ref_table( + ref_hub='country_rh', + ref_satellites=['country_1_rs'], + historized='latest' +) }} +{% endraw %} +``` + +### Usage (snapshot-based) + +```jinja +{% raw %} +{{ datavault4dbt.ref_table( + ref_hub='country_rh', + ref_satellites=['country_1_rs'], + historized='snapshot', + snapshot_relation='control_snap_v1' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +ref_hub: 'country_rh' +ref_satellites: + - country_1_rs +historized: 'latest' +{%- endset -%} + +{{ datavault4dbt.ref_table(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/ref_table.yml b/macros/tables/ref_table.yml new file mode 100644 index 00000000..57d3a62a --- /dev/null +++ b/macros/tables/ref_table.yml @@ -0,0 +1,79 @@ +version: 2 + +macros: + - name: ref_table + description: '{{ doc("ref_table") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: ref_hub + type: string + description: > + Name of the underlying ref_hub model. + - name: ref_satellites + type: any + description: > + Name(s) of the reference satellites to be included in this ref_table. Optional: 'include' and 'exclude' + as dictionary keys for each satellite. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: historized + type: string + description: > + Possible values are 'full', 'latest', or 'snapshot'. Influences how much history this reference table + will hold. + - name: snapshot_relation + type: string + description: > + Only required if 'historized' is set to 'snapshot'. Name of the snapshot_v1 model to be used. + - name: snapshot_trigger_column + type: string + description: > + Only required if 'historized' is set to 'snapshot'. Defaults to global variable + 'datavault4dbt.snapshot_trigger_column'. Only needs to be set if alias deviates from global variable. + + - name: snowflake__ref_table + description: Snowflake-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: databricks__ref_table + description: Databricks-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: exasol__ref_table + description: Exasol-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: fabric__ref_table + description: Microsoft Fabric-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: oracle__ref_table + description: Oracle-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: postgres__ref_table + description: PostgreSQL-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: redshift__ref_table + description: Amazon Redshift-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: sqlserver__ref_table + description: SQL Server-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: synapse__ref_table + description: Azure Synapse Analytics-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: trino__ref_table + description: Trino-specific implementation of ref_table. Accepts the same arguments as the parent ref_table macro. + + - name: default__ref_table + description: > + Default implementation of ref_table, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent ref_table macro. diff --git a/macros/tables/sat_v0.md b/macros/tables/sat_v0.md new file mode 100644 index 00000000..299ea23b --- /dev/null +++ b/macros/tables/sat_v0.md @@ -0,0 +1,52 @@ +{% docs sat_v0 %} + +## Satellite v0 + +Creates a standard satellite version 0, materialized as an incremental table. Applied on top of the staging layer, +connected to either a Hub or a Link. Each satellite can only be loaded by one source model, since we typically +recommend a satellite split by source system. + +On top of each version 0 satellite, a version 1 satellite should be created using the sat_v1 macro, which extends +the v0 satellite by a virtually calculated load end date. + +Features: +- Can handle multiple updates per batch, without losing intermediate changes — initial loading is supported. +- Uses a dynamic high-water-mark to optimize loading performance of multiple loads. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.sat_v0( + parent_hashkey='hk_account_h', + src_hashdiff='hd_account_data_sfdc_s', + src_payload=['name', 'address', 'country', 'phone', 'email'], + source_model='stage_account' +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +parent_hashkey: 'hk_account_h' +src_hashdiff: 'hd_account_data_sfdc_s' +src_payload: + - name + - address + - country + - phone + - email +source_model: 'stage_account' +{%- endset -%} + +{{ datavault4dbt.sat_v0(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/sat_v0.yml b/macros/tables/sat_v0.yml new file mode 100644 index 00000000..60e0c4f8 --- /dev/null +++ b/macros/tables/sat_v0.yml @@ -0,0 +1,91 @@ +version: 2 + +macros: + - name: sat_v0 + description: '{{ doc("sat_v0") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: parent_hashkey + type: string + description: > + Name of the hashkey column inside the stage of the object that this satellite is attached to. + Examples: 'hk_account_h' (attached to hub account), 'hk_account_contact_l' (attached to a link). + - name: src_hashdiff + type: string + description: > + Name of the hashdiff column of this satellite, that was created inside the staging area and is + calculated out of the entire payload of this satellite. The stage must hold one hashdiff per + satellite entity. Examples: 'hd_account_data_sfdc_s'. + - name: src_payload + type: list[string] + description: > + A list of all the descriptive attributes that should be included in this satellite. Needs to be the + columns that are fed into the hashdiff calculation of this satellite. + Examples: ['name', 'address', 'country', 'phone', 'email']. + - name: source_model + type: string + description: > + Name of the underlying staging model, must be available inside dbt as a model. + Examples: 'stage_account' — This satellite is loaded out of the stage for account. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: disable_hwm + type: bool + description: > + Whether the High Water Mark should be turned off. Optional, default False. + - name: source_is_single_batch + type: bool + description: > + Whether the source contains only one batch. Optional, default False. + - name: additional_columns + type: any + description: > + Additional columns from source system or derived columns which should be part of the Sat. + The columns need to be available in all source models. Optional parameter, defaults to empty list. + + - name: snowflake__sat_v0 + description: Snowflake-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: databricks__sat_v0 + description: Databricks-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: exasol__sat_v0 + description: Exasol-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: fabric__sat_v0 + description: Microsoft Fabric-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: oracle__sat_v0 + description: Oracle-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: postgres__sat_v0 + description: PostgreSQL-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: redshift__sat_v0 + description: Amazon Redshift-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: sqlserver__sat_v0 + description: SQL Server-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: synapse__sat_v0 + description: Azure Synapse Analytics-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: trino__sat_v0 + description: Trino-specific implementation of sat_v0. Accepts the same arguments as the parent sat_v0 macro. + + - name: default__sat_v0 + description: > + Default implementation of sat_v0, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent sat_v0 macro. diff --git a/macros/tables/sat_v1.md b/macros/tables/sat_v1.md new file mode 100644 index 00000000..9e5bdc41 --- /dev/null +++ b/macros/tables/sat_v1.md @@ -0,0 +1,51 @@ +{% docs sat_v1 %} + +## Satellite v1 + +Calculates a virtualized load end date on top of a version 0 satellite. This column is generated for usage in the +PIT tables, and only virtualized to follow the insert-only approach. A version 1 satellite should be materialized +as a view by default. Usually one version 1 sat would be created for each version 0 sat. + +### Usage + +```jinja +{% raw %} +{{ datavault4dbt.sat_v1( + sat_v0='account_data_sfdc_0_s', + hashkey='hk_account_h', + hashdiff='hd_account_data_sfdc_s' +) }} +{% endraw %} +``` + +### With is_current flag + +```jinja +{% raw %} +{{ datavault4dbt.sat_v1( + sat_v0='account_data_sfdc_0_s', + hashkey='hk_account_h', + hashdiff='hd_account_data_sfdc_s', + add_is_current_flag=true +) }} +{% endraw %} +``` + +### Metadata block usage + +`meta` is a Jinja string variable containing all macro parameters as YAML. Define it using a +`set` block in the model SQL file — the macro parses the YAML string at runtime: + +```jinja +{% raw %} +{%- set meta -%} +sat_v0: 'account_data_sfdc_0_s' +hashkey: 'hk_account_h' +hashdiff: 'hd_account_data_sfdc_s' +{%- endset -%} + +{{ datavault4dbt.sat_v1(yaml_metadata=meta) }} +{% endraw %} +``` + +{% enddocs %} diff --git a/macros/tables/sat_v1.yml b/macros/tables/sat_v1.yml new file mode 100644 index 00000000..db8f0fa4 --- /dev/null +++ b/macros/tables/sat_v1.yml @@ -0,0 +1,90 @@ +version: 2 + +macros: + - name: sat_v1 + description: '{{ doc("sat_v1") }}' + arguments: + - name: yaml_metadata + type: string + description: > + Optional. A YAML-formatted string that defines all parameters in a single block instead + of specifying each one directly in the macro call. + - name: sat_v0 + type: string + description: > + Name of the underlying version 0 satellite. + Examples: 'account_data_sfdc_0_s' — This satellite would be the version 1 satellite of the underlying + version 0 data satellite for account. + - name: hashkey + type: string + description: > + Name of the parent hashkey column inside the version 0 satellite. Would either be the hashkey of a + hub or a link. Needs to be similar to the 'parent_hashkey' parameter inside the sat_v0 model. + Examples: 'hk_account_h' (hub), 'hk_account_contact_l' (link). + - name: hashdiff + type: string + description: > + Name of the hashdiff column inside the underlying version 0 satellite. Needs to be similar to the + 'src_hashdiff' parameter inside the sat_v0 model. + Examples: 'hd_account_data_sfdc_s'. + - name: src_ldts + type: string + description: > + Name of the ldts column inside the source models. Is optional, will use the global variable 'datavault4dbt.ldts_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: src_rsrc + type: string + description: > + Name of the rsrc column inside the source models. Is optional, will use the global variable 'datavault4dbt.rsrc_alias'. + Needs to use the same column name as defined as alias inside the staging model. + - name: ledts_alias + type: string + description: > + Desired alias for the load end date column. Is optional, will use the global variable 'datavault4dbt.ledts_alias' + if not set here. + - name: add_is_current_flag + type: bool + description: > + Optional parameter to add a new column to the v1 sat based on the load end date timestamp (ledts). Default is false. + If set to true it will add this is_current flag to the v1 sat. For each record this column will be set to true if + the load end date timestamp is equal to the variable end of all times. If not, the record is not current and will + be set to false. + - name: include_payload + type: bool + description: > + Optional parameter to specify if the v1 sat should have the payload columns from sat v0 or not. Default is true. + + - name: snowflake__sat_v1 + description: Snowflake-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: databricks__sat_v1 + description: Databricks-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: exasol__sat_v1 + description: Exasol-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: fabric__sat_v1 + description: Microsoft Fabric-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: oracle__sat_v1 + description: Oracle-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: postgres__sat_v1 + description: PostgreSQL-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: redshift__sat_v1 + description: Amazon Redshift-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: sqlserver__sat_v1 + description: SQL Server-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: synapse__sat_v1 + description: Azure Synapse Analytics-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: trino__sat_v1 + description: Trino-specific implementation of sat_v1. Accepts the same arguments as the parent sat_v1 macro. + + - name: default__sat_v1 + description: > + Default implementation of sat_v1, co-located in the BigQuery adapter folder as BigQuery uses this fallback + via adapter.dispatch(). Accepts the same arguments as the parent sat_v1 macro.