feat: make the hash input cast datatype configurable per adapter

[tkirschke]

Context

A user profiling a slow full refresh on MS SQL Server reported that changing the VARCHAR(MAX) casts in hash_standardization.sql to VARCHAR(8000) took a single satellite load from over a minute to a few seconds.

The report holds up. VARCHAR(MAX) is a large-value type: stored off-row, not handled in memory like a regular VARCHAR(n), and it blocks several optimizations for the REPLACE() / UPPER() / HASHBYTES() chain wrapped around it. There was no design reason for MAX specifically — Synapse and Fabric, which share the T-SQL codepath, already used VARCHAR(4000). SQL Server was the outlier.

Changes

Two casts happen before hashing, and they are now configurable independently:

Variable	Applies to
datavault4dbt.hash_input_attribute_dtype	the cast of one single column in attribute_standardise
datavault4dbt.hash_input_concat_dtype	the cast of the fully concatenated payload in concattenated_standardise

Both are adapter-keyed mapping dictionaries following the datavault4dbt.beginning_of_all_times convention, resolved by the new macros/supporting/hash_input_dtype.sql. That macro follows the lighter first_day_of_week.sql pattern (single default__ implementation using target.type) rather than the 11-macro-per-adapter string_default_dtype.sql pattern.

The current hardcoded values are kept as defaults:

	attribute	concat
bigquery / snowflake / databricks	STRING	STRING
exasol	VARCHAR(20000) UTF8	VARCHAR(2000000) UTF8
postgres / trino	VARCHAR	VARCHAR
redshift	(no cast, see below)	VARCHAR
synapse / fabric	VARCHAR(4000)	VARCHAR(4000)
oracle	VARCHAR2(2000)	VARCHAR2(2000)
sqlserver	VARCHAR(MAX)	VARCHAR(MAX)

Multi Active Satellites are deliberately excluded

multi_active_concattenated_standardise keeps its hardcoded datatype for all adapters and is untouched by this PR.

The reason is STRING_AGG(), which aggregates the payload of all active records of one group before hashing. On SQL Server it only returns VARCHAR(MAX) if its input expression is VARCHAR(MAX); with a shorter input it returns VARCHAR(8000) and raises an error once a group's aggregate exceeds 8000 bytes. Since that limit applies to a whole group rather than a single record, it is far easier to hit than the per-record limit of a regular satellite.

Excluding MA sats gives the property we want: a user can shorten the cast for hubs, links and regular satellites and get the speedup, with no way to accidentally break their MA sats through a global variable.

Truncation caveat

A datatype shorter than the actual hash input truncates it silently on most adapters. Truncated input changes the resulting hashes, and two rows differing only past the truncation point collapse into the same hashkey or hashdiff. Any later change to these variables is a full reload of the affected entities. This is called out on the global variables page and in the SQL Server adapter notes.

Verification

No dbt profile was available, so this was verified statically:

• Every standardise_prefix / standardise_suffix / expr expression was extracted from both main and this branch, evaluated with the shipped defaults, and compared — all 33 macros render byte-identical to main. The generated SQL does not change for anyone who does not override the variables.
• The 11 multi_active blocks are textually identical to main, not just render-equivalent.
• All 223 format expressions evaluate without arity errors.
• dbt_project.yml parses as YAML; both macro files pass a Jinja parse.
• hash_input_dtype was render-tested across all branches: dict hit, dict missing the own adapter (warning + fallback), scalar override, and variable absent entirely.

Note

redshift__attribute_standardise has no cast at all — it trims the raw expression. Rather than introduce one and change Redshift behavior, it was left as is, so redshift has no key in hash_input_attribute_dtype. The macro is never called with type='attribute' on Redshift, so no warning fires in practice.

🤖 Generated with Claude Code