Skip to content

Commit 57a32f7

Browse files
georgesittastreysp
andauthored
Chore: update docs on @this_model (#4599)
Co-authored-by: Trey Spiller <1831878+treysp@users.noreply.github.com>
1 parent 2f8e3b7 commit 57a32f7

7 files changed

Lines changed: 98 additions & 14 deletions

File tree

docs/concepts/macros/macro_variables.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,11 +130,21 @@ SQLMesh provides additional predefined variables used to modify model behavior b
130130
* 'auditing' - The audit is being run.
131131
* 'testing' - The model query logic is being evaluated in the context of a unit test.
132132
* @gateway - A string value containing the name of the current [gateway](../../guides/connections.md).
133-
* @this_model - A string value containing the name of the physical table the model view selects from. Typically used to create [generic audits](../audits.md#generic-audits). In the case of [on_virtual_update statements](../models/sql_models.md#optional-on-virtual-update-statements) it contains the qualified view name instead.
134-
* Can be used in model definitions when SQLGlot cannot fully parse a statement and you need to reference the model's underlying physical table directly.
135-
* Can be passed as an argument to macros that access or interact with the underlying physical table.
133+
* @this_model - The physical table name that the model's view selects from. Typically used to create [generic audits](../audits.md#generic-audits). When used in [on_virtual_update statements](../models/sql_models.md#optional-on-virtual-update-statements), it contains the qualified view name instead.
136134
* @model_kind_name - A string value containing the name of the current model kind. Intended to be used in scenarios where you need to control the [physical properties in model defaults](../../reference/model_configuration.md#model-defaults).
137135

136+
!!! note "Embedding variables in strings"
137+
138+
Macro variable references sometimes use the curly brace syntax `@{variable}`, which serves a different purpose than the regular `@variable` syntax.
139+
140+
The curly brace syntax tells SQLMesh that the rendered string should be treated as an identifier, instead of simply replacing the macro variable value.
141+
142+
For example, if `variable` is defined as `@DEF(`variable`, foo.bar)`, then `@variable` produces `foo.bar`, while `@{variable}` produces `"foo.bar"`. This is because SQLMesh converts `foo.bar` into an identifier, using double quotes to correctly include the `.` character in the identifier name.
143+
144+
In practice, `@{variable}` is most commonly used to interpolate a value within an identifier, e.g., `@{variable}_suffix`, whereas `@variable` is used to do plain substitutions for string literals.
145+
146+
Learn more [above](#embedding-variables-in-strings).
147+
138148
#### Before all and after all variables
139149

140150
The following variables are also available in [`before_all` and `after_all` statements](../../guides/configuration.md#before_all-and-after_all-statements), as well as in macros invoked within them.

docs/concepts/macros/sqlmesh_macros.md

Lines changed: 65 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,59 @@ It uses the following five step approach to accomplish this:
3838

3939
5. Modify the semantic representation of the SQL query with the substituted variable values from (3) and functions from (4).
4040

41+
### Embedding variables in strings
42+
43+
SQLMesh always incorporates macro variable values into the semantic representation of a SQL query (step 5 above). To do that, it infers the role each macro variable value plays in the query.
44+
45+
For context, two commonly used types of string in SQL are:
46+
47+
- String literals, which represent text values and are surrounded by single quotes, such as `'the_string'`
48+
- Identifiers, which reference database objects like column, table, alias, and function names
49+
- They may be unquoted or quoted with double quotes, backticks, or brackets, depending on the SQL dialect
50+
51+
In a normal query, SQLMesh can easily determine which role a given string is playing. However, it is more difficult if a macro variable is embedded directly into a string - especially if the string is in the `MODEL` block (and not the query itself).
52+
53+
For example, consider a project that defines a [gateway variable](#gateway-variables) named `gateway_var`. The project includes a model that references `@gateway_var` as part of the schema in the model's `name`, which is a SQL *identifier*.
54+
55+
This is how we might try to write the model:
56+
57+
``` sql title="Incorrectly rendered to string literal"
58+
MODEL (
59+
name the_@gateway_var_schema.table
60+
);
61+
```
62+
63+
From SQLMesh's perspective, the model schema is the combination of three sub-strings: `the_`, the value of `@gateway_var`, and `_schema`.
64+
65+
SQLMesh will concatenate those strings, but it does not have the context to know that it is building a SQL identifier and will return a string literal.
66+
67+
To provide the context SQLMesh needs, you must add curly braces to the macro variable reference: `@{gateway_var}` instead of `@gateway_var`:
68+
69+
``` sql title="Correctly rendered to identifier"
70+
MODEL (
71+
name the_@{gateway_var}_schema.table
72+
);
73+
```
74+
75+
The curly braces let SQLMesh know that it should treat the string as a SQL identifier, which it will then quote based on the SQL dialect's quoting rules.
76+
77+
The most common use of the curly brace syntax is embedding macro variables into strings, it can also be used to differentiate string literals and identifiers in SQL queries. For example, consider a macro variable `my_variable` whose value is `col`.
78+
79+
If we `SELECT` this value with regular macro syntax, it will render to a string literal:
80+
81+
``` sql
82+
SELECT @my_variable AS the_column; -- renders to SELECT 'col' AS the_column
83+
```
84+
85+
`'col'` is surrounded with single quotes, and the SQL engine will use that string as the column's data value.
86+
87+
If we use curly braces, SQLMesh will know that we want to use the rendered string as an identifier:
88+
89+
``` sql
90+
SELECT @{my_variable} AS the_column; -- renders to SELECT col AS the_column
91+
```
92+
93+
`col` is not surrounded with single quotes, and the SQL engine will determine that the query is referencing a column or other object named `col`.
4194

4295
## User-defined variables
4396

@@ -174,6 +227,8 @@ SELECT
174227
FROM @customer.some_source
175228
```
176229

230+
Note the use of both regular `@field_a` and curly brace syntax `@{field_b}` macro variable references in the model query. Learn more [above](#embedding-variables-in-strings)
231+
177232
Blueprint variables can be accessed using the syntax shown above, or through the `@BLUEPRINT_VAR()` macro function, which also supports specifying default values in case the variable is undefined (similar to `@VAR()`).
178233

179234
### Local variables
@@ -448,7 +503,13 @@ FROM table
448503

449504
This syntax works regardless of whether the array values are quoted or not.
450505

451-
NOTE: SQLMesh macros support placing macro values at the end of a column name simply using `column_@x`. However if you wish to substitute the variable anywhere else in the identifier, you need to use the more explicit substitution syntax `@{}`. This avoids ambiguity. These are valid uses: `@{x}_column` or `my_@{x}_column`.
506+
!!! note "Embedding macros in strings"
507+
508+
SQLMesh macros support placing macro values at the end of a column name using `column_@x`.
509+
510+
However, if you wish to substitute the variable anywhere else in the identifier, you need to use the more explicit curly brace syntax `@{}` to avoid ambiguity. For example, these are valid uses: `@{x}_column` or `my_@{x}_column`.
511+
512+
Learn more about embedding macros in strings [above](#embedding-variables-in-strings)
452513

453514
### @IF
454515

@@ -1087,7 +1148,9 @@ The `template` can contain the following placeholders that will be substituted:
10871148
- `@{schema_name}` - The name of the physical schema that SQLMesh is using for the model version table, eg `sqlmesh__landing`
10881149
- `@{table_name}` - The name of the physical table that SQLMesh is using for the model version, eg `landing__customers__2517971505`
10891150

1090-
It can be used in a `MODEL` block:
1151+
Note the use of the curly brace syntax `@{}` in the template placeholders - learn more [above](#embedding-variables-in-strings).
1152+
1153+
The `@resolve_template` macro can be used in a `MODEL` block:
10911154

10921155
```sql linenums="1" hl_lines="5"
10931156
MODEL (

docs/concepts/models/external_models.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,9 @@ FROM
7070
@{gateway}_db.external_table;
7171
```
7272

73-
This table will be named differently depending on which `--gateway` SQLMesh is run with. For example:
73+
This table will be named differently depending on which `--gateway` SQLMesh is run with (learn more about the curly brace `@{gateway}` syntax [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings)).
74+
75+
For example:
7476

7577
- `sqlmesh --gateway dev plan` - SQLMesh will try to query `dev_db.external_table`
7678
- `sqlmesh --gateway prod plan` - SQLMesh will try to query `prod_db.external_table`

docs/concepts/models/python_models.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,8 @@ def entrypoint(
365365
)
366366
```
367367

368+
Note the use of curly brace syntax `@{customer}` in the model name above. It is used to ensure SQLMesh can combine the macro variable into the model name identifier correctly - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
369+
368370
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints="@gen_blueprints()"`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.
369371

370372
For example, the definition of the `gen_blueprints` may look like this:

docs/concepts/models/sql_models.md

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ Note that the SQL command `UNCACHE TABLE countries` inside the `@IF()` macro doe
9595

9696
The optional on-virtual-update statements allow you to execute SQL commands after the completion of the [Virtual Update](#virtual-update).
9797

98-
These can be used, for example, to grant privileges on views of the virtual layer.
98+
These can be used, for example, to grant privileges on views of the virtual layer.
9999

100100
These SQL statements must be enclosed within an `ON_VIRTUAL_UPDATE_BEGIN;` ...; `ON_VIRTUAL_UPDATE_END;` block like this:
101101

@@ -109,11 +109,11 @@ SELECT
109109
r.id::INT
110110
FROM raw.restaurants AS r;
111111

112-
ON_VIRTUAL_UPDATE_BEGIN;
112+
ON_VIRTUAL_UPDATE_BEGIN;
113113
GRANT SELECT ON VIEW @this_model TO ROLE role_name;
114-
JINJA_STATEMENT_BEGIN;
114+
JINJA_STATEMENT_BEGIN;
115115
GRANT SELECT ON VIEW {{ this_model }} TO ROLE admin;
116-
JINJA_END;
116+
JINJA_END;
117117
ON_VIRTUAL_UPDATE_END;
118118
```
119119

@@ -175,6 +175,10 @@ SELECT
175175
FROM customer2.some_source
176176
```
177177

178+
Note the use of curly brace syntax `@{field_b} AS field_b` in the model query above. It is used to tell SQLMesh that the rendered variable value should be treated as a SQL identifier instead of a string literal.
179+
180+
You can see the different behavior in the first rendered model. `@field_a` is resolved to the string literal `'x'` (with single quotes) and `@{field_b}` is resolved to the identifier `y` (without quotes). Learn more about the curly brace syntax [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
181+
178182
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints @gen_blueprints()`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.
179183

180184
For example, the definition of the `gen_blueprints` may look like this:
@@ -249,7 +253,7 @@ One could also define this model by simply returning a string that contained the
249253

250254
The `@model` decorator is the Python equivalent of the `MODEL` DDL.
251255

252-
In addition to model metadata and configuration information, one can also set the keyword arguments `pre_statements`, `post_statements` and `on_virtual_update` to a list of SQL strings and/or SQLGlot expressions to define the pre/post-statements and on-virtual-update-statements of the model, respectively.
256+
In addition to model metadata and configuration information, one can also set the keyword arguments `pre_statements`, `post_statements` and `on_virtual_update` to a list of SQL strings and/or SQLGlot expressions to define the pre/post-statements and on-virtual-update-statements of the model, respectively.
253257

254258
!!! note
255259

docs/guides/isolated_systems.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,14 +70,16 @@ MODEL (
7070
)
7171
```
7272

73-
To embed the gateway name directly in the schema name, use the `@{gateway}` syntax:
73+
To embed the gateway name directly in the schema name, use the curly brace `@{gateway}` syntax:
7474

7575
```sql linenums="1"
7676
MODEL (
7777
name @{gateway}_schema.my_model
7878
)
7979
```
8080

81+
Learn more about the curly brace `@{}` syntax [here](../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
82+
8183
## Workflow
8284

8385
### Linking systems

docs/integrations/engines/trino.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,12 +178,13 @@ This would perform the following mappings:
178178
!!! info "Placeholders"
179179
You may use the `@{catalog_name}` and `@{schema_name}` placeholders in the mapping value.
180180

181-
If there is a match on one of the patterns then the catalog / schema that SQLMesh is about to use in the `CREATE SCHEMA` statement will be subsitituted into these placeholders.
181+
If there is a match on one of the patterns then the catalog / schema that SQLMesh is about to use in the `CREATE SCHEMA` statement will be substituted into these placeholders.
182182

183+
Note the use of curly brace syntax `@{}` when referencing these placeholders - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
183184

184185
#### Tables
185186

186-
Often, you dont need to configure an explicit table location because if you have configured explicit schema locations, table locations are automatically inferred by Trino to be a subdirectory under the schema location.
187+
Often, you don't need to configure an explicit table location because if you have configured explicit schema locations, table locations are automatically inferred by Trino to be a subdirectory under the schema location.
187188

188189
However, if you need to, you can configure an explicit table location by adding a `location` property to the model `physical_properties`.
189190

0 commit comments

Comments
 (0)