You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/macros/macro_variables.md
+13-3Lines changed: 13 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -130,11 +130,21 @@ SQLMesh provides additional predefined variables used to modify model behavior b
130
130
* 'auditing' - The audit is being run.
131
131
* 'testing' - The model query logic is being evaluated in the context of a unit test.
132
132
*@gateway - A string value containing the name of the current [gateway](../../guides/connections.md).
133
-
* @this_model - A string value containing the name of the physical table the model view selects from. Typically used to create [generic audits](../audits.md#generic-audits). In the case of [on_virtual_update statements](../models/sql_models.md#optional-on-virtual-update-statements) it contains the qualified view name instead.
134
-
* Can be used in model definitions when SQLGlot cannot fully parse a statement and you need to reference the model's underlying physical table directly.
135
-
* Can be passed as an argument to macros that access or interact with the underlying physical table.
133
+
* @this_model - The physical table name that the model's view selects from. Typically used to create [generic audits](../audits.md#generic-audits). When used in [on_virtual_update statements](../models/sql_models.md#optional-on-virtual-update-statements), it contains the qualified view name instead.
136
134
* @model_kind_name - A string value containing the name of the current model kind. Intended to be used in scenarios where you need to control the [physical properties in model defaults](../../reference/model_configuration.md#model-defaults).
137
135
136
+
!!! note "Embedding variables in strings"
137
+
138
+
Macro variable references sometimes use the curly brace syntax `@{variable}`, which serves a different purpose than the regular `@variable` syntax.
139
+
140
+
The curly brace syntax tells SQLMesh that the rendered string should be treated as an identifier, instead of simply replacing the macro variable value.
141
+
142
+
For example, if `variable` is defined as `@DEF(`variable`, foo.bar)`, then `@variable` produces `foo.bar`, while `@{variable}` produces `"foo.bar"`. This is because SQLMesh converts `foo.bar` into an identifier, using double quotes to correctly include the `.` character in the identifier name.
143
+
144
+
In practice, `@{variable}` is most commonly used to interpolate a value within an identifier, e.g., `@{variable}_suffix`, whereas `@variable` is used to do plain substitutions for string literals.
145
+
146
+
Learn more [above](#embedding-variables-in-strings).
147
+
138
148
#### Before all and after all variables
139
149
140
150
The following variables are also available in [`before_all` and `after_all` statements](../../guides/configuration.md#before_all-and-after_all-statements), as well as in macros invoked within them.
Copy file name to clipboardExpand all lines: docs/concepts/macros/sqlmesh_macros.md
+65-2Lines changed: 65 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,6 +38,59 @@ It uses the following five step approach to accomplish this:
38
38
39
39
5. Modify the semantic representation of the SQL query with the substituted variable values from (3) and functions from (4).
40
40
41
+
### Embedding variables in strings
42
+
43
+
SQLMesh always incorporates macro variable values into the semantic representation of a SQL query (step 5 above). To do that, it infers the role each macro variable value plays in the query.
44
+
45
+
For context, two commonly used types of string in SQL are:
46
+
47
+
- String literals, which represent text values and are surrounded by single quotes, such as `'the_string'`
48
+
- Identifiers, which reference database objects like column, table, alias, and function names
49
+
- They may be unquoted or quoted with double quotes, backticks, or brackets, depending on the SQL dialect
50
+
51
+
In a normal query, SQLMesh can easily determine which role a given string is playing. However, it is more difficult if a macro variable is embedded directly into a string - especially if the string is in the `MODEL` block (and not the query itself).
52
+
53
+
For example, consider a project that defines a [gateway variable](#gateway-variables) named `gateway_var`. The project includes a model that references `@gateway_var` as part of the schema in the model's `name`, which is a SQL *identifier*.
54
+
55
+
This is how we might try to write the model:
56
+
57
+
```sql title="Incorrectly rendered to string literal"
58
+
MODEL (
59
+
name the_@gateway_var_schema.table
60
+
);
61
+
```
62
+
63
+
From SQLMesh's perspective, the model schema is the combination of three sub-strings: `the_`, the value of `@gateway_var`, and `_schema`.
64
+
65
+
SQLMesh will concatenate those strings, but it does not have the context to know that it is building a SQL identifier and will return a string literal.
66
+
67
+
To provide the context SQLMesh needs, you must add curly braces to the macro variable reference: `@{gateway_var}` instead of `@gateway_var`:
68
+
69
+
```sql title="Correctly rendered to identifier"
70
+
MODEL (
71
+
name the_@{gateway_var}_schema.table
72
+
);
73
+
```
74
+
75
+
The curly braces let SQLMesh know that it should treat the string as a SQL identifier, which it will then quote based on the SQL dialect's quoting rules.
76
+
77
+
The most common use of the curly brace syntax is embedding macro variables into strings, it can also be used to differentiate string literals and identifiers in SQL queries. For example, consider a macro variable `my_variable` whose value is `col`.
78
+
79
+
If we `SELECT` this value with regular macro syntax, it will render to a string literal:
80
+
81
+
```sql
82
+
SELECT @my_variable AS the_column; -- renders to SELECT 'col' AS the_column
83
+
```
84
+
85
+
`'col'` is surrounded with single quotes, and the SQL engine will use that string as the column's data value.
86
+
87
+
If we use curly braces, SQLMesh will know that we want to use the rendered string as an identifier:
88
+
89
+
```sql
90
+
SELECT @{my_variable} AS the_column; -- renders to SELECT col AS the_column
91
+
```
92
+
93
+
`col` is not surrounded with single quotes, and the SQL engine will determine that the query is referencing a column or other object named `col`.
41
94
42
95
## User-defined variables
43
96
@@ -174,6 +227,8 @@ SELECT
174
227
FROM @customer.some_source
175
228
```
176
229
230
+
Note the use of both regular `@field_a` and curly brace syntax `@{field_b}` macro variable references in the model query. Learn more [above](#embedding-variables-in-strings)
231
+
177
232
Blueprint variables can be accessed using the syntax shown above, or through the `@BLUEPRINT_VAR()` macro function, which also supports specifying default values in case the variable is undefined (similar to `@VAR()`).
178
233
179
234
### Local variables
@@ -448,7 +503,13 @@ FROM table
448
503
449
504
This syntax works regardless of whether the array values are quoted or not.
450
505
451
-
NOTE: SQLMesh macros support placing macro values at the end of a column name simply using `column_@x`. However if you wish to substitute the variable anywhere else in the identifier, you need to use the more explicit substitution syntax `@{}`. This avoids ambiguity. These are valid uses: `@{x}_column` or `my_@{x}_column`.
506
+
!!! note "Embedding macros in strings"
507
+
508
+
SQLMesh macros support placing macro values at the end of a column name using `column_@x`.
509
+
510
+
However, if you wish to substitute the variable anywhere else in the identifier, you need to use the more explicit curly brace syntax `@{}` to avoid ambiguity. For example, these are valid uses: `@{x}_column` or `my_@{x}_column`.
511
+
512
+
Learn more about embedding macros in strings [above](#embedding-variables-in-strings)
452
513
453
514
### @IF
454
515
@@ -1087,7 +1148,9 @@ The `template` can contain the following placeholders that will be substituted:
1087
1148
-`@{schema_name}` - The name of the physical schema that SQLMesh is using for the model version table, eg `sqlmesh__landing`
1088
1149
-`@{table_name}` - The name of the physical table that SQLMesh is using for the model version, eg `landing__customers__2517971505`
1089
1150
1090
-
It can be used in a `MODEL` block:
1151
+
Note the use of the curly brace syntax `@{}` in the template placeholders - learn more [above](#embedding-variables-in-strings).
1152
+
1153
+
The `@resolve_template` macro can be used in a `MODEL` block:
Copy file name to clipboardExpand all lines: docs/concepts/models/external_models.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,7 +70,9 @@ FROM
70
70
@{gateway}_db.external_table;
71
71
```
72
72
73
-
This table will be named differently depending on which `--gateway` SQLMesh is run with. For example:
73
+
This table will be named differently depending on which `--gateway` SQLMesh is run with (learn more about the curly brace `@{gateway}` syntax [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings)).
74
+
75
+
For example:
74
76
75
77
-`sqlmesh --gateway dev plan` - SQLMesh will try to query `dev_db.external_table`
76
78
-`sqlmesh --gateway prod plan` - SQLMesh will try to query `prod_db.external_table`
Copy file name to clipboardExpand all lines: docs/concepts/models/python_models.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -365,6 +365,8 @@ def entrypoint(
365
365
)
366
366
```
367
367
368
+
Note the use of curly brace syntax `@{customer}` in the model name above. It is used to ensure SQLMesh can combine the macro variable into the model name identifier correctly - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
369
+
368
370
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints="@gen_blueprints()"`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.
369
371
370
372
For example, the definition of the `gen_blueprints` may look like this:
Copy file name to clipboardExpand all lines: docs/concepts/models/sql_models.md
+9-5Lines changed: 9 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -95,7 +95,7 @@ Note that the SQL command `UNCACHE TABLE countries` inside the `@IF()` macro doe
95
95
96
96
The optional on-virtual-update statements allow you to execute SQL commands after the completion of the [Virtual Update](#virtual-update).
97
97
98
-
These can be used, for example, to grant privileges on views of the virtual layer.
98
+
These can be used, for example, to grant privileges on views of the virtual layer.
99
99
100
100
These SQL statements must be enclosed within an `ON_VIRTUAL_UPDATE_BEGIN;` ...; `ON_VIRTUAL_UPDATE_END;` block like this:
101
101
@@ -109,11 +109,11 @@ SELECT
109
109
r.id::INT
110
110
FROMraw.restaurantsAS r;
111
111
112
-
ON_VIRTUAL_UPDATE_BEGIN;
112
+
ON_VIRTUAL_UPDATE_BEGIN;
113
113
GRANTSELECTON VIEW @this_model TO ROLE role_name;
114
-
JINJA_STATEMENT_BEGIN;
114
+
JINJA_STATEMENT_BEGIN;
115
115
GRANTSELECTON VIEW {{ this_model }} TO ROLE admin;
116
-
JINJA_END;
116
+
JINJA_END;
117
117
ON_VIRTUAL_UPDATE_END;
118
118
```
119
119
@@ -175,6 +175,10 @@ SELECT
175
175
FROMcustomer2.some_source
176
176
```
177
177
178
+
Note the use of curly brace syntax `@{field_b} AS field_b` in the model query above. It is used to tell SQLMesh that the rendered variable value should be treated as a SQL identifier instead of a string literal.
179
+
180
+
You can see the different behavior in the first rendered model. `@field_a` is resolved to the string literal `'x'` (with single quotes) and `@{field_b}` is resolved to the identifier `y` (without quotes). Learn more about the curly brace syntax [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
181
+
178
182
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints @gen_blueprints()`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.
179
183
180
184
For example, the definition of the `gen_blueprints` may look like this:
@@ -249,7 +253,7 @@ One could also define this model by simply returning a string that contained the
249
253
250
254
The `@model` decorator is the Python equivalent of the `MODEL` DDL.
251
255
252
-
In addition to model metadata and configuration information, one can also set the keyword arguments `pre_statements`, `post_statements` and `on_virtual_update` to a list of SQL strings and/or SQLGlot expressions to define the pre/post-statements and on-virtual-update-statements of the model, respectively.
256
+
In addition to model metadata and configuration information, one can also set the keyword arguments `pre_statements`, `post_statements` and `on_virtual_update` to a list of SQL strings and/or SQLGlot expressions to define the pre/post-statements and on-virtual-update-statements of the model, respectively.
Copy file name to clipboardExpand all lines: docs/integrations/engines/trino.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -178,12 +178,13 @@ This would perform the following mappings:
178
178
!!! info "Placeholders"
179
179
You may use the `@{catalog_name}` and `@{schema_name}` placeholders in the mapping value.
180
180
181
-
If there is a match on one of the patterns then the catalog / schema that SQLMesh is about to use in the `CREATE SCHEMA` statement will be subsitituted into these placeholders.
181
+
If there is a match on one of the patterns then the catalog / schema that SQLMesh is about to use in the `CREATE SCHEMA` statement will be substituted into these placeholders.
182
182
183
+
Note the use of curly brace syntax `@{}` when referencing these placeholders - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
183
184
184
185
#### Tables
185
186
186
-
Often, you dont need to configure an explicit table location because if you have configured explicit schema locations, table locations are automatically inferred by Trino to be a subdirectory under the schema location.
187
+
Often, you don't need to configure an explicit table location because if you have configured explicit schema locations, table locations are automatically inferred by Trino to be a subdirectory under the schema location.
187
188
188
189
However, if you need to, you can configure an explicit table location by adding a `location` property to the model `physical_properties`.
0 commit comments