From 942d437036e3b8fed80a61bbbacf7206ec1b6209 Mon Sep 17 00:00:00 2001 From: Trey Spiller Date: Fri, 13 Jun 2025 18:04:20 -0500 Subject: [PATCH 1/7] Add interactive init to quickstart docs --- docs/integrations/overview.md | 32 ++--- docs/quickstart/cli.md | 233 +++++++++++++++++++++++++++------- docs/reference/cli.md | 10 +- 3 files changed, 209 insertions(+), 66 deletions(-) diff --git a/docs/integrations/overview.md b/docs/integrations/overview.md index 9f829ceab7..be7606cec0 100644 --- a/docs/integrations/overview.md +++ b/docs/integrations/overview.md @@ -9,20 +9,20 @@ SQLMesh supports integrations with the following tools: * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli) ## Execution engines -SQLMesh supports the following execution engines for running SQLMesh projects: +SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses): -* [Athena](./engines/athena.md) -* [Azure SQL](./engines/azuresql.md) -* [BigQuery](./engines/bigquery.md) -* [ClickHouse](./engines/clickhouse.md) -* [Databricks](./engines/databricks.md) -* [DuckDB](./engines/duckdb.md) -* [MotherDuck](./engines/motherduck.md) -* [MSSQL](./engines/mssql.md) -* [MySQL](./engines/mysql.md) -* [Postgres](./engines/postgres.md) -* [GCP Postgres](./engines/gcp-postgres.md) -* [Redshift](./engines/redshift.md) -* [Snowflake](./engines/snowflake.md) -* [Spark](./engines/spark.md) -* [Trino](./engines/trino.md) +* [Athena](./engines/athena.md) (athena) +* [Azure SQL](./engines/azuresql.md) (azuresql) +* [BigQuery](./engines/bigquery.md) (bigquery) +* [ClickHouse](./engines/clickhouse.md) (clickhouse) +* [Databricks](./engines/databricks.md) (databricks) +* [DuckDB](./engines/duckdb.md) (duckdb) +* [MotherDuck](./engines/motherduck.md) (motherduck) +* [MSSQL](./engines/mssql.md) (mssql) +* [MySQL](./engines/mysql.md) (mysql) +* [Postgres](./engines/postgres.md) (postgres) +* [GCP Postgres](./engines/gcp-postgres.md) (gcppostgres) +* [Redshift](./engines/redshift.md) (redshift) +* [Snowflake](./engines/snowflake.md) (snowflake) +* [Spark](./engines/spark.md) (spark) +* [Trino](./engines/trino.md) (trino) diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index cf990eb704..2bff176787 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -1,6 +1,8 @@ # CLI -In this quickstart, you'll use the SQLMesh command line interface (CLI) to get up and running with SQLMesh's scaffold generator. This example project will run locally on your computer using [DuckDB](https://duckdb.org/) as an embedded SQL engine. +In this quickstart, you'll use the SQLMesh command line interface (CLI) to get up and running with SQLMesh's scaffold generator. + +It will create an example project that runs locally on your computer using [DuckDB](https://duckdb.org/) as an embedded SQL engine. Before beginning, ensure that you meet all the [prerequisites](../prerequisites.md) for using SQLMesh. @@ -39,36 +41,173 @@ mkdir sqlmesh-example cd sqlmesh-example ``` -If using a python virtual environment, ensure it's activated first by running the `source .env/bin/activate` command from the folder used during [installation](../installation.md). +If using a Python virtual environment, ensure it's activated first by running the `source .env/bin/activate` command from the folder used during [installation](../installation.md). + +### 1.1 Initialize the project + +SQLMesh includes a scaffold generator to initialize a new SQLMesh project. + +The scaffold generator will ask you some questions and create a SQLMesh configuration file based on your responses. -Create a SQLMesh scaffold with the following command, specifying a default SQL dialect for your models. The dialect should correspond to the dialect most of your models are written in; it can be overridden for specific models in the model's `MODEL` specification. All SQL dialects [supported by the SQLGlot library](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py) are allowed. +Depending on your answers, it will also create multiple files for the SQLmesh example project used in this quickstart. -In this example, we specify the `duckdb` dialect: +Start the scaffold generator by executing the `sqlmesh init` command: ```bash -sqlmesh init duckdb +sqlmesh init ``` -The scaffold will include a SQLMesh configuration file for the example project. +??? info "Skip the questions" + + If you don't want to use the interactive scaffold generator, you can initialize your project with arguments to the [`sqlmesh init` command](../reference/cli.md#init). + + The only required argument is `engine`, which specifies the SQL engine your project will use. Specify one of the engine `type`s in the [list of supported engines](../integrations/overview.md#execution-engines). + + In this example, we specify the `duckdb` engine: + + ```bash + sqlmesh init duckdb + ``` + + The scaffold will include a SQLMesh configuration file and example project directories and files. You're now ready to continue the quickstart [below](#2-create-a-prod-environment). + +#### Project type + +The first question asks about the type of project you want to create. Enter the number corresponding to the type of project you want to create and press `Enter`. + +``` bash +────────────────────────────── +Welcome to SQLMesh! +────────────────────────────── + +What type of project do you want to set up? + + [1] DEFAULT - Create SQLMesh example project models and files + [2] dbt - You have an existing dbt project and want to run it with SQLMesh + [3] EMPTY - Create a SQLMesh configuration file and project directories only + +Enter a number: 1 +``` + +For this quickstart, choose the `DEFAULT` option `1` so the example project files are included in the project directories. + +#### SQL engine + +The second question asks which SQL engine your project will use. SQLMesh will include that engine's connection settings in the configuration file, which you will fill in later to connect your project to the engine. + +For this quickstart, choose the `DuckDB` option `1` so we can run the example project with the built-in DuckDB engine that doesn't need additional configuration. + +``` bash +Choose your SQL engine: + + [1] DuckDB + [2] Snowflake + [3] Databricks + [4] BigQuery + [5] MotherDuck + [6] ClickHouse + [7] Redshift + [8] Spark + [9] Trino + [10] Azure SQL + [11] MSSQL + [12] Postgres + [13] GCP Postgres + [14] MySQL + [15] Athena + [16] RisingWave + +Enter a number: 1 +``` + +#### CLI mode + +SQLMesh's core commands have multiple options that alter their behavior. Some of those options streamline the SQLMesh `plan` workflow and CLI output. + +If you prefer a streamlined workflow, choose the `FLOW` CLI mode to automatically include those options in your project configuration file. If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode. + +``` bash +Choose your SQLMesh CLI experience: + + [1] DEFAULT - See and control every detail + [2] FLOW - Automatically run changes and show summary output + +Enter a number: 1 +``` + +#### Ready to go + +Your project is now ready to go, and SQLMesh displays a message with some good next steps. + +If you chose the DuckDB engine, you're ready to move forward and run the example project with DuckDB. + +If you chose a different engine, add your engine's connection information to the `config.yaml` file before you run any additional SQLMesh commands. + +``` bash +Your SQLMesh project is ready! + +Next steps: +• Update your gateway connection settings (e.g., username/password) in the project configuration file: + /Users/trey/tobiko/sqlmesh/sqlmesh-example-fresh/test/config.yaml +• Run command in CLI: sqlmesh plan +• (Optional) Explain a plan: sqlmesh plan --explain + +Quickstart guide: +https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/ + +Need help? +• Docs: https://sqlmesh.readthedocs.io +• Slack: https://www.tobikodata.com/slack +• GitHub: https://github.com/TobikoData/sqlmesh/issues +``` ??? info "Learn more about the project's configuration" SQLMesh project-level configuration parameters are specified in the `config.yaml` file in the project directory. - This example project uses the embedded DuckDB SQL engine, so its configuration specifies `duckdb` as the local gateway's connection and the `local` gateway as the default. + This example project uses the embedded DuckDB SQL engine, so its configuration specifies `duckdb` as the gateway's connection type. All available configuration settings are included in the file, with optional settings set to their default value and commented out. - The command to run the scaffold generator **requires** a default SQL dialect for your models, which it places in the config `model_defaults` `dialect` key. In this example, we specified the `duckdb` SQL dialect as the default: + SQLMesh requires a default model SQL dialect. SQLMesh automatically specifies the SQL dialect for your project's SQL engine, which it places in the config `model_defaults` `dialect` key. In this example, we specified the DuckDB engine, so `duckdb` is the default SQL dialect: ```yaml linenums="1" + # --- Gateway Connection --- gateways: - local: + duckdb: connection: - type: duckdb - database: ./db.db - - default_gateway: local + # For more information on configuring the connection to your execution engine, visit: + # https://sqlmesh.readthedocs.io/en/stable/reference/configuration/#connection + # https://sqlmesh.readthedocs.io/en/stable/integrations/engines/duckdb/#connection-options + # + type: duckdb # <-- DuckDB engine + database: db.db + # concurrent_tasks: 1 + # register_comments: True # <-- Optional setting `register_comments` has a default value of True + # pre_ping: False + # pretty_sql: False + # catalogs: # <-- Optional setting `catalogs` has no default value + # extensions: + # connector_config: + # secrets: + # token: + + default_gateway: duckdb + + # --- Model Defaults --- + # https://sqlmesh.readthedocs.io/en/stable/reference/model_configuration/#model-defaults model_defaults: - dialect: duckdb + dialect: duckdb # <-- Models written in DuckDB SQL dialect by default + start: 2025-06-12 # Start date for backfill history + cron: '@daily' # Run models daily at 12am UTC (can override per model) + + # --- Linting Rules --- + # Enforce standards for your team + # https://sqlmesh.readthedocs.io/en/stable/guides/linter/ + + linter: + enabled: true + rules: + - ambiguousorinvalidcolumn + - invalidselectstarexpansion ``` Learn more about SQLMesh project configuration [here](../reference/configuration.md). @@ -133,13 +272,19 @@ SQLMesh's key actions are creating and applying *plans* to *environments*. At th SQLMesh's key actions are creating and applying *plans* to *environments*. - A [SQLMesh environment](../concepts/environments.md) is an isolated namespace containing models and the data they generated. The most important environment is `prod` ("production"), which consists of the databases behind the applications your business uses to operate each day. Environments other than `prod` provide a place where you can test and preview changes to model code before they go live and affect business operations. + A [SQLMesh environment](../concepts/environments.md) is an isolated namespace containing models and the data they generated. + + The most important environment is `prod` ("production"), which consists of the databases behind the applications your business uses to operate each day. Environments other than `prod` provide a place where you can test and preview changes to model code before they go live and affect business operations. + + A [SQLMesh plan](../concepts/plans.md) contains a comparison of one environment to another and the set of changes needed to bring them into alignment. - A [SQLMesh plan](../concepts/plans.md) contains a comparison of one environment to another and the set of changes needed to bring them into alignment. For example, if a new SQL model was added, tested, and run in the `dev` environment, it would need to be added and run in the `prod` environment to bring them into alignment. SQLMesh identifies all such changes and classifies them as either breaking or non-breaking. + For example, if a new SQL model was added, tested, and run in the `dev` environment, it would need to be added and run in the `prod` environment to bring them into alignment. SQLMesh identifies all such changes and classifies them as either breaking or non-breaking. - Breaking changes are those that invalidate data already existing in an environment. For example, if a `WHERE` clause was added to a model in the `dev` environment, existing data created by that model in the `prod` environment are now invalid because they may contain rows that would be filtered out by the new `WHERE` clause. Other changes, like adding a new column to a model in `dev`, are non-breaking because all the existing data in `prod` are still valid to use - only new data must be added to align the environments. + Breaking changes are those that invalidate data already existing in an environment. For example, if a `WHERE` clause was added to a model in the `dev` environment, existing data created by that model in the `prod` environment are now invalid because they may contain rows that would be filtered out by the new `WHERE` clause. - After SQLMesh creates a plan, it summarizes the breaking and non-breaking changes so you can understand what will happen if you apply the plan. It will prompt you to "backfill" data to apply the plan - in this context, backfill is a generic term for updating or adding to a table's data (including an initial load or full refresh). + Other changes, like adding a new column to a model in `dev`, are non-breaking because all the existing data in `prod` are still valid to use - only new data must be added to align the environments. + + After SQLMesh creates a plan, it summarizes the breaking and non-breaking changes so you can understand what will happen if you apply the plan. It will prompt you to "backfill" data to apply the plan. (In this context, backfill is a generic term for updating or adding to a table's data, including an initial load or full refresh.) The first SQLMesh plan must execute every model to populate the production environment. Running `sqlmesh plan` will generate the plan and the following output: @@ -257,12 +402,9 @@ Updating physical layer ━━━━━━━━━━━━━━━━━━ ✔ Physical layer updated -[1/1] sqlmesh_example.seed_model [insert seed file] -0.02s -[1/1] sqlmesh_example.incremental_model [insert 2020-01-01 - -2025-04-17] 0.03s -[1/1] sqlmesh_example.full_model [full refresh, audits ✔1] -0.05s +[1/1] sqlmesh_example.seed_model [insert seed file] 0.02s +[1/1] sqlmesh_example.incremental_model [insert 2020-01-01 -2025-04-17] 0.03s +[1/1] sqlmesh_example.full_model [full refresh, audits ✔1] 0.05s Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 ✔ Model batches executed @@ -327,7 +469,7 @@ Run `sqlmesh plan dev` to create a development environment called `dev`: $ sqlmesh plan dev ====================================================================== Successfully Ran 1 tests against duckdb ----------------------------------------------------------------------- +---------------------------------------------------------------------- New environment `dev` will be created from `prod` @@ -341,25 +483,25 @@ Models: └── sqlmesh_example__dev.full_model --- - -+++ - + ++++ + @@ -14,6 +14,7 @@ - + SELECT - id, - item_id, + id, + item_id, + 'z' AS new_column, - event_date + event_date FROM sqlmesh_example.seed_model WHERE -Directly Modified: sqlmesh_example__dev.incremental_model +Directly Modified: sqlmesh_example__dev.incremental_model (Non-breaking) └── Indirectly Modified Children: - └── sqlmesh_example__dev.full_model (Indirect Non-breaking) + └── sqlmesh_example__dev.full_model (Indirect Non-breaking) Models needing backfill: -└── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-17] +└── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-17] Apply - Backfill Tables [y/n]: ``` @@ -378,8 +520,7 @@ Updating physical layer ━━━━━━━━━━━━━━━━━━ ✔ Physical layer updated -[1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - -2025-04-17] 0.03s +[1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-17] 0.03s Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 1/1 • 0:00:00 ✔ Model batches executed @@ -447,7 +588,7 @@ Enter `y` and press `Enter` at the `Apply - Virtual Update [y/n]:` prompt to app $ sqlmesh plan ====================================================================== Successfully Ran 1 tests against duckdb ----------------------------------------------------------------------- +---------------------------------------------------------------------- Differences from the `prod` environment: @@ -458,20 +599,20 @@ Models: └── sqlmesh_example.full_model --- - -+++ - + ++++ + @@ -14,6 +14,7 @@ - + SELECT - id, - item_id, + id, + item_id, + 'z' AS new_column, - event_date + event_date FROM sqlmesh_example.seed_model WHERE -Directly Modified: sqlmesh_example.incremental_model (Non-breaking) +Directly Modified: sqlmesh_example.incremental_model (Non-breaking) └── Indirectly Modified Children: └── sqlmesh_example.full_model (Indirect Non-breaking) Apply - Virtual Update [y/n]: y diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 66f55de516..d5e7b2ce07 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -267,15 +267,17 @@ Options: ## init ``` -Usage: sqlmesh init [OPTIONS] [SQL_DIALECT] +Usage: sqlmesh init [OPTIONS] [ENGINE] Create a new SQLMesh repository. Options: - -t, --template TEXT Project template. Supported values: dbt, - dlt, default, empty. + -t, --template TEXT Project template. Supported values: dbt, dlt, default, + empty. --dlt-pipeline TEXT DLT pipeline for which to generate a SQLMesh project. - For use with dlt template. + Use alongside template: dlt + --dlt-path TEXT The directory where the DLT pipeline resides. Use + alongside template: dlt --help Show this message and exit. ``` From 83cd0979bd7fe9352fef8d2cc0439e29aed7c238 Mon Sep 17 00:00:00 2001 From: Trey Spiller <1831878+treysp@users.noreply.github.com> Date: Mon, 23 Jun 2025 11:06:55 -0500 Subject: [PATCH 2/7] Update docs/quickstart/cli.md Co-authored-by: Sung Won Chung --- docs/quickstart/cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index 2bff176787..8cee1563cd 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -161,7 +161,7 @@ Need help? • GitHub: https://github.com/TobikoData/sqlmesh/issues ``` -??? info "Learn more about the project's configuration" +??? info "Learn more about the project's configuration: `config.yaml`" SQLMesh project-level configuration parameters are specified in the `config.yaml` file in the project directory. This example project uses the embedded DuckDB SQL engine, so its configuration specifies `duckdb` as the gateway's connection type. All available configuration settings are included in the file, with optional settings set to their default value and commented out. From 3691b659b8fe5798c0d279254255612a04a41d7e Mon Sep 17 00:00:00 2001 From: Trey Spiller <1831878+treysp@users.noreply.github.com> Date: Mon, 23 Jun 2025 11:07:11 -0500 Subject: [PATCH 3/7] Update docs/quickstart/cli.md Co-authored-by: Sung Won Chung --- docs/quickstart/cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index 8cee1563cd..e03db32d86 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -124,7 +124,7 @@ Enter a number: 1 SQLMesh's core commands have multiple options that alter their behavior. Some of those options streamline the SQLMesh `plan` workflow and CLI output. -If you prefer a streamlined workflow, choose the `FLOW` CLI mode to automatically include those options in your project configuration file. If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode. +If you prefer a streamlined workflow (no prompts, no diff previews, auto-apply changes), choose the `FLOW` CLI mode to automatically include those options in your project configuration file. If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode. ``` bash Choose your SQLMesh CLI experience: From e77c2fbd3633d942865b3c204b38689632f3477b Mon Sep 17 00:00:00 2001 From: Trey Spiller <1831878+treysp@users.noreply.github.com> Date: Mon, 23 Jun 2025 11:07:26 -0500 Subject: [PATCH 4/7] Update docs/integrations/overview.md Co-authored-by: Sung Won Chung --- docs/integrations/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/overview.md b/docs/integrations/overview.md index be7606cec0..5e850afbf6 100644 --- a/docs/integrations/overview.md +++ b/docs/integrations/overview.md @@ -9,7 +9,7 @@ SQLMesh supports integrations with the following tools: * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli) ## Execution engines -SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses): +SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses - example usage: `pip install "sqlmesh[databricks]"`): * [Athena](./engines/athena.md) (athena) * [Azure SQL](./engines/azuresql.md) (azuresql) From 28404b2908d33d01aca88049569f24331accc86f Mon Sep 17 00:00:00 2001 From: Trey Spiller Date: Mon, 23 Jun 2025 15:57:04 -0500 Subject: [PATCH 5/7] Add explain and DuckDB CLI sections --- docs/quickstart/cli.md | 132 ++++++++++++++---- .../cli/cli-quickstart_duckdb-tables.png | Bin 0 -> 11515 bytes .../cli/cli-quickstart_duckdb-views.png | Bin 0 -> 7245 bytes 3 files changed, 107 insertions(+), 25 deletions(-) create mode 100644 docs/quickstart/cli/cli-quickstart_duckdb-tables.png create mode 100644 docs/quickstart/cli/cli-quickstart_duckdb-views.png diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index e03db32d86..93386987e3 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -124,7 +124,9 @@ Enter a number: 1 SQLMesh's core commands have multiple options that alter their behavior. Some of those options streamline the SQLMesh `plan` workflow and CLI output. -If you prefer a streamlined workflow (no prompts, no diff previews, auto-apply changes), choose the `FLOW` CLI mode to automatically include those options in your project configuration file. If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode. +If you prefer a streamlined workflow (no prompts, no file diff previews, auto-apply changes), choose the `FLOW` CLI mode to automatically include those options in your project configuration file. + +If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode, which we will use in this quickstart: ``` bash Choose your SQLMesh CLI experience: @@ -147,18 +149,18 @@ If you chose a different engine, add your engine's connection information to the Your SQLMesh project is ready! Next steps: -• Update your gateway connection settings (e.g., username/password) in the project configuration file: - /Users/trey/tobiko/sqlmesh/sqlmesh-example-fresh/test/config.yaml -• Run command in CLI: sqlmesh plan -• (Optional) Explain a plan: sqlmesh plan --explain +- Update your gateway connection settings (e.g., username/password) in the project configuration file: + /sqlmesh-example/config.yaml +- Run command in CLI: sqlmesh plan +- (Optional) Explain a plan: sqlmesh plan --explain Quickstart guide: https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/ Need help? -• Docs: https://sqlmesh.readthedocs.io -• Slack: https://www.tobikodata.com/slack -• GitHub: https://github.com/TobikoData/sqlmesh/issues +- Docs: https://sqlmesh.readthedocs.io +- Slack: https://www.tobikodata.com/slack +- GitHub: https://github.com/TobikoData/sqlmesh/issues ``` ??? info "Learn more about the project's configuration: `config.yaml`" @@ -212,7 +214,7 @@ Need help? Learn more about SQLMesh project configuration [here](../reference/configuration.md). -The scaffold will also include multiple directories where SQLMesh project files are stored and multiple files that constitute the example project (e.g., SQL models). +The scaffold generator creates multiple directories where SQLMesh project files are stored and multiple files that constitute the example project (e.g., SQL models). ??? info "Learn more about the project directories and files" SQLMesh uses a scaffold generator to initiate a new project. The generator will create multiple sub-directories and files for organizing your SQLMesh project code. @@ -245,7 +247,7 @@ The scaffold will also include multiple directories where SQLMesh project files - ./tests - test_full_model.yaml -Finally, the scaffold will include data for the example project to use. +Finally, the scaffold generator creates data for the example project to use. ??? info "Learn more about the project's data" The data used in this example project is contained in the `seed_data.csv` file in the `/seeds` project directory. The data reflects sales of 3 items over 7 days in January 2020. @@ -291,7 +293,7 @@ The first SQLMesh plan must execute every model to populate the production envir ```bash linenums="1" $ sqlmesh plan ====================================================================== -Successfully Ran 1 tests against duckdb +Successfully Ran 1 tests against duckdb in 0.1 seconds. ---------------------------------------------------------------------- `prod` environment will be initialized @@ -303,20 +305,18 @@ Models: └── sqlmesh_example.seed_model Models needing backfill: ├── sqlmesh_example.full_model: [full refresh] -├── sqlmesh_example.incremental_model: [2020-01-01 - 2025-04-17] +├── sqlmesh_example.incremental_model: [2020-01-01 - 2025-06-22] └── sqlmesh_example.seed_model: [full refresh] Apply - Backfill Tables [y/n]: ``` Line 3 of the output notes that `sqlmesh plan` successfully executed the project's test `tests/test_full_model.yaml` with duckdb. -Line 5 describes what environments the plan will affect when applied - a new `prod` environment in this case. - -Lines 7-11 of the output show that SQLMesh detected three new models relative to the current empty environment. +Line 6 describes what environments the plan will affect when applied - a new `prod` environment in this case. -Lines 12-16 list each model that will be executed by the plan, along with the date intervals or refresh types. For both `full_model` and `seed_model`, it shows `[full refresh]`, while for `incremental_model` it shows a specific date range `[2020-01-01 - 2025-04-17]`. The incremental model date range begins from 2020-01-01 because the `full` model kind always fully rebuilds its table. +Lines 8-12 of the output show that SQLMesh detected three new models relative to the current empty environment. -The `seed_model` date range begins on the same day the plan was made because `SEED` models have no temporality associated with them other than whether they have been modified since the previous SQLMesh plan. +Lines 13-16 list each model that will be executed by the plan, along with the date intervals or refresh types. For both `full_model` and `seed_model`, it shows `[full refresh]`, while for `incremental_model` it shows a specific date range `[2020-01-01 - 2025-06-22]`. The incremental model date range begins from 2020-01-01 because its definition specifies a model start date of `2020-01-01`. ??? info "Learn more about the project's models" @@ -393,23 +393,23 @@ The `seed_model` date range begins on the same day the plan was made because `SE GROUP BY item_id ``` -Line 16 asks you whether to proceed with executing the model backfills described in lines 11-14. Enter `y` and press `Enter`, and SQLMesh will execute the models and return this output: +Line 18 asks you whether to proceed with executing the model backfills described in lines 13-16. Enter `y` and press `Enter`, and SQLMesh will execute the models and return this output: ```bash linenums="1" Apply - Backfill Tables [y/n]: y -Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 +Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 ✔ Physical layer updated -[1/1] sqlmesh_example.seed_model [insert seed file] 0.02s -[1/1] sqlmesh_example.incremental_model [insert 2020-01-01 -2025-04-17] 0.03s -[1/1] sqlmesh_example.full_model [full refresh, audits ✔1] 0.05s -Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 +[1/1] sqlmesh_example.seed_model [insert seed file] 0.01s +[1/1] sqlmesh_example.incremental_model [insert 2020-01-01 - 2025-06-22] 0.01s +[1/1] sqlmesh_example.full_model [full refresh, audits ✔1] 0.01s +Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 ✔ Model batches executed -Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 +Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00 ✔ Virtual layer updated ``` @@ -428,7 +428,89 @@ Lines 12-14 show the progress and completion of the second step - executing mode Lines 16-18 show the progress and completion of the final step - virtually updating the plan's target environment, which makes the data available for querying. -You've now created a new production environment with all of history backfilled. +??? "Learn more about the plan's actions" + + Before applying a plan, you can view a detailed description of the actions it will take by passing the explain flag in your `sqlmesh plan` command: + + ```bash + sqlmesh plan --explain + ``` + + Passing the explain flag for the quickstart example project above adds the following information to the output: + + ```bash + Explained plan + ├── Validate SQL and create physical layer tables and views if they do not exist + │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ │ ├── Dry run model query without inserting results + │ │ └── Create table if it doesn't exist + │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ │ ├── Dry run model query without inserting results + │ │ └── Create table if it doesn't exist + │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + │ ├── Dry run model query without inserting results + │ └── Create table if it doesn't exist + ├── Backfill models by running their queries and run standalone audits + │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ │ └── Fully refresh table + │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ │ ├── Fully refresh table + │ │ └── Run 'assert_positive_order_ids' audit + │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + │ └── Fully refresh table + └── Update the virtual layer for environment 'prod' + └── Create or update views in the virtual layer to point at new physical tables and views + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + ``` + + The explanation has three top-level sections, corresponding to the three types of actions a plan takes: + + - Validate SQL and create physical layer tables and views if they do not exist + - Backfill models by running their queries and run standalone audits + - Update the virtual layer for environment 'prod' + + Each section lists the affected models and provides more information about what will occur. For example, the first model in the first section is: + + ```bash + ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ ├── Dry run model query without inserting results + │ └── Create table if it doesn't exist + ``` + + The first line shows the model name `sqlmesh_example.seed_model` and the physical layer table SQLMesh will create to store its data: `db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172`. The second and third lines tell us that in this step SQLMesh will dry-run the model query and create the physical layer table if it doesn't exist. + + The second section describes what will occur during the backfill step. The second model in this section is: + + ```bash + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ ├── Fully refresh table + │ └── Run 'assert_positive_order_ids' audit + ``` + + The first line shows the model name `sqlmesh_example.full_model` and the physical layer table SQLMesh will insert the model's data into: `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. The second and third lines tell us that the backfill action will fully refresh the model's physical table and run the `assert_positive_order_ids` audit. + + The final section describes SQLMesh's action during the virtual layer update step. The first model in this section is: + + ```bash + └── Create or update views in the virtual layer to point at new physical tables and views + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + ``` + + The virtual layer step will update the `sqlmesh_example.full_model` virtual layer view to `SELECT * FROM` the physical table `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. + +Let's take a quick look at the project's DuckDB database file to see the objects SQLMesh created. First, we open the built-in DuckDB CLI tool with the `duckdb db.db` command, then run our two queries: + +Our first query shows the three physical tables SQLMesh created in the `sqlmesh__sqlmesh_example` schema (one table for each model): + +![Example project physical layer tables in the DuckDB CLI](./cli/cli-quickstart_duckdb-tables.png) + +Our second query shows that in the `sqlmesh` schema SQLMesh created three virtual layer views that read from the three physical tables: + +![Example project virtual layer views in the DuckDB CLI](./cli/cli-quickstart_duckdb-views.png) + +You've now created a new production environment with all of history backfilled! ## 3. Update a model diff --git a/docs/quickstart/cli/cli-quickstart_duckdb-tables.png b/docs/quickstart/cli/cli-quickstart_duckdb-tables.png new file mode 100644 index 0000000000000000000000000000000000000000..27d7180f8d920209fdd76c1e6d7a57d25aaba424 GIT binary patch literal 11515 zcmZX(1yo$k5-p4bAKWrna19U!2p&RkCpaYd;O-FIo#3v)Em(rP5AN>n5Zw8PeBXWd zzW4W<)o13cuCA)RcU7M?CqzL`937Pi6$S%UkzE|z{18r{ zi7!4x7AR0m_PQ0{c;BNl5rp*#nL>m}M6CBrq)LnE(ky9rp1wcnNc!NhR_h$x`-aE3 zolqm49+1ICH(fNtcg*6C^(puwI2_>%&Wf{H%pzmUJ5?SA0>eSqRGcCE13JFEnR1<4 z)}s8={i%|1Hry2rXF;e7e-W|ETp${kRe1)tz!0S{r7-kf6pg`g1M_b9-O^b{Bj@3i z&2|GfYS!~b67H*5)vlsiME2<(uieSn{^r0gqcZtXi~0SV+ExyfPynx-4(zYwDda>!4#|w7hy9e%cCIx zW#n<6+Kyfky%7A}m3C{&w*E|GPBcQ`EaNx8RWN`^_s8pI`l|v1C;D>n_8J4J-g^O_Vpkn=%QwBd~tDY6^8quJ8Vn_GA{$PwvnKQHcIj-4#E}MaJbs$off$Ml_6a`Mi z&sFU-0T*cD3rN|+2ErS6wcHk=3^iZ`2Ze|9+1xXP>nzyLi3yy4MvQR7R~QC@7z=S! zEtd2wXKz+5isN@mwOcnvqmLg0^G5~np;RD;guf4nrw<2~Ob0o4Mps+7AFh<8Zp81s zf?c2#-s!SQElk3*r_9lH0AzbBA#8nHBAyEjwtqOBfJZb#FeaRl!mY*; z8+(R;SIgsSw6)i9WQmai&8mgZH}Z9a_&>vQ(>G6@K}~6u`j%!(0Z_-$>w7UGiD{=+ zf~wT&XB4}+bBwJYQmL9qK0ISa!>WTy*+MDC%04TA+X@5NB7Z;~X`Sv@+!aFv{bfui z7Xx--o|aD+o32;0r%rgjrKM!`CAGNARxK$By(qhHm(P_JHOGB%GC18fqfp~^rN+~s zb4HL;)>USZIK!Ui!@l!Cna1sq{8G#DEa&4tAxU+~zvvSR%FM~W{L1SWj;WY|FZC1E zw@T8Bnf^^t0(hc12T5PSimVBlFpY!eh2tNE-V1%bZiQz?)s22wTG$d+}I;Ztgb1$mL)i{1h zk=uZ0tg(&u#sG#E78bPRB%EXtZXFx1^)qLip2>MT86~Z2u4hqD4Ypa0M7gBlo4L#8 z`3sWlPK2nIgG+>TE<_|~Sa-@7MA87{7cMuOmD)B-S&%N?j@<46*8#CUZfJfIU|pQm zpgK2$=RdJ;H(SM9={E!e{cZNJ#zFp}wz158Di>^xm&EH0UA&h+!qNRe2v-airUf$W zXUU&+pMPvuq+XHo+ue~|hh-oaiGX!IgF&~A*Kf$VT|yY)Isy^DWHh+ko?Q~-fv}-) z;s6idR)}4_emA(BHArc;wL4DnO#kW}Yl3)1E@=KQznmQ@nz)wl_^1ss%XVD;4@}BsS2CXn24BZgsn+# znFIQIO))Beta+C9Bi-xh4ZOgi6ypUfYk6=*)Opy-ASNVIQe0b~xz6Q?m+I}@ulW9$ zG2FN8e4N#)W%d{Z!H5^I3=&^|I+bFaPg~_;P2w*1|3OQ9QeMU+#v}@`Ogx3)Qct?| z#6xB~2fzMBadoJTQg*Fl${a~9O{%2tw3G7a;q{O2bH7hk*%Mi+?J~K&unpi9z57?} zqf4_-&8I#t(CmE`2!_-CF&%c*g@J^MK-FV}NfwerFP9PwZ*6D;_u6I9eLd5?u&)jE zN7QC=g(> zxKfSLr1n=;#(t$CW}+4~NbRgRXXqW#IaSqUtxf(UkRKfdaT7&sq*MOkMsLgk6`(`I z=0kkRYj*No#kM`b1Hqh3c3Z#MB}=qv6>}w0cv7`DJW()=ld(&Qfrie$$4#$Dxx(rL z@&b*;7j9vMAlXCjk^!5ZJX60$6g3BNIT9=5t}h|*43T)qQqxkzUWWo6 zjM*Y~-DqD5Y~z{9!DdgZ-ja^<7;^`)#5qJJ))F%Ejlg0@!PJd}%BiM1?zFC<d3?ix^D|ITzDYx?Pzh>H=@{J(44pY1A?3Z)i*6iuv+5uvTT|X& zQ!@UfX$S5M;KwxX_+bc^tu_JfH8fz$)b$U1Pdwd`tZ<4v(X5x#LHt`5BtXY{;b#sO zZs|iL7SPvwhfsig$CSJ(7>lSYoW$`F^tWFj_cYr`3P>RKj7!6u*rFvU#By11uxcdV z*9g%UoIP>1a>6U;}sVb;>reXXNM1ofXWWbw>iS2`zJ&$Tx!gKFd0 zv;a!EFg~(HJ1BJ~1y4vgx2|w44DS6_jnsFE?R>W~;x7$0czI~w@KkAb8vj1Tx_)(m zK-GnLGa=XDXjtIp6I`nUGf#0u+jwe=q+9-G^S9tW^;S1)Gd|Yk)r>w2ZGs3Q`P*uz zR)rpMiuLB8U;r5_W42_uZ_1a~g(Ts)3AG66La<7x*3qHUB7x67BAyxAnc=%Sh|inQ zj+UCld3)n9Dx6YcnFAFdnT^XM_DTlgc8L|ewk?d|{7Rfu6tVt` zt7Of4H%MrU#rak6H5nzS0#=uFjRqbggROU2v}*JG!+ zlG5+Z#d~zFr%f8BmKJCnNrAJDem~YG}sUCd% z-Mc6#si5F;xsDVhYrIIHCxx_;?TI*POL!W8+9*McpnFvc!n%F8WH)3rf|`gCy(W3R zp9wagGB9lKb9$$2v~`(qea>5CV9Hmu&4j^#Z!bUc09xhhcr9mqYZ7 zwm#%k*z@$OqMZK&%VWMXp62}OLoF3uO_1+L zfET&g%{jl@Ug$k-(%Y~n!&CX?ds8u}-25p$?Lya@UquCbI@*tSz19`{JrVpEClCC} zZGPu64NyOGRV^s@VCO3Hn|CRH6J-1kZ6E|^oXuDtS#(M-tWWUOuv^#c=zkj(eEEp{ zp~+Uc3cPLp+GHhkK3$`RE8^+qYNycya(v1Ec+sB4RfzFFMR_Rkl_N^w1WX|JJZu^*`EYf*Q zMf*yUs+W`99o>?khJ>~5u?gvcgr3nvDm$eo_8mGMz`(mt%dl&F<1)kmpioaodIS@} zVZWtSNDbzPRzX?y*ixQ-mJhOsL##$jdW*&2ZPjaSq;;QW$2br-<8zdu8|H8DK1@wH z6KM-Dv*FE2-Re3+Jcja4E3TaU;{9-ejA{R`V;7}{q#JJLnYllX_=(L%Oh*qJ;L{Gu z5DX=GtyjA`Y8z8b)h4Sn(i`@Tl-il^m});f_8M}T%29NNop_(7o8+ShhLihMM->ux zsCd7xLyG18!c+5VDH;fVE4cfPOrR{<=a6z(hCsr7>24m!Tt}hQIlftmJf~4>u=(c) z??}22^HqGx%eGNt@WACNrxm$YgAQY?QhRdvpfb!JYz2YL??nbTW5}cZox`B9GdC74 z&{Jxx^#JktMCtW)7g;Pd}>KU5M)LGvb>J=A)9-AIci?j%R2xpG_ zq<~$H*Y(og%#EF?@xTYA8A}SC9w8NK{e=D*M^UNVUy~oWi}A?FSZ2vH4>;=hm~?&o zC6=u2qrb4_1hW}X=l#-U?D3a1C$cO3f;2J zq8u-9=FooFK&W*IvCS{w5_Zurvb8w6CQO4|L85$+`Hh7>-Ss8xvL=#vbWNDrqrvi+ z&gTh?t*~^TD&->r8?LbW+Uy~#BH7I(eHBHHYflC^F|)8dHbWkqErHs)!d)OCM3epg z(z8i)8Mg(}8FFE}AFL81D|_CAO4GgFdxjuM>knI3I+oy1KHAZGf+1y6Iit><1V1<7 z=#L91*$uJOSlR-IK1E`i0!S?>Rt-s%g^DR}8-CJzyjGC~aTgPi%_)G*F-dD>RDX=a zNk%s;UzPVv1+uWbvzw(iC8G$GB|NcDn@XzWp+%Z-+!Y(7$c~^Rd4N$ps}zIZiNC2g zsqIv-_>R^*qseh!aM=bN6O9n1ikO^szFS1;gfQGN3?Q@DU+^>-motx!}hVwXVEM zm1o?OJ4|_z0Jh)zH;lth1Mar`cYlfkv)8emOeQa4ICKxxI1i_9l0{vvEW@ox!&y#3 z2A}jWg5;U?+FUNSPJNCTK?6a8tGhM996YAGvbfrbt|s=ccq#tuO61sVELMK_q&qB0 zGCz*>RGkgK0D;J#6$~?wSlOUPVV7!QA|i*A)phZ_0)3qDHo=LpF?g5!$M{zv3I}J< z2{{2}buAwXXx_Fef6;Z!aG3>4h0g;zO%g*eQ*82Y-#-TY!;7xo=`Ql!LMmswp_>O) zlnja(3p7$L`=XaF$@SP@HM6`WmMrgUW6i<~pCX;xMA!*@3O>|=1tsmLkp_NUOIZx& z0k`&2*OwA6BsPv+S5TiQgrc&=Nsrd&HNx)S&?`BK*ZzgMrBm}d5p@dI0yQ|I(YG5O zg&g7e7II*q8|2v4lpgVb8HADhy(L*8e*cq`t#@?+3%@M|D;GIeZ2QDT7aS=XUPN*2 zTN|EF68e3ccC&JVe8+F?XXDNq;*bA~6@IAH-?n-B@v0-zN{bj0df1(QK=ozlTov2z zdKzWrr_3&)F$o!36ZR8KQG6gKd?LKH4k*r$9BcGt}+Us5!#f;>9BpjW^n?hyegBMS%y2s|sV8gXM#gm;x=74B1yOw;@W4Wc9EpY#W!|0-tkYz_Y(={DH23@(^p= z`04?U$7RD3-4dY37tvlV!~e{r)>U_5JSgoFfKW?!W+-~KzBz}WgDrM(gxjK(jMe8Y zj>ILNal60V=Om`PF;%1|(6!uJ;Io&g49Z1CfehNNFOwzTT~>K9mjB)lVCZP`g! z_J;*Yq7+#2Kvsq*WSfjSV(SliAA4Yp$s*rDDl+SGyq>SU)b9-a>iQvXH?s0E`Dt|$ z@g<|f5eGG7D&conP@b4XY?L(`bw1tFJD2){VZJaZ_ReR(f-5lBy{c&6=WLn zH&Ek^((71Y-gD2~gY`kpEDhrEx9Zeg+WD6uwnYOwdU*b2EG;XX1d6(f8Ra=f=fB<`-{nO5o z`R&ym81ZjWdc?u%4*ufrFp-sSz6$I=w9B8n`E3{`SMfD0bo(SVY^au=cny0$BPJ6> zv3_*=O33Le-8n2T5k)Z-aHtP~qE?T~qy=C{d4I_B$s{>VxT%7V($>iMpc3&t#xY6w zha-FZU)5f5ouRiSYw!{Vw3Rjul$Hf6AJ!a`)viTbd3FZf^iU!b=Lt~wXlpSWs-wjwe$@`4+aGAs>l&E)j|f_6t8TW ze#kBA*xCQQBNUit3)2S@b#`bBgO42`(@RaYudD6E0Q;kGw6VR#v53nOb1xs09U>Pl zMw{!7fp#=Ix~qG(&rX51YcaDR4Ww98qtb4<4eE*rcmU51e1%tygV9N*!K+z8K0teo zZ+sh7sZnF-4UetU*uq|R?0|LVX_bsHqPu}UfFA)TntK5$oZQ>jGEsu3Ol&od?K~Qd zrz-hKlUian=d1MtSTxvKNL%xC&LlcB<9xvOaY6hEWU>2?lnj^Z=)fH{{C6(7n&0Ny`nHRq0&?h z;jsDJ+H1b)s0{AGo6F#a!rdht?L;8$d5S-T99gkF&ERO6YzqVQZPMW~>dwuP2}PCv zS?`3!fw=d_yGV0wz2xL%(L#;N^U_QO<(1`x` zO0ENg1qa>r7S0uV5LQ4)N9(!ZhtZT4|iq^E9#jcD_mL z!~WpdtCvllC8=?!wwbqQWLk8G?}mu4-$dH=0n)20K5zKSE(~ioX6yW^s1JJp!oI9F zslD~ZP5im&_Hglx4{Vpl1NJ>YefAIvAknkn{_ccGtNC5YB1OijDVyA*g@t*&ez&Oq z=OX^VaBIsfvsN!m_R6%@^Lu9Ns5E|jymZa6!a(V7;&ZgH$ukZ17p#!FI?_|0zd5xWWeup?UU@bPZ-NiSoV(L61nb_TYb|JL?`((6O2Nj$3A0$ zm9A;Yu{PA4OWNUNo?E!p;HwS1t|S(Xl#NN6PEFdM@UZQ%jfk7@V@T(Bxm8 zNJH{QCxhi;;G}Dhz8a{A9$h?#rxX&PH8Rzn6p3!6$k?N&@T@G5d5In$LUWd-)QMWB z>|AM{0(o41OC_((`ZzucwdXPww!}$}femUWP64>PR{3G=!?Q#61@m%b;+Y-V;WUf#RXS-#Vqn3U ze2#`)$|<@EOvuoTW?kKV;Q>&15|}@~oged`-%+gpG?WZ2NMhW-Yex5oZewsrNXcJ) zjE2>ZbB7g2`YY506nivO-!~&Vt@S4ta~NwLB?Nf*MTwl3uZt3`1l%nmK7VtHXp}ia zSmWk>X-`z7>eez>%$Zj$ha^m;)>9%!Sz-qzMZ)13S_W^~5zi3*aWR(AO02x2a+Tgs z;M#NaMFqEzJ+5Jn>iUq#Jb4qX!`9)YnTg$vCPseQp_LN8X91%ed{K!|QY|e@6|fum zS(zZ&LQe8ejJZ>_vWaP{&5(sdMxN#Ji%G{ zCe`8jHJNL)n^VN^{|S4kg;e5Xw=0iHNLqcRIFz5b9JkDPrfz^*2P^U)Hpag;wWtU~ z&03*m{{r$h*{BJMrBZPUO@nZ|jUsQi}{@=^~?c9Y$MMaV9@BWV6YT-6QkX8UxJWOcO@VpKWm!F1-0xi$#mvl z)uViFUbU^*j@NNZ)95CrYP=dr`MuzJ(Z!nXR_U)(XB*4O*W>SbJgFoy^(t05*5&+| z6f3A_65m|6_@PZr)dKoLw|*!JrP%G^kSA+x{-|-i3vAaJrdIBGG9To$GlqNnMmg@I z*pEd7pbUHrwLYv@kX$Bi>}>m>kNtWFO_b+AdyemCgR?4?HZSfh$8Bo%>Z1y_e)wa4 z7i2+JL9gS6h(OAD@nP-NFqwfprWqoqX~PY8(DN~mcT#|5sKaZ1r!~;Ln_L>D{Cc~s z<4syXV0xX?{&>8vuWu7S*a}4HHcI7ZW;UCfalJbimgtThtAOrE%BK;4?j-L3AL1DM z)LG>X=o~`yKs50Fg_+DbV!1dfxafX&rs4xw7&^o%=6rkD&p`PufcFUt8N#{DSkj`=5@5?lv@XU*-vTpiuGd4@$B& zcRmSYSPvtNkUi9Gm^bs58SF0=Z>l zU$#_0MD)IB$5J2P`Fl?PN3k~9Q}hlu`HXC(h+lJ5S!XBTh?C)ffvIysf&L|vV0J?5 zWo;5G-{hOZalFK?v`RCrYS_-85iMl$EMLO`2lCFSIk$1Z{3^2QGW#Rv)7F%l(K4s_+XYlNLZWx;SwMR~Y~12E zJ-y&hr37Co9i6yhIg4$EEWc8QA{@nu5#ojzzIDq;t#R@*(}hZ}TrvvVrg|Vr@>>cE zr%vk*4BZ>OhIDctz=Z4xZ{e8 zFfD|`6w9t~{QOTrq(B@)!&JZ~M14Q_ez^QkSfZO1(L8$bxL)kH4}5*(u>@UfbwAuO z&A5{D`~VF<>TKrir>&G(E2%rzyn{amu0ctlHB(MQSbLY>ezA{r48S7ha(^h3wl?-t z6YkJllMv2_2L{S44^JJiM@j@#acIJ=i|Hja7-g7~qWi%ply+5yoW|-!NH8D7R=F zZ7*4{lb^km#rOs)26k^iq)3*pGb;}w(@N*OtWgoUur#e!I{{9)`45SpPxPiF+>QjM zH#AXtji%{W5~wVn&_n zv^p)bb!^BhdNR}b7-#X!IuRT!#Z5J(`gzYA?!O{e$e*TjPWfKKcy*>zi=7@1-+j2r z2rI{RUW|~ImCE)VP<-~n27I4xCi3TDxfe2$$hsGEgZ7MpCycNbNWmKzp7SrXDZ;ir z?%BRdw&E|HuN1d*e(=?gKZFI=38xJAWmj+a;HPvbD(usn+L7DJ6 zOot63mBb5po_q>HKzj>QT>(QtjN^kv)oMfd7my- z%f&>WdF$^Sh=QWJtTt*1-n0cye0&s-x~gHIqX*KB$P%s|4Gjw0<`lvf33KdE8oXHr=D8N#3Xw&;-4e;fzw0GWFzj zxp>GJ7IgEq!$Qalyj&y&@QGCG8<@rT30glLVeIk@=Yw8D;MwNzk$>fU-~>uR!%=&a#>UV!CUiI=RL6gI^(JZ zE%}R7Jy|SdHEz3LBlnC!bRBaPW!m2UWsm*{w%@LnG)UzuUWHoQ5=6+5pZ0Lfbmb6* zR`$PCL$SRbn|pRsz62jPC)BD+ZqO^!+G$7K?r{SJ2+LUDqzqV&)JMwND-yFb>e)gr z1wuAxWFvEZqF$1*J4*r31gH=czNBRT`{q0@gV*)v&XuluPZfX|b_g{(i(<}kxqwz! z@-Nsbs0Z^nfW}si|MEJMsH-j;I1Z-Yx(-w9XN9E+nA&_%Q!`wKgY2d$??``oZ(yVy z*y%c@wV6#`JGSx{6u{U-D0P=+n-z{x1sWye2vWBLPR2)i{X&LD4B}1)Z@Fht;z6XXG)P?)?4=P5=sA)tks_=L;#_r|6 z;}zFGJCOJvmFx_A*imgr&WLBM*Oblqn+*%mym(@kq~**`9p5Dhfs|AJw}4bL_^u%l z!E{S#uJey<8#^o;S#Z^HRE-doYxJ;*hTCVJ30YT%$gWMonEyhr22kZmPWUHAJ_8A#yl6>fsXYIQC?s$EF@~gAoiY{cJFB$D8rRE#)cC+1Aa>fRntEe&+mL3 zON=ewze|jI;m23+K>y4ibzrdD`u=x@@qiV9p@R3geH{!V){}d%on}8=FZnnL oPqje`;~fnP^BVZ~QUS}d!EUxKOGX0yKMzJyR8HjQCtctF4>LaqO#lD@ literal 0 HcmV?d00001 diff --git a/docs/quickstart/cli/cli-quickstart_duckdb-views.png b/docs/quickstart/cli/cli-quickstart_duckdb-views.png new file mode 100644 index 0000000000000000000000000000000000000000..5d6af7fc870eca18e81c9e3105ad16b2065e2b7b GIT binary patch literal 7245 zcmb7{cTiJbxAy}my-2VCQbY(H1R^z55rI&Z-kX3(QFZIl3?dRnTyy2n=lH^oE&$&7wC zt!)7!?vAU14T~}X@|QH28A}mJ?=uGn2rW2`DV!Eu(A(QfJ%_z0^ZvQ!PzITFIi7qV zp<3W*Xw=Smp*q}PV((NkCHp%ERDH<-o*RTLEq|Qt3q&@wm054Dgl?aVXbdNbpVA3u zW`wo*P{%5?aIM0N6J4h&o4l3pySlBj`*=bxUp051gr2MwyzH-^3$<$VcXTNeD%SCT z8SgTt@HXL5cGK8#l0%2CyOWKnax0l!9j_aAYr}A;m(qfcnrs0dmWhn3{Ab&0>{=)i zcZThGfjmcJPz}9~25EC+n{cb0RsHqCEQc`@_)*yNI7QDFSc$a6~DhQM+Ast223?aCA`i&Ri?zuD5 zF@QXisT+GVaOzp2a)IYdyidH%eqfqrS6SkH`13n$^Djqd z8|g?YYiwxZaEJAC!T9#r-LxCUj1ZvYmak~DI_N#8T-&E#R_oV(nYx!y#a3{|OKmkb zmR4ra%Sd<3_o^l>w+osg4~111xPZ4ea4$G3<@?@nR9@*@NgUU6n;Bu!KaIDa%;uyd z758mIq2;$ysYN6EPoy_9;b_C|gubm`PV9Qqfir=&28M=C%s*of9uNiGk?KU4Jh?a` zW6!b1HJ{-$A{6{rPy<{c5!`ZJ_)}Wv`fIL0$x4hT&{{DCh zH?EiRXE>@BqNV+5$!`kyUQZ-$Pk;v4W;Q+g!epVM?ewTO#t{X1Qi5f5QjW?T^EnPZ z?UP(t1y{8AJfG?y0<@jall{i*-^1|Ka&FlOq>eEY&0A;Ve>(YS`Li&>BUvJPhrQ;H zS@u1qg4_0-_T#8li$+cM7%Kh6i(Y<|JfGFesjgYgZr=VvrjRExm?u2h5Mu+*U9}1w z$>zpVD0qn1kZ`NH7zTRc)pXNvL7TJ3CWUh$=Vkccuq)dRahUg+ve98pLwZT&rV@qt z+|?C_hcDC9H9xlIjo9}MZ6So(`p9{ki`2Md6lda3xvJpQ;9!-NePxdR z-|zxUMjywDeqD~@;;~RS(~Tz;o@I0X-n|)V3<*I&L#6w(v^dGm+5ATX!}QDFTyvr- zVj63`KMEO6Xjw9y z+HMWAOs5bataWNNSFi_To}zATn4A=tfhFPKU0_hF&K1#mTUaZ^@zm5LJ*qeJu>VJh zqel`aIoU6vULBH8CkGR#NhMZ_y7`;FFWh;}BFgF}D~E_seCeB%;=&2jV=~mqGp=3S zWZjXg1GLOM3iGQ;d0r7*H@NDK%gcGt7I8vYg@C<#9HNp^!fr|k_jf}yvWw(p?VY=c zqzijyO``^JQpUg}`}2m|o?fZVBB~N4xiab9;tXkGdXjL<7F+5WRO$+%5354t2PF~p zEeHQ5J=TlwBBy0lh9@eX>Z@A0F|CX{Um0+KWSS$2n2Ee4g%TnX9^I+UzFY5VCk4Z@%gqjVI7M0$^z$L-#5%ZCpx;^|AZ8scO3E z^E}GjzU*v?9LGrWAS;~d{+@pLG$5aEO05< z1x+D`L{VhE5a7ERC z@b#44iGA&TNOQ}UPxA+5Dr2I{p1G_mlWw(~3Ki7c8mf#mcvvZi zkC4sCdmI;%$(vdNMx7F^1$Ck z$#`No-#s}u7?=5%E!T`*itD`hRDVD2cFowehLMi(MKYJ@_`$ndW>w-%^n*Tn2uZz#gb|s_Fr*pUN zFkOB*DmzxVU+>@#W&T0`!gI-#X6H-pb)ZBaW1QbuSzVyEK`{%X+He$Cgpw_h)00@~ zO$n|Qj*|C0IPV*myzOAHZK@KXKUOC0-V2|~rHmE~G8D9Fb~0i*y%P4;lUqBho(J|e zOp`50_l|Ik`w3Xu+u6%hl6cCkIDvox-@C6U$-8@s)@5{(8^T4d7T%3xv@&_!t0+!k zq$DM)mQb{0uO!ti%<%f-&66v_q~7R((Yzv3*y`%KS!o^LPB+za`dcEKF zikF%_y}iv24oAr;gkzHl3|jH+~vf_N*;;UKI#7wRf}!l`n38FU@>+ zKT#>;ttcJ>U#kf~ zdne@5ogqKyF7%ePdR;i|b)k9P>HQbS`_&n_MY;uNwj=o3t+ZsSIu>78g(T;yxyIZ+ zTQ5;zWrk7%0J_B4Xu^*kuPEX4Ka3QK6a9_}!8J5Av^Wz66SQHU!ht+?DCKxQR4lm_Y%d?%65dElOC!_Fc#F8l=ff@?ayaICSxfLJ42Tc?iosu;*Odi* zCI*p)%?RuUvId&Bxohy3lq{SS#UVY;x6UF+YYhza_Dy0h^U-A5nMEZPMwdU9OUfpu zH=AY~JdNs^nQuE*m{g?0Gz?|Siw*nB&gr%k0!sH>Bj8Ui%Q?H4nVAi2TrDgnOHbp4 z<%&zELUz=8B4)ZSB3gu(r!YxGvxkuFwEO?8He>N8(GJUA}?kgAh7#X z0`fta(`f!iH(&qy$`A5u zQ$}@`^_CZn;nVE(9LMhU9Q7KFb$j@eoQx?~g&>#T$`dIk`P*?LNfi;ZPQDw6(NQW1 z7mBMPzZEo3LHG9wGo6k0cxNv+w>|OMK3|q?@;Eoe5?LVoto@c88BWByVRC^YI*Cg9 zNr~@Pe9Fg?r^|Wl(GsrBKIhsKP=p@ng}RP6i*!FWb}sI`VLEO)uf@!tMz5J3`re{J zKU~O5djoDkm)YoH;G37RiRtV$X;zvte*lgJk^rEBMkn7Ep`&~w($6n#dt9DBH?8w_ z9M+^umzW-2$)m}t)}?5CNW^s3fk#*(@J@r|%gLki{_|BHiH4$bqt2-N6$&BBTZ?YW zZb^G?%RKkY@-A#ZTurJ2nfG$hTNA1SR-Y`El}1*7e@r*kTD)W^(0I1;x3=0(KQWn~ z7mlb(KZ%#D>(!(JwQYforHRHhk<&SWJC$Bvi)xM}F8cjw8e^Mjq%AN=1b$>(p~T54 z_j20)0Q~^E0l$c`f(Yo96lpp5x65#i9hU)9{Sz4pmEI^8dQlexl$i#1X}Ab0+~I`y zb+Y6giovNXo%0ccoO>ZZpHCY&O+I6?9oNLRicC-Zs9T*B7`N1i&34U3%m^>fVxk2k zUpSfHJrm##-Iz);9KavB-j?q22{QY4=57~LZW^#|zK#e2H~V0dzH#<#9HohY50oO| z(BJmLfgPqvM>U6a+ER>Tsu`^7FNb;$zUeuZCTr&}Hjh{;_y0R-n*+s#haW4{t(rpg zr00jHpS0l_DC z+Plg$%K9)}U6#cEOscuqGYuTfH-*z6Rb*fOF9W1i?lFLzXbdw`{;&WFtZqGT%f%G{-O z|1c?oAehZIr&$2sUK)b||79lR;T=g-LdsekZSMQ+>lF+F10DoYecgsFz6w}q*v(k* zwp}*)6PvE>yMcdWkj-MrU4c+3^gR*J`)yI~Qsyf-eJ zi%rN%kzOCpLS9OHCq>Z_x&-g;0m7LsL%u}jTq`BTAfvCZk0iezQ(QO9Wylw3+ibbnnQ#XgCybV|s>{GG8t%3FL;H8ta!avVmlVqi%60{vaGqMbFl8gV((d0DzXtbwcRaLZ(`? z-nZ4U2<;z*M^;S*^9&3M84SB_aE+e`6SzLsIFP%>U!Aw`tFdHUCms2WvY-nR!=ey+ z;V_%9YE;Nkz3mb0%Pw8+m7C?30ik6@yb*BUJYqF)1C4JA8$~LtP5GXGtB+dZe&trnE`J0{l&X95WQHtOz}SQJ~;V|b?=&FzHwA-tPjd;94@Qdm<#BGVEbhLy%awGHO^dM}Nw7rcTz0}^%yn)Lx%$ty=ArZqIKIYQ<|RS-?& z;S6sj_^AIgca6Di+4m`nb5gCW&%Ix6zkqJUvJ%SnyDltzS)54ktYWTiC>%-6=2QS1 z;H_*KsG8j8G_+jEQ6)4Gsh_$J{SqFoCn}eg>5qQid=?y3uYCM#BBkHSPT~{#Nw!FN zB~?<>pST-=SoBmIOUqnQf!4cekR-Sj=p15dSbk7CigjK0O0QsHHBAF{b+S3h$^Fx! z1GrkDzpv|xS*HY?ul=6fKa*Y~0R1&Ea&{wn{KYeHV;n+KI)4AaaJn3-L2gfIB*Oyk zIWnMinJJf)xVQ;;O@QAKrgoLTl2$2CaoBpFe%9u!?3OCbTeHLmzt>k0P}Ikb-b_eK zv4m+o6jzw)w5qj>Wgb?lmF_x+XXv{!m-4y0Y{ZX^g#T^b&84&U2M*OQ}YEP)TLJbR1E)Fxb?#-dTmHRe<~V&KT$nHIP~mn6?xHxhuggn z>dyO6ooRt(GWKuXc7@q~mfLxnKQnhS0Ve$mC2eyUxC368(S!TaD+=mIB6JkI37b_# zT7)su8V0GF{DibiC+wBiyvvToYj9?#CqIY?Bn0IX_P1mNbieC(D44%9{LvEjut%?~*W`$bm5e2+1|Esk$Y`*idy3I4x_dVX0mK3AbBdy71WCB=Rz>2D(+ zz=*<(HPN=iTt?$#lxyZa0Ijp0jBW~(#^8F7jaxtGd44cgHMw#i`mLJhkHCk_r)Ah$ z|KJ(}y$8I8Zy?>$RCI3%(^HxJu7j&>4YG8wTv7*NO}W-x#y|6=eqo=DWTkAHr5~Yn zM_e=7gP?H1eB-$VszQ4`lmhfHN4?+3kC)J>{)EGgPoK!R<{!tuxP^w+EL10k#Nfs* zW}O@pJa4g460&A1N*~>AVC(qcK+bM*ETbG)g5NcPA|Hid+klV{>CHR3gvz}Z4Kvm( z2SOM=u}*=F%A262LmiHM7tAUjm2Eg4@K9-a^rMt>&;Uh2Zd=hjqlLa@vc!sgm+8v5 zXN}N}KmHNk-2c2cAFJQ|A-($lYbKS?)ljzUt8$~S3~=iPQ&kuIOW4#X@loKmMar;( z?|*u_Cfy*A;Hq$ufOp%b72&-?%p0}EZmL+5j;uhwq7rHG@KC%4J^<7oa%`{W2#RKR zd72sz|FR%y( zD~tarM9;@e&fZt~SLnme5amB+DcJwGyLc}aLM&K`l-(l#fpijsLM-(KJ$`|cWz&%xzc}#)bbVw5oE#8Jjc^qd1?-Szjy$#Mirm;G{5S&C MRkR Date: Mon, 23 Jun 2025 16:14:05 -0500 Subject: [PATCH 6/7] Update docs/quickstart/cli.md Co-authored-by: Sung Won Chung --- docs/quickstart/cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index 93386987e3..db94464f9b 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -428,7 +428,7 @@ Lines 12-14 show the progress and completion of the second step - executing mode Lines 16-18 show the progress and completion of the final step - virtually updating the plan's target environment, which makes the data available for querying. -??? "Learn more about the plan's actions" +??? "Learn more about the plan's actions: `sqlmesh plan --explain`" Before applying a plan, you can view a detailed description of the actions it will take by passing the explain flag in your `sqlmesh plan` command: From 7e5976898019584c07af5bda1cdc7fab895c421d Mon Sep 17 00:00:00 2001 From: Trey Spiller Date: Mon, 23 Jun 2025 16:16:58 -0500 Subject: [PATCH 7/7] Move explain section up --- docs/quickstart/cli.md | 146 ++++++++++++++++++++--------------------- 1 file changed, 73 insertions(+), 73 deletions(-) diff --git a/docs/quickstart/cli.md b/docs/quickstart/cli.md index db94464f9b..c89da59b75 100644 --- a/docs/quickstart/cli.md +++ b/docs/quickstart/cli.md @@ -288,6 +288,78 @@ SQLMesh's key actions are creating and applying *plans* to *environments*. At th After SQLMesh creates a plan, it summarizes the breaking and non-breaking changes so you can understand what will happen if you apply the plan. It will prompt you to "backfill" data to apply the plan. (In this context, backfill is a generic term for updating or adding to a table's data, including an initial load or full refresh.) +??? info "Learn more about a plan's actions: `sqlmesh plan --explain`" + + Before applying a plan, you can view a detailed description of the actions it will take by passing the explain flag in your `sqlmesh plan` command: + + ```bash + sqlmesh plan --explain + ``` + + Passing the explain flag for the quickstart example project above adds the following information to the output: + + ```bash + Explained plan + ├── Validate SQL and create physical layer tables and views if they do not exist + │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ │ ├── Dry run model query without inserting results + │ │ └── Create table if it doesn't exist + │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ │ ├── Dry run model query without inserting results + │ │ └── Create table if it doesn't exist + │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + │ ├── Dry run model query without inserting results + │ └── Create table if it doesn't exist + ├── Backfill models by running their queries and run standalone audits + │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ │ └── Fully refresh table + │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ │ ├── Fully refresh table + │ │ └── Run 'assert_positive_order_ids' audit + │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + │ └── Fully refresh table + └── Update the virtual layer for environment 'prod' + └── Create or update views in the virtual layer to point at new physical tables and views + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 + ``` + + The explanation has three top-level sections, corresponding to the three types of actions a plan takes: + + - Validate SQL and create physical layer tables and views if they do not exist + - Backfill models by running their queries and run standalone audits + - Update the virtual layer for environment 'prod' + + Each section lists the affected models and provides more information about what will occur. For example, the first model in the first section is: + + ```bash + ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 + │ ├── Dry run model query without inserting results + │ └── Create table if it doesn't exist + ``` + + The first line shows the model name `sqlmesh_example.seed_model` and the physical layer table SQLMesh will create to store its data: `db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172`. The second and third lines tell us that in this step SQLMesh will dry-run the model query and create the physical layer table if it doesn't exist. + + The second section describes what will occur during the backfill step. The second model in this section is: + + ```bash + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + │ ├── Fully refresh table + │ └── Run 'assert_positive_order_ids' audit + ``` + + The first line shows the model name `sqlmesh_example.full_model` and the physical layer table SQLMesh will insert the model's data into: `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. The second and third lines tell us that the backfill action will fully refresh the model's physical table and run the `assert_positive_order_ids` audit. + + The final section describes SQLMesh's action during the virtual layer update step. The first model in this section is: + + ```bash + └── Create or update views in the virtual layer to point at new physical tables and views + ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 + ``` + + The virtual layer step will update the `sqlmesh_example.full_model` virtual layer view to `SELECT * FROM` the physical table `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. + The first SQLMesh plan must execute every model to populate the production environment. Running `sqlmesh plan` will generate the plan and the following output: ```bash linenums="1" @@ -428,79 +500,7 @@ Lines 12-14 show the progress and completion of the second step - executing mode Lines 16-18 show the progress and completion of the final step - virtually updating the plan's target environment, which makes the data available for querying. -??? "Learn more about the plan's actions: `sqlmesh plan --explain`" - - Before applying a plan, you can view a detailed description of the actions it will take by passing the explain flag in your `sqlmesh plan` command: - - ```bash - sqlmesh plan --explain - ``` - - Passing the explain flag for the quickstart example project above adds the following information to the output: - - ```bash - Explained plan - ├── Validate SQL and create physical layer tables and views if they do not exist - │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 - │ │ ├── Dry run model query without inserting results - │ │ └── Create table if it doesn't exist - │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 - │ │ ├── Dry run model query without inserting results - │ │ └── Create table if it doesn't exist - │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 - │ ├── Dry run model query without inserting results - │ └── Create table if it doesn't exist - ├── Backfill models by running their queries and run standalone audits - │ ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 - │ │ └── Fully refresh table - │ ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 - │ │ ├── Fully refresh table - │ │ └── Run 'assert_positive_order_ids' audit - │ └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 - │ └── Fully refresh table - └── Update the virtual layer for environment 'prod' - └── Create or update views in the virtual layer to point at new physical tables and views - ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 - ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 - └── sqlmesh_example.incremental_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__incremental_model__1880815781 - ``` - - The explanation has three top-level sections, corresponding to the three types of actions a plan takes: - - - Validate SQL and create physical layer tables and views if they do not exist - - Backfill models by running their queries and run standalone audits - - Update the virtual layer for environment 'prod' - - Each section lists the affected models and provides more information about what will occur. For example, the first model in the first section is: - - ```bash - ├── sqlmesh_example.seed_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172 - │ ├── Dry run model query without inserting results - │ └── Create table if it doesn't exist - ``` - - The first line shows the model name `sqlmesh_example.seed_model` and the physical layer table SQLMesh will create to store its data: `db.sqlmesh__sqlmesh_example.sqlmesh_example__seed_model__2185867172`. The second and third lines tell us that in this step SQLMesh will dry-run the model query and create the physical layer table if it doesn't exist. - - The second section describes what will occur during the backfill step. The second model in this section is: - - ```bash - ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 - │ ├── Fully refresh table - │ └── Run 'assert_positive_order_ids' audit - ``` - - The first line shows the model name `sqlmesh_example.full_model` and the physical layer table SQLMesh will insert the model's data into: `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. The second and third lines tell us that the backfill action will fully refresh the model's physical table and run the `assert_positive_order_ids` audit. - - The final section describes SQLMesh's action during the virtual layer update step. The first model in this section is: - - ```bash - └── Create or update views in the virtual layer to point at new physical tables and views - ├── sqlmesh_example.full_model -> db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865 - ``` - - The virtual layer step will update the `sqlmesh_example.full_model` virtual layer view to `SELECT * FROM` the physical table `db.sqlmesh__sqlmesh_example.sqlmesh_example__full_model__2278521865`. - -Let's take a quick look at the project's DuckDB database file to see the objects SQLMesh created. First, we open the built-in DuckDB CLI tool with the `duckdb db.db` command, then run our two queries: +Let's take a quick look at the project's DuckDB database file to see the objects SQLMesh created. First, we open the built-in DuckDB CLI tool with the `duckdb db.db` command, then run our two queries. Our first query shows the three physical tables SQLMesh created in the `sqlmesh__sqlmesh_example` schema (one table for each model):