Skip to content

Commit e347de5

Browse files
author
Sung Won Chung
committed
update prose
1 parent fcd6f7e commit e347de5

1 file changed

Lines changed: 17 additions & 9 deletions

File tree

docs/examples/sqlmesh_cli_crash_course.md

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -328,17 +328,19 @@ Run data diff against prod. This is a good way to verify the changes are behavin
328328

329329
## **Enhanced Testing Workflow**
330330

331-
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your changes. The workflow is as follows:
331+
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your testing. The workflow is as follows:
332332

333333
1. Create and audit external models outside of SQLMesh's control (ex: data loaded in by Fivetran, Airbyte, etc.)
334334
2. Automatically generate unit tests
335335
3. Ad hoc query the data directly in the CLI
336-
4. Add linting
336+
4. Lint your models to catch known syntax errors
337337

338338
---
339339

340340
### Create and Audit External Models
341341

342+
SQLMesh will automatically parse fully qualified table/view names that are outside of SQLMesh's control (ex: `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`). You can add audits to test data quality. If they fail, SQLMesh prevents downstream models from wastefully running.
343+
342344
=== "SQLMesh"
343345

344346
```bash
@@ -354,7 +356,7 @@ You'll use these commands to validate your changes are behaving as expected. Aud
354356
??? "Example Output"
355357
Note: this is an example from a separate Tobiko Cloud project, so you can't following along in the github repo above.
356358

357-
- Generated external models from the `bigquery-public-data` dataset.
359+
- Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` tabled parsed in the model's SQL.
358360
- I added an audit to the external model to ensure `event_date` is not null.
359361
- Viewed a plan preview of the changes that will be made to the external model.
360362

@@ -474,6 +476,8 @@ You'll use these commands to validate your changes are behaving as expected. Aud
474476

475477
### Automatically Generate Unit Tests
476478

479+
This ensures your business logic is working as expected with static sample data. This is great for testing complex business logic (ex: `CASE WHEN` conditions). No need to write them manually neither!
480+
477481
=== "SQLMesh"
478482

479483
```bash
@@ -579,9 +583,9 @@ You'll use these commands to validate your changes are behaving as expected. Aud
579583

580584
### Run Ad Hoc Queries
581585

582-
This is great to validate the look and feel of your changes without context switching to your query console.
586+
You can run live queries directly from the CLI. This is great to validate the look and feel of your changes without context switching to your query console.
583587

584-
Pro tip: run this after running `sqlmesh table_diff` to get a full picture of the changes.
588+
Pro tip: run this after running `sqlmesh table_diff` to get a full picture of your changes.
585589

586590
=== "SQLMesh"
587591

@@ -676,7 +680,7 @@ This is great to catch issues before wasting runtime in your data warehouse. You
676680

677681
## **Debugging Workflow**
678682

679-
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
683+
You'll use these commands ad hoc to validate your changes are behaving as expected. This is great to get more details beyond the defaults above. The workflow is as follows:
680684

681685
1. Render the model to verify the SQL is looking as expected
682686
2. Run in verbose mode to verify SQLMesh's behavior.
@@ -771,7 +775,7 @@ This is great to verify the SQL is looking as expected before applying the chang
771775

772776
### Apply Plan Changes in Verbose Mode
773777

774-
This is useful to see exactly what SQLMesh is doing in the physical and virtual layers. After, you can copy/paste the fully qualified table/view name into your query console to validate the data (if that's your preference).
778+
You can see detailed operations in the physical and virtual layers. This is useful to see exactly what SQLMesh is doing every step. After, you can copy/paste the fully qualified table/view name into your query console to validate the data (if that's your preference).
775779

776780
=== "SQLMesh"
777781

@@ -852,7 +856,7 @@ This is useful to see exactly what SQLMesh is doing in the physical and virtual
852856

853857
### View Logs Easily
854858

855-
Each time you perform a SQLMesh command, it creates a log file in the `logs` directory. This is useful to see what exact queries were executed to apply your changes. Admittedly, this is outside of native functionality, but it's a quick and easy way to view the logs.
859+
Each time you perform a SQLMesh command, it creates a log file in the `logs` directory. You can view them by manually navigating to the correct file name with latest timestamp or with this simple shell command. This is useful to see what exact queries were executed to apply your changes. Admittedly, this is outside of native functionality, but it's a quick and easy way to view logs.
856860

857861
```bash
858862
# install this open source tool that enhances the default `cat` command
@@ -901,7 +905,7 @@ bat --theme='ansi' $(ls -t logs/ | head -n 1 | sed 's/^/logs\//')
901905

902906
## **Run on Production Schedule**
903907

904-
If you're using open source SQLMesh, you can run this in your orchestrator (ex: Dagster, GitHub Actions, etc.) every 5 minutes or at your lowest model cron schedule (ex: every 1 hour). Don't worry! It will only run executions that need to be run.
908+
If you're using open source SQLMesh, you can run this command in your orchestrator (ex: Dagster, GitHub Actions, etc.) every 5 minutes or at your lowest model cron schedule (ex: every 1 hour). Don't worry! It will only run executions that need to be run.
905909

906910
If you're using Tobiko Cloud, this configures automatically without additional configuration.
907911

@@ -961,6 +965,8 @@ This command is intended be run on a schedule. It will skip the physical and vir
961965

962966
### Run Models with Incomplete Intervals (Warning)
963967

968+
You can run models that execute backfills each time you invoke a run whether ad hoc or on a schedule.
969+
964970
!!! warning "Run Models with Incomplete Intervals"
965971
This only applies to incremental models that have `allow_partials` set to `true`.
966972
This is generally not recommended for production environments as you risk shipping incomplete data which will be perceived as broken data.
@@ -1080,6 +1086,8 @@ When you apply the plan to `prod` after the dev worfklow, it will NOT backfill h
10801086
```
10811087

10821088

1089+
`sqlmesh plan`
1090+
10831091
When this is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models in development.
10841092

10851093
```bash

0 commit comments

Comments
 (0)