You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/examples/sqlmesh_cli_crash_course.md
+17-9Lines changed: 17 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -328,17 +328,19 @@ Run data diff against prod. This is a good way to verify the changes are behavin
328
328
329
329
## **Enhanced Testing Workflow**
330
330
331
-
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your changes. The workflow is as follows:
331
+
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your testing. The workflow is as follows:
332
332
333
333
1. Create and audit external models outside of SQLMesh's control (ex: data loaded in by Fivetran, Airbyte, etc.)
334
334
2. Automatically generate unit tests
335
335
3. Ad hoc query the data directly in the CLI
336
-
4.Add linting
336
+
4.Lint your models to catch known syntax errors
337
337
338
338
---
339
339
340
340
### Create and Audit External Models
341
341
342
+
SQLMesh will automatically parse fully qualified table/view names that are outside of SQLMesh's control (ex: `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`). You can add audits to test data quality. If they fail, SQLMesh prevents downstream models from wastefully running.
343
+
342
344
=== "SQLMesh"
343
345
344
346
```bash
@@ -354,7 +356,7 @@ You'll use these commands to validate your changes are behaving as expected. Aud
354
356
??? "Example Output"
355
357
Note: this is an example from a separate Tobiko Cloud project, so you can't following along in the github repo above.
356
358
357
-
- Generated external models from the `bigquery-public-data` dataset.
359
+
- Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` tabled parsed in the model's SQL.
358
360
- I added an audit to the external model to ensure `event_date` is not null.
359
361
- Viewed a plan preview of the changes that will be made to the external model.
360
362
@@ -474,6 +476,8 @@ You'll use these commands to validate your changes are behaving as expected. Aud
474
476
475
477
### Automatically Generate Unit Tests
476
478
479
+
This ensures your business logic is working as expected with static sample data. This is great for testing complex business logic (ex: `CASE WHEN` conditions). No need to write them manually neither!
480
+
477
481
=== "SQLMesh"
478
482
479
483
```bash
@@ -579,9 +583,9 @@ You'll use these commands to validate your changes are behaving as expected. Aud
579
583
580
584
### Run Ad Hoc Queries
581
585
582
-
This is great to validate the look and feel of your changes without context switching to your query console.
586
+
You can run live queries directly from the CLI. This is great to validate the look and feel of your changes without context switching to your query console.
583
587
584
-
Pro tip: run this after running `sqlmesh table_diff` to get a full picture of the changes.
588
+
Pro tip: run this after running `sqlmesh table_diff` to get a full picture of your changes.
585
589
586
590
=== "SQLMesh"
587
591
@@ -676,7 +680,7 @@ This is great to catch issues before wasting runtime in your data warehouse. You
676
680
677
681
## **Debugging Workflow**
678
682
679
-
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
683
+
You'll use these commands ad hoc to validate your changes are behaving as expected. This is great to get more details beyond the defaults above. The workflow is as follows:
680
684
681
685
1. Render the model to verify the SQL is looking as expected
682
686
2. Run in verbose mode to verify SQLMesh's behavior.
@@ -771,7 +775,7 @@ This is great to verify the SQL is looking as expected before applying the chang
771
775
772
776
### Apply Plan Changes in Verbose Mode
773
777
774
-
This is useful to see exactly what SQLMesh is doing in the physical and virtual layers. After, you can copy/paste the fully qualified table/view name into your query console to validate the data (if that's your preference).
778
+
You can see detailed operations in the physical and virtual layers. This is useful to see exactly what SQLMesh is doing every step. After, you can copy/paste the fully qualified table/view name into your query console to validate the data (if that's your preference).
775
779
776
780
=== "SQLMesh"
777
781
@@ -852,7 +856,7 @@ This is useful to see exactly what SQLMesh is doing in the physical and virtual
852
856
853
857
### View Logs Easily
854
858
855
-
Each time you perform a SQLMesh command, it creates a log file in the `logs` directory. This is useful to see what exact queries were executed to apply your changes. Admittedly, this is outside of native functionality, but it's a quick and easy way to view the logs.
859
+
Each time you perform a SQLMesh command, it creates a log file in the `logs` directory. You can view them by manually navigating to the correct file name with latest timestamp or with this simple shell command. This is useful to see what exact queries were executed to apply your changes. Admittedly, this is outside of native functionality, but it's a quick and easy way to view logs.
856
860
857
861
```bash
858
862
# install this open source tool that enhances the default `cat` command
@@ -901,7 +905,7 @@ bat --theme='ansi' $(ls -t logs/ | head -n 1 | sed 's/^/logs\//')
901
905
902
906
## **Run on Production Schedule**
903
907
904
-
If you're using open source SQLMesh, you can run this in your orchestrator (ex: Dagster, GitHub Actions, etc.) every 5 minutes or at your lowest model cron schedule (ex: every 1 hour). Don't worry! It will only run executions that need to be run.
908
+
If you're using open source SQLMesh, you can run this command in your orchestrator (ex: Dagster, GitHub Actions, etc.) every 5 minutes or at your lowest model cron schedule (ex: every 1 hour). Don't worry! It will only run executions that need to be run.
905
909
906
910
If you're using Tobiko Cloud, this configures automatically without additional configuration.
907
911
@@ -961,6 +965,8 @@ This command is intended be run on a schedule. It will skip the physical and vir
961
965
962
966
### Run Models with Incomplete Intervals (Warning)
963
967
968
+
You can run models that execute backfills each time you invoke a run whether ad hoc or on a schedule.
969
+
964
970
!!! warning "Run Models with Incomplete Intervals"
965
971
This only applies to incremental models that have `allow_partials` set to `true`.
966
972
This is generally not recommended for production environments as you risk shipping incomplete data which will be perceived as broken data.
@@ -1080,6 +1086,8 @@ When you apply the plan to `prod` after the dev worfklow, it will NOT backfill h
1080
1086
```
1081
1087
1082
1088
1089
+
`sqlmesh plan`
1090
+
1083
1091
When this is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models in development.
0 commit comments