Skip to content

Commit a5ba92f

Browse files
sungchun12sungchun12
committed
add debugging draft
1 parent f934466 commit a5ba92f

1 file changed

Lines changed: 166 additions & 31 deletions

File tree

‎docs/examples/sqlmesh_cli_crash_course.md‎

Lines changed: 166 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This doc is designed to get you intimate with a **majority** of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
44

5-
This is designed based on community observations, face to face conversations, live screenshares, and debugging sessions. This is *not* an exhaustive list, but it is an earnest one.
5+
This is inspired by community observations, face to face conversations, live screenshares, and debugging sessions. This is *not* an exhaustive list, but it is an earnest one.
66

77
You can follow along in this: [open source github repo](https://github.com/sungchun12/sqlmesh-cli-crash-course)
88

@@ -590,8 +590,8 @@ Pro tip: run this after running `sqlmesh table_diff` to get a full picture of th
590590
```
591591

592592
```bash
593-
-- construct arbitrary query
594-
sqlmesh fetchdf "select * from <schema__environment>.<model_name> limit 5" -- double underscore is important. Not needed for prod.
593+
# construct arbitrary query
594+
sqlmesh fetchdf "select * from <schema__environment>.<model_name> limit 5" # double underscore is important. Not needed for prod.
595595
```
596596

597597
=== "Tobiko Cloud"
@@ -681,101 +681,236 @@ You'll use these commands ad hoc to validate your changes are behaving as expect
681681
1. Render the model to verify the SQL is looking as expected
682682
2. Run in debug mode to verify SQLMesh's behavior.
683683

684+
### Render your SQL Changes
684685

685-
## **Run Commands**
686+
This is great to verify the SQL is looking as expected before applying the changes. This is especially important if you're migrating from another query engine (ex: postgres to databricks).
686687

687688
=== "SQLMesh"
688689

689690
```bash
690-
sqlmesh run
691+
sqlmesh render sqlmesh_example.incremental_model
691692
```
692693

693-
=== "Tobiko Cloud"
694+
```bash
695+
sqlmesh render sqlmesh_example.incremental_model --dialect databricks
696+
```
694697

695698
```bash
696-
tcloud sqlmesh run
699+
sqlmesh render <model_name> --dialect <dialect>
697700
```
698701

699-
asdf
702+
=== "Tobiko Cloud"
700703

701-
=== "SQLMesh"
704+
```bash
705+
tcloud sqlmesh render sqlmesh_example.incremental_model
706+
```
702707

703708
```bash
704-
sqlmesh run dev
709+
tcloud sqlmesh render sqlmesh_example.incremental_model --dialect databricks
705710
```
706711

707-
=== "Tobiko Cloud"
712+
```bash
713+
tcloud sqlmesh render <model_name> --dialect <dialect>
714+
```
708715

709716
```bash
710-
tcloud sqlmesh run dev
717+
??? "Example Output"
718+
719+
It outputs the full SQL code in the default or target dialect.
720+
721+
```sql
722+
-- rendered sql in default dialect
723+
SELECT
724+
"seed_model"."id" AS "id",
725+
"seed_model"."item_id" AS "item_id",
726+
"seed_model"."event_date" AS "event_date",
727+
7 AS "new_column"
728+
FROM "db"."sqlmesh__sqlmesh_example"."sqlmesh_example__seed_model__3294646944" AS "seed_model" /*
729+
db.sqlmesh_example.seed_model */
730+
WHERE
731+
"seed_model"."event_date" <= CAST('1970-01-01' AS DATE) -- placeholder dates for date macros
732+
AND "seed_model"."event_date" >= CAST('1970-01-01' AS DATE)
711733
```
712734

713-
asdf
735+
```sql
736+
-- rendered sql in databricks dialect
737+
SELECT
738+
`seed_model`.`id` AS `id`,
739+
`seed_model`.`item_id` AS `item_id`,
740+
`seed_model`.`event_date` AS `event_date`,
741+
7 AS `new_column`
742+
FROM `db`.`sqlmesh__sqlmesh_example`.`sqlmesh_example__seed_model__3294646944` AS `seed_model` /*
743+
db.sqlmesh_example.seed_model */
744+
WHERE
745+
`seed_model`.`event_date` <= CAST('1970-01-01' AS DATE)
746+
AND `seed_model`.`event_date` >= CAST('1970-01-01' AS DATE)
747+
```
748+
749+
```sql
750+
-- original sqlmesh model code
751+
MODEL (
752+
name sqlmesh_example.incremental_model,
753+
kind INCREMENTAL_BY_TIME_RANGE (
754+
time_column event_date
755+
),
756+
start '2020-01-01',
757+
cron '@daily',
758+
grain (id, event_date)
759+
);
760+
761+
SELECT
762+
id,
763+
item_id,
764+
event_date,
765+
7 as new_column
766+
FROM
767+
sqlmesh_example.seed_model
768+
WHERE
769+
event_date BETWEEN @start_date AND @end_date
770+
```
771+
772+
### Apply Plan Changes in Verbose Mode
773+
774+
This is useful to see exactly what SQLMesh is doing in the physical and virtual layers. After, you can copy/paste the fully qualified table/view name into your query console to validate the data (if that's your preference).
714775

715776
=== "SQLMesh"
716777

717778
```bash
718-
sqlmesh run --ignore-cron
779+
sqlmesh plan dev -vv
780+
```
781+
782+
```bash
783+
sqlmesh plan <environment> -vv
719784
```
720785

721786
=== "Tobiko Cloud"
722787

723788
```bash
724-
tcloud sqlmesh run --ignore-cron
789+
tcloud sqlmesh plan dev -vv
725790
```
726791

727-
asdf
792+
```bash
793+
tcloud sqlmesh plan <environment> -vv
794+
```
728795

729-
- allow_partials
796+
??? "Example Output"
730797

731-
## **Working with Incremental Forward-Only Models**
798+
```bash
799+
[WARNING] Linter warnings for
800+
/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
801+
- nomissingaudits: Model `audits` must be configured to test data quality.
802+
[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
803+
- nomissingaudits: Model `audits` must be configured to test data quality.
804+
[WARNING] Linter warnings for
805+
/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
806+
- nomissingaudits: Model `audits` must be configured to test data quality.
807+
[WARNING] Linter warnings for
808+
/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
809+
- nomissingaudits: Model `audits` must be configured to test data quality.
732810

733-
https://www.loom.com/share/209181d9532d44969313ac0ac23f501f
811+
Differences from the `dev` environment:
734812

735-
- codify this especially the commented out steps
813+
Models:
814+
├── Directly Modified:
815+
│ └── db.sqlmesh_example__dev.incremental_model
816+
└── Indirectly Modified:
817+
├── db.sqlmesh_example__dev.full_model
818+
└── db.sqlmesh_example__dev.view_model
819+
820+
---
821+
822+
+++
823+
824+
@@ -15,7 +15,7 @@
825+
826+
id,
827+
item_id,
828+
event_date,
829+
- 9 AS new_column
830+
+ 7 AS new_column
831+
FROM sqlmesh_example.seed_model
832+
WHERE
833+
event_date BETWEEN @start_date AND @end_date
834+
835+
Directly Modified: db.sqlmesh_example__dev.incremental_model (Breaking)
836+
└── Indirectly Modified Children:
837+
├── db.sqlmesh_example__dev.full_model (Breaking)
838+
└── db.sqlmesh_example__dev.view_model (Indirect Breaking)
839+
Apply - Virtual Update [y/n]: y
736840

841+
SKIP: No physical layer updates to perform
737842

738-
## **Validating Changes**
843+
SKIP: No model batches to execute
739844

740-
You'll these commands ad hoc to verify your changes are behaving as expected.
845+
db.sqlmesh_example__dev.incremental_model updated # you'll notice that it's updated vs. promoted because we changed the existing view definition
846+
db.sqlmesh_example__dev.full_model updated
847+
db.sqlmesh_example__dev.view_model updated
848+
Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
849+
850+
✔ Virtual layer updated
851+
```
852+
853+
## **Run Commands**
741854

742855
=== "SQLMesh"
743856

744857
```bash
745-
sqlmesh render sqlmesh_example.incremental_model
858+
sqlmesh run
746859
```
747860

748861
=== "Tobiko Cloud"
749862

750863
```bash
751-
tcloud sqlmesh render sqlmesh_example.incremental_model
864+
tcloud sqlmesh run
752865
```
753866

754867
asdf
755868

869+
=== "SQLMesh"
756870

871+
```bash
872+
sqlmesh run dev
873+
```
757874

758-
asdf
759-
875+
=== "Tobiko Cloud"
760876

877+
```bash
878+
tcloud sqlmesh run dev
879+
```
761880

762881
asdf
763882

764-
## **Miscellaneous**
765-
766883
=== "SQLMesh"
767884

768885
```bash
769-
sqlmesh migrate
886+
sqlmesh run --ignore-cron
770887
```
771888

772889
=== "Tobiko Cloud"
773890

774891
```bash
775-
tcloud sqlmesh migrate
892+
tcloud sqlmesh run --ignore-cron
776893
```
777894

778-
- migrate state schemas
895+
asdf
896+
897+
- allow_partials
898+
899+
## **Working with Incremental Forward-Only Models**
900+
901+
https://www.loom.com/share/209181d9532d44969313ac0ac23f501f
902+
903+
- codify this especially the commented out steps
904+
905+
906+
## **Validating Changes**
907+
908+
You'll these commands ad hoc to verify your changes are behaving as expected.
909+
910+
911+
912+
913+
## **Miscellaneous**
779914

780915
=== "SQLMesh"
781916

0 commit comments

Comments
 (0)