Skip to content

Commit c0d8ec6

Browse files
sungchun12sungchun12
committed
narrative design
1 parent cc86a97 commit c0d8ec6

1 file changed

Lines changed: 160 additions & 4 deletions

File tree

‎docs/examples/sqlmesh_cli_crash_course.md‎

Lines changed: 160 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,181 @@
22

33
This doc is designed to get you intimate with 90% of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
44

5-
This is *not* an exhaustive list, but it is an earnest one.
5+
This is designed based on community observations, face to face conversations, live screenshares, and debugging sessions. This is *not* an exhaustive list, but it is an earnest one.
66

7-
## **Development Commands**
7+
You can follow along in this: [open source github repo](https://github.com/sungchun12/sqlmesh-cli-crash-course)
88

9-
You’ll use these commands 80% of the time because this is how you apply code changes.
9+
## **Development Workflow**
10+
11+
You’ll use these commands 80% of the time because this is how you apply code changes.
12+
13+
The workflow is as follows:
14+
15+
1. Make changes to your models directly in SQL and python files (pre-made in examples below)
16+
2. Plan the changes in your dev environment (assumes a prod environment exists-see quickstart)
17+
3. Apply the changes to your dev environment
18+
4. Audit the changes (test data quality)
19+
5. Run data diff against prod
20+
6. Apply the changes to prod
21+
22+
---
23+
24+
All these steps are bundled into a single command below:
25+
26+
- Plan the changes in your dev environment.
27+
- Apply the changes to your dev environment by entering `y` at the prompt.
28+
- Audit the changes (test data quality). This happens automatically when you apply the changes to dev.
1029

1130
=== "SQLMesh"
1231

1332
```bash
1433
sqlmesh plan dev
1534
```
1635

36+
```bash
37+
sqlmesh plan <environment>
38+
```
39+
1740
=== "Tobiko Cloud"
1841

1942
```bash
2043
tcloud sqlmesh plan dev
2144
```
2245

23-
asdf
46+
```bash
47+
tcloud sqlmesh plan <environment>
48+
```
49+
50+
??? "Example Output"
51+
I made a big change to the `incremental_model` and `full_model`.
52+
53+
- Showed me the models impacted by the changes.
54+
- Showed me the changes that will be made to the models.
55+
- Showed me the models that need to be backfilled.
56+
- Prompted me to apply the changes to `dev`.
57+
- Showed me the audit failures that raise as warnings.
58+
- Updated the physical layer to validate the SQL.
59+
- Executed the model batches by inserting the data into the physical layer.
60+
- Updated the virtual layer with view pointers to reflect the changes.
61+
62+
```bash
63+
Differences from the `dev` environment:
64+
65+
Models:
66+
├── Directly Modified:
67+
│ ├── sqlmesh_example__dev.incremental_model
68+
│ └── sqlmesh_example__dev.full_model
69+
└── Indirectly Modified:
70+
└── sqlmesh_example__dev.view_model
71+
72+
---
73+
74+
+++
75+
76+
@@ -9,7 +9,8 @@
77+
78+
SELECT
79+
item_id,
80+
COUNT(DISTINCT id) AS num_orders,
81+
- 6 AS new_column
82+
+ new_column
83+
FROM sqlmesh_example.incremental_model
84+
GROUP BY
85+
- item_id
86+
+ item_id,
87+
+ new_column
88+
89+
Directly Modified: sqlmesh_example__dev.full_model (Breaking)
90+
91+
---
92+
93+
+++
94+
95+
@@ -15,7 +15,7 @@
96+
97+
id,
98+
item_id,
99+
event_date,
100+
- 5 AS new_column
101+
+ 7 AS new_column
102+
FROM sqlmesh_example.seed_model
103+
WHERE
104+
event_date BETWEEN @start_date AND @end_date
105+
106+
Directly Modified: sqlmesh_example__dev.incremental_model (Breaking)
107+
└── Indirectly Modified Children:
108+
└── sqlmesh_example__dev.view_model (Indirect Breaking)
109+
Models needing backfill:
110+
├── sqlmesh_example__dev.full_model: [full refresh]
111+
├── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-16]
112+
└── sqlmesh_example__dev.view_model: [recreate view]
113+
Apply - Backfill Tables [y/n]: y
114+
115+
Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 2/2 • 0:00:00
116+
117+
✔ Physical layer updated
118+
119+
[1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-16] 0.03s
120+
Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0% • pending • 0:00:00
121+
sqlmesh_example__dev.incremental_model .
122+
[WARNING] sqlmesh_example__dev.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
123+
/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
124+
[1/1] sqlmesh_example__dev.full_model [full refresh, audits ❌1] 0.01s
125+
Executing model batches ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.3% • 1/3 • 0:00:00
126+
sqlmesh_example__dev.full_model .
127+
[WARNING] sqlmesh_example__dev.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
128+
/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
129+
[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔2 ❌1] 0.01s
130+
Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
131+
132+
✔ Model batches executed
133+
134+
Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
135+
136+
✔ Virtual layer updated
137+
```
138+
139+
140+
Run data diff against prod. This is a good way to verify the changes are behaving as expected **after** applying them to `dev`.
141+
142+
=== "SQLMesh"
143+
144+
```bash
145+
sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
146+
```
147+
148+
```bash
149+
sqlmesh table_diff <environment>:<environment> <model_name> --show-sample
150+
```
151+
152+
153+
=== "Tobiko Cloud"
154+
155+
```bash
156+
tcloud sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
157+
```
158+
159+
```bash
160+
tcloud sqlmesh table_diff <environment>:<environment> <model_name> --show-sample
161+
```
162+
163+
Apply the changes to prod. This step is recommended only in CICD as best practice.
164+
165+
For learning purposes and hot fixes, you can apply the changes to prod by entering `y` at the prompt.
166+
167+
=== "SQLMesh"
168+
169+
```bash
170+
sqlmesh plan
171+
```
172+
173+
=== "Tobiko Cloud"
174+
175+
```bash
176+
tcloud sqlmesh plan
177+
```
178+
179+
---
24180

25181
=== "SQLMesh"
26182

0 commit comments

Comments
 (0)