You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This doc is designed to get you intimate with 90% of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
3
+
This doc is designed to get you intimate with a **majority** of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
4
4
5
5
This is designed based on community observations, face to face conversations, live screenshares, and debugging sessions. This is *not* an exhaustive list, but it is an earnest one.
6
6
7
7
You can follow along in this: [open source github repo](https://github.com/sungchun12/sqlmesh-cli-crash-course)
8
8
9
9
## **Development Workflow**
10
10
11
-
You’ll use these commands 80% of the time because this is how you apply code changes.
12
-
13
-
The workflow is as follows:
11
+
You’ll use these commands 80% of the time because this is how you apply code changes. The workflow is as follows:
14
12
15
13
1. Make changes to your models directly in SQL and python files (pre-made in examples below)
16
-
2. Plan the changes in your dev environment (assumes a prod environment exists-see quickstart)
14
+
2. Plan the changes in your dev environment
17
15
3. Apply the changes to your dev environment
18
16
4. Audit the changes (test data quality)
19
17
5. Run data diff against prod
@@ -37,6 +35,12 @@ All these steps are bundled into a single command below:
37
35
sqlmesh plan <environment>
38
36
```
39
37
38
+
If you want to move faster, you can add the `--auto-apply` flag to avoid the manual prompt.
39
+
40
+
```bash
41
+
sqlmesh plan <environment> --auto-apply
42
+
```
43
+
40
44
=== "Tobiko Cloud"
41
45
42
46
```bash
@@ -47,6 +51,12 @@ All these steps are bundled into a single command below:
47
51
tcloud sqlmesh plan <environment>
48
52
```
49
53
54
+
If you want to move faster, you can add the `--auto-apply` flag to avoid the manual prompt.
55
+
56
+
```bash
57
+
tcloud sqlmesh plan <environment> --auto-apply
58
+
```
59
+
50
60
??? "Example Output"
51
61
I made a breaking change to `incremental_model` and `full_model`.
52
62
@@ -227,9 +237,9 @@ Run data diff against prod. This is a good way to verify the changes are behavin
227
237
└─────────┴──────┴─────┘
228
238
```
229
239
230
-
Apply the changes to prod. This step is recommended only in CICD as best practice.
231
-
232
-
For learning purposes and hot fixes, you can apply the changes to prod by entering `y` at the prompt.
240
+
!!! warning "Apply the changes to prod"
241
+
This step is recommended **only in CI/CD** as best practice. # TODO: link to github cicd bot setup
242
+
For learning purposes and hot fixes, you can apply the changes to prod by entering `y` at the prompt.
233
243
234
244
=== "SQLMesh"
235
245
@@ -312,129 +322,35 @@ For learning purposes and hot fixes, you can apply the changes to prod by enteri
312
322
```
313
323
314
324
---
315
-
316
-
=== "SQLMesh"
317
-
318
-
```bash
319
-
sqlmesh plan dev --start 2025-01-01 --end now
320
-
```
321
-
322
-
=== "Tobiko Cloud"
323
-
324
-
```bash
325
-
tcloud sqlmesh plan dev --start 2025-01-01 --end now
326
-
```
327
-
328
-
asdf
329
-
330
-
=== "SQLMesh"
331
-
332
-
```bash
333
-
sqlmesh --debug plan dev
334
-
```
335
-
336
-
=== "Tobiko Cloud"
337
-
338
-
```bash
339
-
tcloud sqlmesh --debug plan dev
340
-
```
341
-
342
-
asdf
343
-
=== "SQLMesh"
344
-
345
-
```bash
346
-
sqlmesh plan dev --auto-apply
347
-
```
348
-
349
-
=== "Tobiko Cloud"
350
-
351
-
```bash
352
-
tcloud sqlmesh plan dev --auto-apply # - good for iterating faster
353
-
```
354
-
355
-
asdf
356
-
357
-
=== "SQLMesh"
358
-
359
-
```bash
360
-
sqlmesh plan dev --dry-run
361
-
```
362
-
363
-
=== "Tobiko Cloud"
364
-
365
-
```bash
366
-
tcloud sqlmesh plan dev --dry-run
367
-
```
368
-
369
-
asdf
370
-
=== "SQLMesh"
371
-
372
-
```bash
373
-
sqlmesh plan dev --diff-rendered
374
-
```
375
-
376
-
=== "Tobiko Cloud"
377
-
378
-
```bash
379
-
tcloud sqlmesh plan dev --diff-rendered
380
-
```
381
-
382
-
asdf
383
-
=== "SQLMesh"
384
-
385
-
```bash
386
-
sqlmesh plan dev --no-diff
387
-
```
388
-
389
-
=== "Tobiko Cloud"
390
-
391
-
```bash
392
-
tcloud sqlmesh plan dev --no-diff
393
-
```
394
-
395
-
asdf
396
325
397
-
=== "SQLMesh"
326
+
## **Enhanced Testing Workflow**
398
327
399
-
```bash
400
-
sqlmesh plan dev --empty-backfill
401
-
```
328
+
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
402
329
403
-
=== "Tobiko Cloud"
404
-
405
-
```bash
406
-
tcloud sqlmesh plan dev --empty-backfill
407
-
```
330
+
1. Create external models outside of SQLMesh's control (ex: data loaded in by Fivetran, Airbyte, etc.)
331
+
2. Automatically generate unit tests
332
+
3. Ad hoc query the data directly in the CLI
408
333
409
-
asdf
410
334
411
335
=== "SQLMesh"
412
336
413
337
```bash
414
-
sqlmesh plan dev --forward-only
338
+
sqlmesh create_external_models
415
339
```
416
340
417
341
=== "Tobiko Cloud"
418
342
419
343
```bash
420
-
tcloud sqlmesh plan dev --forward-only
344
+
tcloud sqlmesh create_external_models
421
345
```
422
346
423
-
asdf
424
-
425
-
=== "SQLMesh"
426
-
427
-
```bash
428
-
sqlmesh create_external_models
429
-
```
347
+
## **Debugging Workflow**
430
348
431
-
=== "Tobiko Cloud"
349
+
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
432
350
433
-
```bash
434
-
tcloud sqlmesh create_external_models
435
-
```
351
+
1. Render the model to verify the SQL is looking as expected
352
+
2. Run in debug mode to verify SQLMesh's behavior.
436
353
437
-
asdf
438
354
439
355
## **Run Commands**
440
356
@@ -507,20 +423,6 @@ You'll these commands ad hoc to verify your changes are behaving as expected.
0 commit comments