Skip to content

Commit 01d3b88

Browse files
committed
more changes
1 parent 766695d commit 01d3b88

1 file changed

Lines changed: 29 additions & 127 deletions

File tree

docs/examples/sqlmesh_cli_crash_course.md

Lines changed: 29 additions & 127 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,17 @@
11
# SQLMesh CLI Crash Course
22

3-
This doc is designed to get you intimate with 90% of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
3+
This doc is designed to get you intimate with a **majority** of the SQLMesh commands you’ll use to build *and* maintain data pipelines. The goal is after 30 minutes, using SQLMesh becomes muscle memory. This is designed to live on your second monitor or in a side by side window, so you can swiftly copy/paste into your terminal.
44

55
This is designed based on community observations, face to face conversations, live screenshares, and debugging sessions. This is *not* an exhaustive list, but it is an earnest one.
66

77
You can follow along in this: [open source github repo](https://github.com/sungchun12/sqlmesh-cli-crash-course)
88

99
## **Development Workflow**
1010

11-
You’ll use these commands 80% of the time because this is how you apply code changes.
12-
13-
The workflow is as follows:
11+
You’ll use these commands 80% of the time because this is how you apply code changes. The workflow is as follows:
1412

1513
1. Make changes to your models directly in SQL and python files (pre-made in examples below)
16-
2. Plan the changes in your dev environment (assumes a prod environment exists-see quickstart)
14+
2. Plan the changes in your dev environment
1715
3. Apply the changes to your dev environment
1816
4. Audit the changes (test data quality)
1917
5. Run data diff against prod
@@ -37,6 +35,12 @@ All these steps are bundled into a single command below:
3735
sqlmesh plan <environment>
3836
```
3937

38+
If you want to move faster, you can add the `--auto-apply` flag to avoid the manual prompt.
39+
40+
```bash
41+
sqlmesh plan <environment> --auto-apply
42+
```
43+
4044
=== "Tobiko Cloud"
4145

4246
```bash
@@ -47,6 +51,12 @@ All these steps are bundled into a single command below:
4751
tcloud sqlmesh plan <environment>
4852
```
4953

54+
If you want to move faster, you can add the `--auto-apply` flag to avoid the manual prompt.
55+
56+
```bash
57+
tcloud sqlmesh plan <environment> --auto-apply
58+
```
59+
5060
??? "Example Output"
5161
I made a breaking change to `incremental_model` and `full_model`.
5262

@@ -227,9 +237,9 @@ Run data diff against prod. This is a good way to verify the changes are behavin
227237
└─────────┴──────┴─────┘
228238
```
229239

230-
Apply the changes to prod. This step is recommended only in CICD as best practice.
231-
232-
For learning purposes and hot fixes, you can apply the changes to prod by entering `y` at the prompt.
240+
!!! warning "Apply the changes to prod"
241+
This step is recommended **only in CI/CD** as best practice. # TODO: link to github cicd bot setup
242+
For learning purposes and hot fixes, you can apply the changes to prod by entering `y` at the prompt.
233243

234244
=== "SQLMesh"
235245

@@ -312,129 +322,35 @@ For learning purposes and hot fixes, you can apply the changes to prod by enteri
312322
```
313323

314324
---
315-
316-
=== "SQLMesh"
317-
318-
```bash
319-
sqlmesh plan dev --start 2025-01-01 --end now
320-
```
321-
322-
=== "Tobiko Cloud"
323-
324-
```bash
325-
tcloud sqlmesh plan dev --start 2025-01-01 --end now
326-
```
327-
328-
asdf
329-
330-
=== "SQLMesh"
331-
332-
```bash
333-
sqlmesh --debug plan dev
334-
```
335-
336-
=== "Tobiko Cloud"
337-
338-
```bash
339-
tcloud sqlmesh --debug plan dev
340-
```
341-
342-
asdf
343-
=== "SQLMesh"
344-
345-
```bash
346-
sqlmesh plan dev --auto-apply
347-
```
348-
349-
=== "Tobiko Cloud"
350-
351-
```bash
352-
tcloud sqlmesh plan dev --auto-apply # - good for iterating faster
353-
```
354-
355-
asdf
356-
357-
=== "SQLMesh"
358-
359-
```bash
360-
sqlmesh plan dev --dry-run
361-
```
362-
363-
=== "Tobiko Cloud"
364-
365-
```bash
366-
tcloud sqlmesh plan dev --dry-run
367-
```
368-
369-
asdf
370-
=== "SQLMesh"
371-
372-
```bash
373-
sqlmesh plan dev --diff-rendered
374-
```
375-
376-
=== "Tobiko Cloud"
377-
378-
```bash
379-
tcloud sqlmesh plan dev --diff-rendered
380-
```
381-
382-
asdf
383-
=== "SQLMesh"
384-
385-
```bash
386-
sqlmesh plan dev --no-diff
387-
```
388-
389-
=== "Tobiko Cloud"
390-
391-
```bash
392-
tcloud sqlmesh plan dev --no-diff
393-
```
394-
395-
asdf
396325

397-
=== "SQLMesh"
326+
## **Enhanced Testing Workflow**
398327

399-
```bash
400-
sqlmesh plan dev --empty-backfill
401-
```
328+
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
402329

403-
=== "Tobiko Cloud"
404-
405-
```bash
406-
tcloud sqlmesh plan dev --empty-backfill
407-
```
330+
1. Create external models outside of SQLMesh's control (ex: data loaded in by Fivetran, Airbyte, etc.)
331+
2. Automatically generate unit tests
332+
3. Ad hoc query the data directly in the CLI
408333

409-
asdf
410334

411335
=== "SQLMesh"
412336

413337
```bash
414-
sqlmesh plan dev --forward-only
338+
sqlmesh create_external_models
415339
```
416340

417341
=== "Tobiko Cloud"
418342

419343
```bash
420-
tcloud sqlmesh plan dev --forward-only
344+
tcloud sqlmesh create_external_models
421345
```
422346

423-
asdf
424-
425-
=== "SQLMesh"
426-
427-
```bash
428-
sqlmesh create_external_models
429-
```
347+
## **Debugging Workflow**
430348

431-
=== "Tobiko Cloud"
349+
You'll use these commands ad hoc to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to evolve into to feel confident about the changes. The workflow is as follows:
432350

433-
```bash
434-
tcloud sqlmesh create_external_models
435-
```
351+
1. Render the model to verify the SQL is looking as expected
352+
2. Run in debug mode to verify SQLMesh's behavior.
436353

437-
asdf
438354

439355
## **Run Commands**
440356

@@ -507,20 +423,6 @@ You'll these commands ad hoc to verify your changes are behaving as expected.
507423

508424
asdf
509425

510-
=== "SQLMesh"
511-
512-
```bash
513-
sqlmesh table_diff ajwioejfioajowe
514-
```
515-
516-
=== "Tobiko Cloud"
517-
518-
```bash
519-
tcloud sqlmesh table_diff ajwioejfioajowe
520-
```
521-
522-
asdf
523-
524426
=== "SQLMesh"
525427

526428
```bash

0 commit comments

Comments
 (0)