Skip to content

Commit 54642d5

Browse files
sungchun12sungchun12
committed
add forward only workflow
1 parent 45e539a commit 54642d5

1 file changed

Lines changed: 134 additions & 2 deletions

File tree

‎docs/examples/sqlmesh_cli_crash_course.md‎

Lines changed: 134 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -994,9 +994,141 @@ This command is intended be run on a schedule. It will skip the physical and vir
994994

995995
### **Forward-Only Development Workflow**
996996

997-
https://www.loom.com/share/209181d9532d44969313ac0ac23f501f
997+
This is an advanced workflow and specifically designed for extremely large incremental models that take a long time to run even during development. It solves for:
998998

999-
- codify this especially the commented out steps
999+
- Transformaing data with constant schema evolution in json and nested array data types.
1000+
- Retaining history of a calculated column and applying a new calculation to new rows going forward.
1001+
1002+
When you apply the plan to `prod` after the dev worfklow, it will NOT backfill historical data. It will only execute model batches **forward only** for new intervals (new rows).
1003+
1004+
=== "SQLMesh"
1005+
1006+
```bash
1007+
sqlmesh plan dev --forward-only
1008+
```
1009+
1010+
```bash
1011+
sqlmesh plan <environment> --forward-only
1012+
```
1013+
1014+
=== "Tobiko Cloud"
1015+
1016+
```bash
1017+
tcloud sqlmesh plan dev --forward-only
1018+
```
1019+
1020+
```bash
1021+
tcloud sqlmesh plan <environment> --forward-only
1022+
```
1023+
1024+
??? "Example Output"
1025+
1026+
- I applied a change to a new column
1027+
- It impacts 2 downstream models
1028+
- I enforced a forward-only plan to avoid backfilling historical data
1029+
- I previewed the changes in a clone of the models impacted (clones will NOT be reused in production)
1030+
1031+
```bash
1032+
Differences from the `dev` environment:
1033+
1034+
Models:
1035+
├── Directly Modified:
1036+
│ └── sqlmesh_example__dev.incremental_model
1037+
└── Indirectly Modified:
1038+
├── sqlmesh_example__dev.view_model
1039+
└── sqlmesh_example__dev.full_model
1040+
1041+
---
1042+
1043+
+++
1044+
1045+
@@ -16,7 +16,7 @@
1046+
1047+
id,
1048+
item_id,
1049+
event_date,
1050+
- 9 AS new_column
1051+
+ 10 AS new_column
1052+
FROM sqlmesh_example.seed_model
1053+
WHERE
1054+
event_date BETWEEN @start_date AND @end_date
1055+
1056+
Directly Modified: sqlmesh_example__dev.incremental_model (Forward-only)
1057+
└── Indirectly Modified Children:
1058+
├── sqlmesh_example__dev.full_model (Forward-only)
1059+
└── sqlmesh_example__dev.view_model (Forward-only)
1060+
Models needing backfill:
1061+
├── sqlmesh_example__dev.full_model: [full refresh] (preview) -- preview is a signal that a clone will be created if supported by the query engine
1062+
├── sqlmesh_example__dev.incremental_model: [2025-04-17 - 2025-04-17] (preview)
1063+
└── sqlmesh_example__dev.view_model: [recreate view] (preview)
1064+
Apply - Preview Tables [y/n]: y
1065+
1066+
Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
1067+
1068+
✔ Physical layer updated
1069+
1070+
[1/1] sqlmesh_example__dev.incremental_model [insert 2025-04-17 - 2025-04-17] 0.01s
1071+
[1/1] sqlmesh_example__dev.full_model [full refresh, audits ✔1] 0.01s
1072+
[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔3] 0.01s
1073+
Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
1074+
1075+
✔ Model batches executed
1076+
1077+
Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
1078+
1079+
✔ Virtual layer updated
1080+
```
1081+
1082+
1083+
When this is applied to `prod`, it will only execute model batches for new intervals (new rows). That's why you notice it only made a virtual layer update for this example.
1084+
1085+
```bash
1086+
Differences from the `prod` environment:
1087+
1088+
Models:
1089+
├── Directly Modified:
1090+
│ └── sqlmesh_example.incremental_model
1091+
└── Indirectly Modified:
1092+
├── sqlmesh_example.view_model
1093+
└── sqlmesh_example.full_model
1094+
1095+
---
1096+
1097+
+++
1098+
1099+
@@ -9,13 +9,14 @@
1100+
1101+
disable_restatement FALSE,
1102+
on_destructive_change 'ERROR'
1103+
),
1104+
- grains ((id, event_date))
1105+
+ grains ((id, event_date)),
1106+
+ allow_partials TRUE
1107+
)
1108+
SELECT
1109+
id,
1110+
item_id,
1111+
event_date,
1112+
- 7 AS new_column
1113+
+ 10 AS new_column
1114+
FROM sqlmesh_example.seed_model
1115+
WHERE
1116+
event_date BETWEEN @start_date AND @end_date
1117+
1118+
Directly Modified: sqlmesh_example.incremental_model (Forward-only)
1119+
└── Indirectly Modified Children:
1120+
├── sqlmesh_example.full_model (Forward-only)
1121+
└── sqlmesh_example.view_model (Forward-only)
1122+
Apply - Virtual Update [y/n]: y
1123+
1124+
SKIP: No physical layer updates to perform
1125+
1126+
SKIP: No model batches to execute
1127+
1128+
Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
1129+
1130+
✔ Virtual layer updated
1131+
```
10001132

10011133

10021134
### **Miscellaneous**

0 commit comments

Comments
 (0)