You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/plans.md
+17-14Lines changed: 17 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -336,6 +336,12 @@ You may restate external models. An [external model](./models/external_models.md
336
336
337
337
The plan's `--start` and `--end` date options determine which data intervals will be reprocessed. Some model kinds cannot be backfilled for limited date ranges, though - learn more [below](#model-kind-limitations).
338
338
339
+
!!! info "Just catching up"
340
+
341
+
Restatement plans "catch models up" to the latest time interval already processed in the environment. They cannot process additional intervals because the required data has not yet been processed upstream.
342
+
343
+
If you pass an `--end` date later than the environment's most recent time interval, SQLMesh will just catch up to the environment and will ignore any additional intervals.
344
+
339
345
To prevent models from ever being restated, set the [disable_restatement](models/overview.md#disable_restatement) attribute to `true`.
340
346
341
347
<a name="restatement-examples"></a>
@@ -376,40 +382,37 @@ These examples demonstrate how to select which models to restate based on model
376
382
377
383
Restatement plans behave differently depending on if you're targeting the `prod` environment or a [development environment](./environments.md#how-to-use-environments).
378
384
379
-
If you target a development environment like so:
385
+
If you target a development environment by including an environment name like `dev`:
380
386
381
387
```bash
382
388
sqlmesh plan dev --restate-model "db.model_a" --start "2024-01-01" --end "2024-01-10"
383
389
```
384
390
385
-
the restatement plan will restate the requested intervals for the specified model in the `dev` environment. Versions of the model in other environments will be unaffected.
391
+
the restatement plan will restate the requested intervals forthe specified modelin the `dev` environment. In other environments, the model will be unaffected.
386
392
387
-
However, if you target the `prod` environment:
393
+
However, if you target the `prod` environment by omitting an environment name:
388
394
389
395
```bash
390
396
sqlmesh plan --restate-model "db.model_a" --start "2024-01-01" --end "2024-01-10"
391
397
```
392
398
393
-
the restatement plan will restate the intervals in the `prod` table *and clear the intervals from state for every other version of that model*.
399
+
the restatement plan will restate the intervals in the `prod` table *and clear the model's time intervals from state in every other environment*.
394
400
395
-
This means that next time you do a run in `dev`, those intervals will be restated in the development environment as well.
396
-
397
-
The reason for this is to prevent old data from getting promoted to `prod`. One of the benefits of SQLMesh is being able to [reuse tables](#virtual-update) from development environments to ensure that production deployments consist of quick, painless pointer swaps.
398
-
399
-
!!! info
400
-
If restating data in `prod` did not also trigger a restatement in `dev`, when `sqlmesh plan` is run against `prod` to deploy changes, a table containing old data may be promoted.
401
+
The next time you do a run in `dev`, the intervals already reprocessed in `prod` are reprocessed in `dev` as well. This is to prevent old data from getting promoted to `prod` in the future.
401
402
402
403
This behavior also clears the affected intervals for downstream tables that only exist in development environments. Consider the following example:
403
404
404
405
- Table `A` exists in `prod`
405
406
- A virtual environment `dev` is created with new tables `B` and `C` downstream of `A`
406
407
- the DAG in `prod` looks like `A`
407
408
- the DAG in `dev` looks like `A <- B <- C`
408
-
- A restatement plan is created against table `A` in `prod`
409
-
- SQLMesh will ensure that the affected intervals are also cleared for `B` and `C` in `dev` even though those tables do not exist in `prod`
409
+
- A restatement plan is executed against table `A` in `prod`
410
+
- SQLMesh will clear the affected intervals for `B` and `C` in `dev` even though those tables do not exist in `prod`
410
411
411
412
!!! info "Bringing development environments up to date"
412
413
413
-
If a restatement plan against `prod` cleared intervals from state for tables in development environments, you need to `sqlmesh run <env>` to trigger the reprocessing of that data.
414
+
A restatement plan against `prod` clears time intervals from state for models in development environments, but it does not trigger a run to reprocess those intervals.
415
+
416
+
Execute `sqlmesh run <environment name>` to trigger reprocessing in the development environment.
414
417
415
-
This is because SQLMesh limits the work done in the `prod` restatement plan to just the `prod` environment. That way the restatement can be applied as quickly as possible and avoid doing unnecessary work.
418
+
This is necessary because a `prod` restatement plan only does work in the `prod` environment for speed and efficiency.
0 commit comments