Skip to content

Commit 759b404

Browse files
authored
Docs: clarify restatement plans catch up to environment intervals (#4112)
1 parent caa6f83 commit 759b404

1 file changed

Lines changed: 17 additions & 14 deletions

File tree

docs/concepts/plans.md

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,12 @@ You may restate external models. An [external model](./models/external_models.md
336336
337337
The plan's `--start` and `--end` date options determine which data intervals will be reprocessed. Some model kinds cannot be backfilled for limited date ranges, though - learn more [below](#model-kind-limitations).
338338
339+
!!! info "Just catching up"
340+
341+
Restatement plans "catch models up" to the latest time interval already processed in the environment. They cannot process additional intervals because the required data has not yet been processed upstream.
342+
343+
If you pass an `--end` date later than the environment's most recent time interval, SQLMesh will just catch up to the environment and will ignore any additional intervals.
344+
339345
To prevent models from ever being restated, set the [disable_restatement](models/overview.md#disable_restatement) attribute to `true`.
340346
341347
<a name="restatement-examples"></a>
@@ -376,40 +382,37 @@ These examples demonstrate how to select which models to restate based on model
376382
377383
Restatement plans behave differently depending on if you're targeting the `prod` environment or a [development environment](./environments.md#how-to-use-environments).
378384
379-
If you target a development environment like so:
385+
If you target a development environment by including an environment name like `dev`:
380386
381387
```bash
382388
sqlmesh plan dev --restate-model "db.model_a" --start "2024-01-01" --end "2024-01-10"
383389
```
384390
385-
the restatement plan will restate the requested intervals for the specified model in the `dev` environment. Versions of the model in other environments will be unaffected.
391+
the restatement plan will restate the requested intervals for the specified model in the `dev` environment. In other environments, the model will be unaffected.
386392
387-
However, if you target the `prod` environment:
393+
However, if you target the `prod` environment by omitting an environment name:
388394
389395
```bash
390396
sqlmesh plan --restate-model "db.model_a" --start "2024-01-01" --end "2024-01-10"
391397
```
392398
393-
the restatement plan will restate the intervals in the `prod` table *and clear the intervals from state for every other version of that model*.
399+
the restatement plan will restate the intervals in the `prod` table *and clear the model's time intervals from state in every other environment*.
394400
395-
This means that next time you do a run in `dev`, those intervals will be restated in the development environment as well.
396-
397-
The reason for this is to prevent old data from getting promoted to `prod`. One of the benefits of SQLMesh is being able to [reuse tables](#virtual-update) from development environments to ensure that production deployments consist of quick, painless pointer swaps.
398-
399-
!!! info
400-
If restating data in `prod` did not also trigger a restatement in `dev`, when `sqlmesh plan` is run against `prod` to deploy changes, a table containing old data may be promoted.
401+
The next time you do a run in `dev`, the intervals already reprocessed in `prod` are reprocessed in `dev` as well. This is to prevent old data from getting promoted to `prod` in the future.
401402
402403
This behavior also clears the affected intervals for downstream tables that only exist in development environments. Consider the following example:
403404
404405
- Table `A` exists in `prod`
405406
- A virtual environment `dev` is created with new tables `B` and `C` downstream of `A`
406407
- the DAG in `prod` looks like `A`
407408
- the DAG in `dev` looks like `A <- B <- C`
408-
- A restatement plan is created against table `A` in `prod`
409-
- SQLMesh will ensure that the affected intervals are also cleared for `B` and `C` in `dev` even though those tables do not exist in `prod`
409+
- A restatement plan is executed against table `A` in `prod`
410+
- SQLMesh will clear the affected intervals for `B` and `C` in `dev` even though those tables do not exist in `prod`
410411
411412
!!! info "Bringing development environments up to date"
412413
413-
If a restatement plan against `prod` cleared intervals from state for tables in development environments, you need to `sqlmesh run <env>` to trigger the reprocessing of that data.
414+
A restatement plan against `prod` clears time intervals from state for models in development environments, but it does not trigger a run to reprocess those intervals.
415+
416+
Execute `sqlmesh run <environment name>` to trigger reprocessing in the development environment.
414417
415-
This is because SQLMesh limits the work done in the `prod` restatement plan to just the `prod` environment. That way the restatement can be applied as quickly as possible and avoid doing unnecessary work.
418+
This is necessary because a `prod` restatement plan only does work in the `prod` environment for speed and efficiency.

0 commit comments

Comments
 (0)