Feat: Allow virtual environments to be given dedicated catalogs

Author: erindru

docs/guides/configuration.md

@@ -244,7 +244,9 @@ This only applies to the _physical tables_ that SQLMesh creates - the views are
 
 SQLMesh stores `prod` environment views in the schema in a model's name - for example, the `prod` views for a model `my_schema.users` will be located in `my_schema`.
 
-By default, for non-prod environments SQLMesh creates a new schema that appends the environment name to the model name's schema. For example, by default the view for a model `my_schema.users` in a SQLMesh environment named `dev` will be located in the schema `my_schema__dev`.
+By default, for non-prod environments SQLMesh creates a new schema that appends the environment name to the model name's schema. For example, by default the view for a model `my_schema.users` in a SQLMesh environment named `dev` will be located in the schema `my_schema__dev` as `my_schema__dev.users`.
+
+##### Show at the table level instead
 
 This behavior can be changed to append a suffix at the end of a _table/view_ name instead. Appending the suffix to a table/view name means that non-prod environment views will be created in the same schema as the `prod` environment. The prod and non-prod views are differentiated by non-prod view names ending with `__<env>`.
 
@@ -260,7 +262,7 @@ Config example:
 
 === "Python"
 
-    The Python `environment_suffix_target` argument takes an `EnvironmentSuffixTarget` enumeration with a value of `EnvironmentSuffixTarget.TABLE` or `EnvironmentSuffixTarget.SCHEMA` (default).
+    The Python `environment_suffix_target` argument takes an `EnvironmentSuffixTarget` enumeration with a value of `EnvironmentSuffixTarget.TABLE`, `EnvironmentSuffixTarget.CATALOG` or `EnvironmentSuffixTarget.SCHEMA` (default).
 
     ```python linenums="1"
     from sqlmesh.core.config import Config, ModelDefaultsConfig, EnvironmentSuffixTarget
@@ -271,20 +273,68 @@ Config example:
     )
     ```
 
-The default behavior of appending the suffix to schemas is recommended because it leaves production with a single clean interface for accessing the views. However, if you are deploying SQLMesh in an environment with tight restrictions on schema creation then this can be a useful way of reducing the number of schemas SQLMesh uses.
+!!! info "Default behavior"
+    The default behavior of appending the suffix to schemas is recommended because it leaves production with a single clean interface for accessing the views. However, if you are deploying SQLMesh in an environment with tight restrictions on schema creation then this can be a useful way of reducing the number of schemas SQLMesh uses.
+
+##### Show at the catalog level instead
+
+If neither the schema (default) nor the table level are sufficient for your use case, you may indicate the environment at the catalog level instead.
+
+This can be useful if you have downstream BI reporting tools and you would like to point them at a development environment to test something out without renaming all the table / schema references within the report query.
+
+In order to achieve this, you may configure [environment_suffix_target](../reference/configuration.md#environments) like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    environment_suffix_target: catalog
+    ```
+
+=== "Python"
+
+    The Python `environment_suffix_target` argument takes an `EnvironmentSuffixTarget` enumeration with a value of `EnvironmentSuffixTarget.TABLE`, `EnvironmentSuffixTarget.CATALOG` or `EnvironmentSuffixTarget.SCHEMA` (default).
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, EnvironmentSuffixTarget
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        environment_suffix_target=EnvironmentSuffixTarget.CATALOG,
+    )
+    ```
+
+Given the example of a model called `my_schema.users`, this will cause the following behavior:
+
+- For the `prod` environment, the default catalog as configured in the gateway will be used
+- For any other environment, eg `dev`, the environment name will be used as the catalog
+
+Therefore, a model named `my_schema.users` in an environment called `dev` will have its view created as `dev.my_schema.users`.
+
+If you would like more control over the catalog name to use for a given environment, you may optionally configure [environment_catalog_mapping](#environment-view-catalogs) as described below.
+
+!!! warning "Caveats"
+    - Using `environment_suffix_target: catalog` only works on engines that support querying across different catalogs. If your engine does not support cross-catalog queries then you will need to use `environment_suffix_target: schema` or `environment_suffix_target: table` instead.
+    - SQLMesh will not attempt to create catalogs on demand or drop them as part of janitor cleanup. Using `environment_suffix_target: catalog` assumes the catalogs already exist in the target database and are being managed outside of SQLMesh.
+    - When using `environment_catalog_mapping`, you need to ensure that your regex patterns do not conflict. If you map multiple environments to the same catalog, SQLMesh will overwrite existing views. To prevent this, it's recommended to make use of the `@{environment_name}` placeholder to ensure that environment catalog names are always unique.
+
 
 #### Environment view catalogs
 
 By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.
 
-Some companies fully segregate `prod` and non-prod environment objects by catalog. For example, they might have a "prod" catalog that contains all `prod` environment physical tables and views and a separate "dev" catalog that contains all `dev` environment physical tables and views.
+It can be desirable to create `prod` and non-prod virtual layer objects in separate catalogs instead. For example, there might be a "prod" catalog that contains all `prod` environment views and a separate "dev" catalog that contains all `dev` environment views.
 
 Separate prod and non-prod catalogs can also be useful if you have a CI/CD pipeline that creates environments, like the [SQLMesh Github Actions CI/CD Bot](../integrations/github.md). You might want to store the CI/CD environment objects in a dedicated catalog since there can be many of them.
 
+!!! info "Virtual layer only"
+    Note that the following setting only affects the [virtual layer](../concepts/glossary.md#virtual-layer). If you need full segregation by catalog between environments in the [physical layer](../concepts/glossary.md#physical-layer) as well, see the [Isolated Systems Guide](../guides/isolated_systems.md).
+
 To configure separate catalogs, provide a mapping from [regex patterns](https://en.wikipedia.org/wiki/Regular_expression) to catalog names. SQLMesh will compare the name of an environment to the regex patterns; when it finds a match it will store the environment's objects in the corresponding catalog.
 
 SQLMesh evaluates the regex patterns in the order defined in the configuration; it uses the catalog for the first matching pattern. If no match is found, the catalog defined in the model or the default catalog defined on the connection will be used.
 
+In addition, you may specify an `@{environment_name}` placeholder in the mapping. It will be substituted with the target environment name at runtime.
+
 Config example:
 
 === "YAML"
@@ -294,6 +344,7 @@ Config example:
       '^prod$': prod
       '^dev.*': dev
       '^analytics_repo.*': cicd
+      '.*': 'user_@{environment_name}'
     ```
 
 === "Python"
@@ -307,6 +358,7 @@ Config example:
             '^prod$': 'prod',
             '^dev.*': 'dev',
             '^analytics_repo.*': 'cicd',
+            '.*': 'user_@{environment_name}'
         },
     )
     ```
@@ -316,6 +368,7 @@ With the example configuration above, SQLMesh would evaluate environment names a
 * If the environment name is `prod`, the catalog will be `prod`.
 * If the environment name starts with `dev`, the catalog will be `dev`.
 * If the environment name starts with `analytics_repo`, the catalog will be `cicd`.
+* If the environment name is anything else, eg `homer_simpson`, the catalog will be `user_homer_simpson`.
 
 *Note:* This feature is only available for engines that support querying across catalogs. At the time of writing, the following engines are **NOT** supported:
 
@@ -441,15 +494,15 @@ SELECT 2 AS col
     └── Directly Modified:
         └── sqlmesh_example__dev.test_model
 
-    ---                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
-    +++                                                                                                                                                                                                                                             
-                                                                                                                                                                                                                                                    
-                                                                                                
-    kind FULL                                                                                                                                                                                                                                    
-    )                                                                                                                                                                                                                                              
-    SELECT                                                                                                                                                                                                                                         
-    -  1 AS col                                                                                                                                                                                                                                     
-    +  2 AS col  
+    ---
+    +++
+
+
+    kind FULL
+    )
+    SELECT
+    -  1 AS col
+    +  2 AS col
     ```
 
 3. Second (metadata) change in `dev`:
@@ -469,27 +522,27 @@ SELECT 5 AS col
     └── Directly Modified:
         └── sqlmesh_example__dev.test_model
 
-    ---                                                                                                                                                                                                                                             
-                                                                                                                                                                                                                                                    
-    +++                                                                                                                                                                                                                                             
-                                                                                                                                                                                                                                                    
-    @@ -1,8 +1,9 @@                                                                                                                                                                                                                                 
-                                                                                                                                                                                                                                                    
-    MODEL (                                                                                                                                                                                                                                        
-    name sqlmesh_example.test_model,                                                                                                                                                                                                             
-    +  owner "John Doe",                                                                                                                                                                                                                                                                                                                                                                                                                                                        
-    kind FULL                                                                                                                                                                                                                                    
-    )                                                                                                                                                                                                                                              
-    SELECT                                                                                                                                                                                                                                         
-    -  1 AS col                                                                                                                                                                                                                                     
-    +  2 AS col                                                                                                                                                                                                                                     
+    ---
+
+    +++
+
+    @@ -1,8 +1,9 @@
+
+    MODEL (
+    name sqlmesh_example.test_model,
+    +  owner "John Doe",
+    kind FULL
+    )
+    SELECT
+    -  1 AS col
+    +  2 AS col
 
     Directly Modified: sqlmesh_example__dev.test_model (Breaking)
     Models needing backfill:
     └── sqlmesh_example__dev.test_model: [full refresh]
     ```
 
-Even though the second change should have been a metadata change (thus not requiring a backfill), it will still be classified as a breaking change because the comparison is against production instead of the previous development state. This is intentional and may cause additional backfills as more changes are accumulated.  
+Even though the second change should have been a metadata change (thus not requiring a backfill), it will still be classified as a breaking change because the comparison is against production instead of the previous development state. This is intentional and may cause additional backfills as more changes are accumulated.
 
 
 ### Gateways

examples/sushi/config.py

@@ -128,7 +128,7 @@
 )
 
 
-environment_suffix_config = Config(
+environment_suffix_table_config = Config(
     default_connection=DuckDBConnectionConfig(),
     model_defaults=model_defaults,
     environment_suffix_target=EnvironmentSuffixTarget.TABLE,
@@ -161,3 +161,7 @@
         ".*": "dev_catalog",
     },
 )
+
+environment_suffix_catalog_config = environment_catalog_mapping_config.model_copy(
+    update={"environment_suffix_target": EnvironmentSuffixTarget.CATALOG}
+)

sqlmesh/core/config/common.py

@@ -10,9 +10,22 @@
 
 
 class EnvironmentSuffixTarget(str, Enum):
+    # Intended to create virtual environments in their own schemas, with names like "<model_schema_name>__<env name>". The view name is untouched.
+    # For example, a model named 'sqlmesh_example.full_model' created in an environment called 'dev'
+    # would have its virtual layer view created as 'sqlmesh_example__dev.full_model'
     SCHEMA = "schema"
+
+    # Intended to create virtual environments in the same schema as their production counterparts by adjusting the table name.
+    # For example, a model named 'sqlmesh_example.full_model' created in an environment called 'dev'
+    # would have its virtual layer view created as "sqlmesh_example.full_model__dev"
     TABLE = "table"
 
+    # Intended to create virtual environments in their own catalogs to preserve the schema and view name of the models
+    # For example, a model named 'sqlmesh_example.full_model' created in an environment called 'dev'
+    # would have its virtual layer view created as "dev.sqlmesh_example.full_model"
+    # note: this only works for engines that can query across catalogs
+    CATALOG = "catalog"
+
     @property
     def is_schema(self) -> bool:
         return self == EnvironmentSuffixTarget.SCHEMA
@@ -21,6 +34,10 @@ def is_schema(self) -> bool:
     def is_table(self) -> bool:
         return self == EnvironmentSuffixTarget.TABLE
 
+    @property
+    def is_catalog(self) -> bool:
+        return self == EnvironmentSuffixTarget.CATALOG
+
     @classproperty
     def default(cls) -> EnvironmentSuffixTarget:
         return EnvironmentSuffixTarget.SCHEMA

sqlmesh/core/config/root.py

@@ -219,6 +219,23 @@ def _normalize_and_validate_fields(cls, data: t.Any) -> t.Any:
                 f"^{k}$": v for k, v in physical_schema_override.items()
             }
 
+        if (environment_suffix_target := data.get("environment_suffix_target")) and isinstance(
+            environment_suffix_target, str
+        ):
+            if (
+                environment_suffix_target.lower() == EnvironmentSuffixTarget.CATALOG.lower()
+                and "environment_catalog_mapping" not in data
+            ):
+                # set the default environment_catalog_mapping for when environment_suffix_target=catalog
+                # but no explicit mapping of environments to catalogs is set by the user
+                data["environment_catalog_mapping"] = {
+                    # no override for prod, use the default catalog on the connection
+                    # note that we cant pass None here without failing the Pydantic validator, so we pass a string constant that can be interpreted as None later
+                    f"^{c.PROD}$": c.SQLMESH_NONE,
+                    # every other environment overriden to a catalog with the same name
+                    ".*": "@{environment_name}",
+                }
+
         return data
 
     @model_validator(mode="after")

sqlmesh/core/constants.py

@@ -85,6 +85,7 @@
 SQLMESH_MACRO = "__sqlmesh__macro__"
 SQLMESH_BUILTIN = "__sqlmesh__builtin__"
 SQLMESH_METADATA = "__sqlmesh__metadata__"
+SQLMESH_NONE = "__sqlmesh_none__"
 
 
 BUILTIN = "builtin"

sqlmesh/core/environment.py

@@ -43,6 +43,10 @@ class EnvironmentNamingInfo(PydanticModel):
     normalize_name: bool = True
     gateway_managed: bool = False
 
+    @property
+    def is_dev(self) -> bool:
+        return self.name.lower() != c.PROD
+
     @field_validator("name", mode="before")
     @classmethod
     def _sanitize_name(cls, v: str) -> str:
@@ -89,8 +93,13 @@ def from_environment_catalog_mapping(
         construction_kwargs = dict(name=name, **kwargs)
         for re_pattern, catalog_name in environment_catalog_mapping.items():
             if re.match(re_pattern, name):
+                if catalog_name == c.SQLMESH_NONE:
+                    catalog_name_override = None
+                else:
+                    catalog_name_override = catalog_name.replace("@{environment_name}", name)
+
                 return cls(
-                    catalog_name_override=catalog_name,
+                    catalog_name_override=catalog_name_override,
                     **construction_kwargs,
                 )
         return cls(**construction_kwargs)