Skip to content

Commit 5652c7a

Browse files
fresioASfresioAS
authored
Merge branch 'main' into add_fabric_warehouse
2 parents d28a781 + 803f7d8 commit 5652c7a

95 files changed

Lines changed: 4563 additions & 937 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/agents/qa-reviewer.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
name: qa-reviewer
3+
description: Use this agent PROACTIVELY when you need to analyze a PR or code changes to provide structured QA testing guidance for human QA testers. This agent reviews PRs and provides specific testing scenarios, example projects to use, commands to run, and validation steps. Examples: <example>Context: A developer just implemented virtual environment isolation for SQLMesh. user: 'I just added support for isolated virtual environments in SQLMesh' assistant: 'Let me use the qa-reviewer agent to create comprehensive QA testing instructions for this feature' <commentary>Since a significant feature was implemented, use the qa-reviewer agent to provide structured testing guidance for QA.</commentary></example> <example>Context: A PR adds a new SQL engine adapter. user: 'Here's the PR that adds BigQuery support to SQLMesh' assistant: 'I'll use the qa-reviewer agent to analyze this change and create QA test scenarios' <commentary>Since a new engine adapter was added, use the qa-reviewer agent to provide testing guidance specific to engine adapters.</commentary></example>
4+
tools: Glob, Grep, LS, Read, NotebookRead, WebFetch, TodoWrite, WebSearch, Bash
5+
model: sonnet
6+
color: green
7+
---
8+
9+
You are a QA Test Specialist with deep expertise in SQLMesh's architecture, testing methodologies, and quality assurance practices. You specialize in analyzing code changes and providing comprehensive, structured testing guidance for human QA testers.
10+
11+
Your core responsibilities:
12+
13+
## Analysis Approach
14+
15+
- Review PRs and code changes to understand the scope and impact of modifications
16+
- Identify all components, features, and workflows that could be affected by the changes
17+
- Consider edge cases, integration points, and potential failure scenarios
18+
- Map changes to existing example projects and testing workflows
19+
- Provide specific, actionable testing instructions that non-developers can follow
20+
- MUST write full instructions to the `plans/` folder with the filename of `<pr_number>_<short description>.md` so they can be reviewed and executed by QA testers
21+
22+
## QA Test Plan Structure
23+
24+
Organize your QA recommendations into clear, actionable sections:
25+
26+
### **Change Summary**
27+
- Brief description of what was changed and why
28+
- Key components and files modified
29+
- Potential impact areas and affected workflows
30+
31+
### **Test Environment Setup**
32+
- Which example project(s) to use for testing (e.g., `examples/sushi/`, `examples/sushi_dbt/`)
33+
- Any necessary environment configuration or setup steps
34+
- Required tools, databases, or dependencies
35+
36+
### **Core Test Scenarios**
37+
- Step-by-step testing procedures with specific commands
38+
- Expected results and success criteria for each test
39+
- Validation commands to confirm expected behavior
40+
- Screenshots or output examples where helpful
41+
42+
### **Edge Case Testing**
43+
- Boundary conditions and error scenarios to test
44+
- Negative test cases and expected failure modes
45+
- Cross-platform considerations (Windows/Linux/macOS)
46+
- Performance and scalability considerations
47+
48+
### **Regression Testing**
49+
- Existing functionality that should be retested
50+
- Critical workflows that must continue working
51+
- Backward compatibility scenarios
52+
53+
### **Integration Testing**
54+
- Cross-component testing scenarios
55+
- Multi-engine testing when relevant
56+
- dbt integration testing if applicable
57+
- UI/CLI integration points
58+
59+
## Example Project Guidance
60+
61+
Provide specific guidance on:
62+
- Which `examples/` project best demonstrates the feature
63+
- How to modify example projects for comprehensive testing
64+
- Custom test scenarios using real-world-like data
65+
- Commands to set up test scenarios and validate results
66+
67+
## Command Examples
68+
69+
Always provide:
70+
- Exact CLI commands to run tests
71+
- Configuration file modifications needed
72+
- Environment variable settings
73+
- Database setup commands when applicable
74+
- Validation queries or commands to check results
75+
76+
## Testing Best Practices
77+
78+
- Focus on user-facing functionality and workflows
79+
- Include both happy path and error scenarios
80+
- Provide clear success/failure criteria
81+
- Consider different user personas (data analysts, engineers, platform teams)
82+
- If the change doesn't have engine specific logic in it, prefer to test against duckdb since that is easiest
83+
- Include performance and scalability considerations
84+
- DO NOT have a step which is running an existing test - these tests are automatically run in CI and should not be duplicated in manual testing instructions
85+
- Assume all example projects are already tested as is and don't suggest doing a test which is running them again
86+
- All tests MUST just use `sqlmesh` cli commands - do not use the Python API. The goal is to run tests that mimic what an actual user would do, which is using the CLI.
87+
- A common pattern could be using the `sqlmesh` cli command and then running a Python script to validate the database or state is in an expected state, but the Python script should not be a test itself, just a validation step.
88+
89+
## Communication Style
90+
91+
- Use clear, numbered steps for testing procedures
92+
- Provide exact commands that can be copy-pasted
93+
- Include expected outputs and how to interpret results
94+
- Explain the "why" behind each test scenario
95+
- Use language accessible to QA testers who may not be developers
96+
- Organize content with clear headings and bullet points
97+
98+
## Important Constraints
99+
100+
- You NEVER write or modify code - you only analyze and provide testing guidance
101+
- You focus on user-facing functionality and workflows
102+
- You always provide specific, actionable testing steps
103+
- You consider the full user journey and realistic usage scenarios
104+
- You validate that your recommendations align with SQLMesh's architecture and patterns
105+
106+
When analyzing changes, assume you're looking at a recent PR or set of code modifications. Focus on providing comprehensive testing guidance that ensures the changes work correctly, don't break existing functionality, and provide a good user experience across different scenarios and environments.

.circleci/get_scm_version.py renamed to .github/scripts/get_scm_version.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
from setuptools_scm import get_version
22

3-
version = get_version(root='../', relative_to=__file__)
3+
version = get_version(root='../../', relative_to=__file__)
44
print(version.split('+')[0])

‎.github/workflows/private-repo-test.yaml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: Private Repo Testing
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
8+
concurrency:
9+
group: 'private-test-${{ github.event.pull_request.number }}'
10+
cancel-in-progress: true
11+
12+
permissions:
13+
contents: read
14+
15+
jobs:
16+
trigger-private-test:
17+
runs-on: ubuntu-latest
18+
steps:
19+
- name: Checkout code
20+
uses: actions/checkout@v4
21+
with:
22+
fetch-depth: 0
23+
- name: Set up Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: '3.12'
27+
- name: Install uv
28+
uses: astral-sh/setup-uv@v6
29+
- name: Set up Node.js for UI build
30+
uses: actions/setup-node@v4
31+
with:
32+
node-version: '20'
33+
- name: Install pnpm
34+
uses: pnpm/action-setup@v4
35+
with:
36+
version: latest
37+
- name: Install UI dependencies
38+
run: pnpm install
39+
- name: Build UI
40+
run: pnpm --prefix web/client run build
41+
- name: Install Python dependencies
42+
run: |
43+
python -m venv .venv
44+
source .venv/bin/activate
45+
pip install build twine setuptools_scm
46+
- name: Generate development version
47+
id: version
48+
run: |
49+
source .venv/bin/activate
50+
# Generate a PEP 440 compliant unique version including run attempt
51+
BASE_VERSION=$(python .github/scripts/get_scm_version.py)
52+
COMMIT_SHA=$(git rev-parse --short HEAD)
53+
# Use PEP 440 compliant format: base.devN+pr.sha.attempt
54+
UNIQUE_VERSION="${BASE_VERSION}+pr${{ github.event.pull_request.number }}.${COMMIT_SHA}.run${{ github.run_attempt }}"
55+
echo "version=$UNIQUE_VERSION" >> $GITHUB_OUTPUT
56+
echo "Generated unique version with run attempt: $UNIQUE_VERSION"
57+
- name: Build package
58+
env:
59+
SETUPTOOLS_SCM_PRETEND_VERSION: ${{ steps.version.outputs.version }}
60+
run: |
61+
source .venv/bin/activate
62+
python -m build
63+
- name: Configure PyPI for private repository
64+
env:
65+
TOBIKO_PRIVATE_PYPI_URL: ${{ secrets.TOBIKO_PRIVATE_PYPI_URL }}
66+
TOBIKO_PRIVATE_PYPI_KEY: ${{ secrets.TOBIKO_PRIVATE_PYPI_KEY }}
67+
run: ./.circleci/update-pypirc.sh
68+
- name: Publish to private PyPI
69+
run: |
70+
source .venv/bin/activate
71+
python -m twine upload -r tobiko-private dist/*
72+
- name: Publish Python Tests package
73+
env:
74+
SETUPTOOLS_SCM_PRETEND_VERSION: ${{ steps.version.outputs.version }}
75+
run: |
76+
source .venv/bin/activate
77+
unset TWINE_USERNAME TWINE_PASSWORD && make publish-tests
78+
- name: Get GitHub App token
79+
id: get_token
80+
uses: actions/create-github-app-token@v1
81+
with:
82+
private-key: ${{ secrets.TOBIKO_RENOVATE_BOT_PRIVATE_KEY }}
83+
app-id: ${{ secrets.TOBIKO_RENOVATE_BOT_APP_ID }}
84+
owner: ${{ secrets.PRIVATE_REPO_OWNER }}
85+
- name: Trigger private repository workflow
86+
uses: convictional/trigger-workflow-and-wait@v1.6.5
87+
with:
88+
owner: ${{ secrets.PRIVATE_REPO_OWNER }}
89+
repo: ${{ secrets.PRIVATE_REPO_NAME }}
90+
github_token: ${{ steps.get_token.outputs.token }}
91+
workflow_file_name: ${{ secrets.PRIVATE_WORKFLOW_FILE }}
92+
client_payload: |
93+
{
94+
"package_version": "${{ steps.version.outputs.version }}",
95+
"pr_number": "${{ github.event.pull_request.number }}"
96+
}

‎.github/workflows/release_shared_js.yaml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: Release shared js code
2+
on:
3+
workflow_dispatch:
4+
inputs:
5+
version:
6+
description: 'Version to release (e.g., 1.0.0)'
7+
required: true
8+
type: string
9+
permissions:
10+
id-token: write
11+
contents: read
12+
jobs:
13+
release:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: Checkout repository
17+
uses: actions/checkout@v4
18+
- name: Check branch is main
19+
run: |
20+
if [[ "${{ github.ref }}" != "refs/heads/main" ]]; then
21+
echo "Error: This workflow can only be run from the main branch"
22+
exit 1
23+
fi
24+
echo "Branch check passed: running from main branch"
25+
- name: Validate version format
26+
run: |
27+
version="${{ github.event.inputs.version }}"
28+
if ! [[ $version =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$ ]]; then
29+
echo "Error: Version must be a valid semantic version (e.g., 1.0.0, 1.0.0-beta.1, 1.0.0+build.1)"
30+
exit 1
31+
fi
32+
echo "Version format is valid: $version"
33+
- name: Setup Node.js
34+
uses: actions/setup-node@v4
35+
with:
36+
node-version: '20'
37+
registry-url: 'https://registry.npmjs.org'
38+
- name: Update npm
39+
run: npm install -g npm@latest
40+
- name: Print npm version
41+
run: npm --version
42+
- name: Install pnpm
43+
uses: pnpm/action-setup@v4
44+
with:
45+
version: 10
46+
- name: Install dependencies
47+
run: pnpm install --frozen-lockfile
48+
- name: Update package.json version
49+
working-directory: web/shared_ui
50+
run: |
51+
npm version ${{ github.event.inputs.version }} --no-git-tag-version
52+
- name: Build package
53+
working-directory: web/shared_ui
54+
run: pnpm run build
55+
- name: Publish to npm
56+
working-directory: web/shared_ui
57+
run: |
58+
npm publish

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ repos:
77
language: python
88
types_or: [python, pyi]
99
require_serial: true
10-
files: &files ^(sqlmesh/|tests/|web/|examples/|setup.py)
10+
files: &files ^(sqlmesh/|sqlmesh_dbt/|tests/|web/|examples/|setup.py)
1111
- id: ruff-format
1212
name: ruff-format
1313
entry: ruff format --force-exclude --line-length 100

‎docs/concepts/plans.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -43,23 +43,24 @@ This is a common choice in scenarios such as an addition of a new column, an act
4343

4444
If any downstream models contain a `select *` from the model, SQLMesh attempts to infer breaking status on a best-effort basis. We recommend explicitly specifying a query's columns to avoid unnecessary recomputation.
4545

46-
### Forward-only change
47-
A modified (either directly or indirectly) model that is categorized as forward-only will continue to use the existing physical table once the change is deployed to production (the `prod` environment). This means that no backfill will take place.
48-
49-
While iterating on forward-only changes in the development environment, the model's output will be stored in either a temporary table or a shallow clone of the production table if supported by the engine.
50-
51-
In either case the data produced this way in the development environment can only be used for preview and will **not** be reused once the change is deployed to production. See [Forward-only Plans](#forward-only-plans) for more details.
52-
53-
This category is assigned by SQLMesh automatically either when a user opts into using a [forward-only plan](#forward-only-plans) or when a model is explicitly configured to be forward-only.
54-
5546
### Summary
5647

5748
| Change Category | Change Type | Behaviour |
5849
|--------------------------------------|--------------------------------------------------------------------------------------------|----------------------------------------------------|
5950
| [Breaking](#breaking-change) | [Direct](glossary.md#direct-modification) or [Indirect](glossary.md#indirect-modification) | [Backfill](glossary.md#backfill) |
6051
| [Non-breaking](#non-breaking-change) | [Direct](glossary.md#direct-modification) | [Backfill](glossary.md#backfill) |
6152
| [Non-breaking](#non-breaking-change) | [Indirect](glossary.md#indirect-modification) | [No Backfill](glossary.md#backfill) |
62-
| [Forward-only](#forward-only-change) | [Direct](glossary.md#direct-modification) or [Indirect](glossary.md#indirect-modification) | [No Backfill](glossary.md#backfill), schema change |
53+
54+
## Forward-only change
55+
In addition to categorizing a change as breaking or non-breaking, it can also be classified as forward-only.
56+
57+
A model change classified as forward-only will continue to use the existing physical table once the change is deployed to production (the `prod` environment). This means that no backfill will take place.
58+
59+
While iterating on forward-only changes in the development environment, the model's output will be stored in either a temporary table or a shallow clone of the production table if supported by the engine.
60+
61+
In either case the data produced this way in the development environment can only be used for preview and will **not** be reused once the change is deployed to production. See [Forward-only Plans](#forward-only-plans) for more details.
62+
63+
This category is assigned by SQLMesh automatically either when a user opts into using a [forward-only plan](#forward-only-plans) or when a model is explicitly configured to be forward-only.
6364

6465
## Plan application
6566
Once a plan has been created and reviewed, it is then applied to the target [environment](environments.md) in order for its changes to take effect.

‎docs/guides/configuration.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,44 @@ sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00__dev
538538

539539
This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
540540

541+
#### Virtual Data Environment Modes
542+
543+
By default, Virtual Data Environments (VDE) are applied across both development and production environments. This allows SQLMesh to reuse physical tables when appropriate, even when promoting from development to production.
544+
545+
However, users may prefer their production environment to be non-virtual. The non-exhaustive list of reasons may include:
546+
547+
- Integration with third-party tools and platforms, such as data catalogs, may not work well with the virtual view layer that SQLMesh imposes by default
548+
- A desire to rely on time travel features provided by cloud data warehouses such as BigQuery, Snowflake, and Databricks
549+
550+
To mitigate this, SQLMesh offers an alternative 'dev-only' mode for using VDE. It can be enabled in the project configuration like so:
551+
552+
=== "YAML"
553+
554+
```yaml linenums="1"
555+
virtual_environment_mode: dev_only
556+
```
557+
558+
=== "Python"
559+
560+
```python linenums="1"
561+
from sqlmesh.core.config import Config
562+
563+
config = Config(
564+
virtual_environment_mode="dev_only",
565+
)
566+
```
567+
568+
'dev-only' mode means that VDE is applied only in development environments. While in production, model tables and views are updated directly and bypass the virtual layer. This also means that physical tables in production will be created using the original, **unversioned** model names. Users will still benefit from VDE and data reuse across development environments.
569+
570+
Please note the following tradeoffs when enabling this mode:
571+
572+
- All data inserted in development environments is used only for [preview](../concepts/plans.md#data-preview-for-forward-only-changes) and will **not** be reused in production
573+
- Reverting a model to a previous version will be applied going forward and may require an explicit data restatement
574+
575+
!!! warning
576+
Switching the mode for an existing project will result in a **complete rebuild** of all models in the project. Refer to the [Table Migration Guide](./table_migration.md) to migrate existing tables without rebuilding them from scratch.
577+
578+
541579
#### Environment view catalogs
542580

543581
By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.

‎docs/reference/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ Configuration options for how SQLMesh manages environment creation and promotion
4646
| `environment_suffix_target` | Whether SQLMesh views should append their environment name to the `schema`, `table` or `catalog` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`) | string | N |
4747
| `gateway_managed_virtual_layer` | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False) | boolean | N |
4848
| `environment_catalog_mapping` | A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment. | dict[string, string] | N |
49+
| `virtual_environment_mode` | Determines the Virtual Data Environment (VDE) mode. If set to `full`, VDE is used in both production and development environments. The `dev_only` option enables VDE only in development environments, while in production, no virtual layer is used and models are materialized directly using their original names (i.e., no versioned physical tables). (Default: `full`) | string | N |
4950

5051
### Models
5152

0 commit comments

Comments
 (0)