Skip to content

Commit f43b2c8

Browse files
committed
feat: Fix mypy errors and refactor MergeMethod
1 parent 50b57db commit f43b2c8

38 files changed

Lines changed: 1985 additions & 21 deletions

.devcontainer/Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
FROM mcr.microsoft.com/devcontainers/base:ubuntu

.devcontainer/devcontainer.json

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"name": "SQLMesh",
3+
"build": {
4+
"dockerfile": "Dockerfile",
5+
"context": ".."
6+
},
7+
"features": {
8+
"ghcr.io/devcontainers/features/python:1": {
9+
"version": "3.10"
10+
},
11+
"ghcr.io/devcontainers/features/node:1": {
12+
"version": "20"
13+
}
14+
},
15+
"customizations": {
16+
"vscode": {
17+
"extensions": [
18+
"ms-python.python",
19+
"esbenp.prettier-vscode"
20+
]
21+
}
22+
},
23+
"forwardPorts": [
24+
8000,
25+
8001
26+
]
27+
}

.devcontainer/post-create.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
3+
# Install node dependencies
4+
sudo -l "$(whoami)" -c "pnpm install"
5+
6+
# Install python dependencies
7+
sudo -l "$(whoami)" -c "pip install -e '.[dev]'"

.gitlab-ci.yml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
2+
stages:
3+
- test
4+
5+
test-vscode:
6+
stage: test
7+
image: node:22
8+
script:
9+
- pnpm install
10+
- pnpm run ci
11+
12+
test-gitlab-integration:
13+
stage: test
14+
image: python:3.12
15+
before_script:
16+
- pip install poetry
17+
- poetry install --no-root
18+
script:
19+
- poetry run pytest tests/integrations/gitlab/cicd/

.pre-commit-config.yaml

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,86 @@ repos:
1717
files: *files
1818
- id: mypy
1919
name: mypy
20-
language: system
20+
language: python
2121
entry: mypy
22+
additional_dependencies: [
23+
"mypy~=1.13.0",
24+
"pydantic>=2.0.0",
25+
"astor",
26+
"click",
27+
"croniter",
28+
"duckdb>=0.10.0,!=0.10.3",
29+
"dateparser<=1.2.1",
30+
"hyperscript>=0.1.0",
31+
"ipywidgets",
32+
"jinja2",
33+
"packaging",
34+
"pandas",
35+
"python-dotenv",
36+
"requests",
37+
"rich[jupyter]",
38+
"ruamel.yaml",
39+
"sqlglot[rs]~=27.8.0",
40+
"tenacity",
41+
"time-machine",
42+
"json-stream",
43+
"agate==1.7.1",
44+
"beautifulsoup4",
45+
"clickhouse-connect",
46+
"cryptography",
47+
"databricks-sql-connector",
48+
"dbt-bigquery",
49+
"dbt-core",
50+
"dbt-duckdb>=1.7.1",
51+
"dbt-snowflake",
52+
"dbt-athena-community",
53+
"dbt-clickhouse",
54+
"dbt-databricks",
55+
"dbt-redshift",
56+
"dbt-trino",
57+
"Faker",
58+
"google-auth",
59+
"google-cloud-bigquery",
60+
"google-cloud-bigquery-storage",
61+
"httpx",
62+
"pandas-stubs",
63+
"pre-commit",
64+
"psycopg2-binary",
65+
"PyAthena[Pandas]",
66+
"PyGithub>=2.6.0",
67+
"python-gitlab>=4.0.0",
68+
"pyodbc>=5.0.0",
69+
"pyperf",
70+
"pyspark~=3.5.0",
71+
"pytest",
72+
"pytest-asyncio",
73+
"pytest-mock",
74+
"pytest-retry",
75+
"pytest-xdist",
76+
"pytz",
77+
"redshift_connector",
78+
"ruff~=0.11.0",
79+
"snowflake-connector-python[pandas,secure-local-storage]>=3.0.2",
80+
"sqlalchemy-stubs",
81+
"trino",
82+
"types-croniter",
83+
"types-dateparser",
84+
"types-PyMySQL",
85+
"types-python-dateutil",
86+
"types-pytz",
87+
"types-requests==2.28.8",
88+
"typing-extensions",
89+
"pygls>=1.2.0,<2.0.0",
90+
"lsprotocol",
91+
"fastapi==0.115.5",
92+
"watchfiles>=0.19.0",
93+
"uvicorn[standard]==0.22.0",
94+
"sse-starlette>=0.2.2",
95+
"pyarrow",
96+
"starlette>=0.27.0",
97+
"httptools",
98+
"websockets"
99+
]
22100
"types": [python]
23101
files: *files
24102
require_serial: true

Makefile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,9 @@ dbt-test:
9696
github-test:
9797
pytest -n auto -m "github"
9898

99+
gitlab-test:
100+
pytest -n auto -m "gitlab"
101+
99102
jupyter-test:
100103
pytest -n auto -m "jupyter"
101104

docs/installation.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ Some extras add features, like the SQLMesh VSCode extension or Github CI/CD bot:
3737
| ------------------- | ------------------------------- |
3838
| VSCode extension | `pip install "sqlmesh[lsp]"` |
3939
| Github CI/CD bot | `pip install "sqlmesh[github]"` |
40+
| Gitlab CI/CD bot | `pip install "sqlmesh[gitlab]"` |
4041
| dbt projects | `pip install "sqlmesh[dbt]"` |
4142
| dlt projects | `pip install "sqlmesh[dlt]"` |
4243
| Slack notifications | `pip install "sqlmesh[slack]"` |

docs/integrations/github.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,8 @@ Regardless of signal approach being used, the bot needs to be configured to use
9797
=== "Python"
9898
9999
```python linenums="1"
100-
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig, MergeMethod
100+
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig
101+
from sqlmesh.core.cicd.config import MergeMethod
101102
from sqlmesh.core.config import Config
102103

103104
config = Config(
@@ -188,7 +189,8 @@ This command must be enabled in the bot's configuration.
188189
=== "Python"
189190

190191
```python linenums="1"
191-
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig, MergeMethod
192+
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig
193+
from sqlmesh.core.cicd.config import MergeMethod
192194
from sqlmesh.core.config import Config
193195
194196
config = Config(
@@ -327,7 +329,8 @@ Example with all properties defined:
327329
=== "Python"
328330

329331
```python linenums="1"
330-
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig, MergeMethod
332+
from sqlmesh.integrations.github.cicd.config import GithubCICDBotConfig
333+
from sqlmesh.core.cicd.config import MergeMethod
331334
from sqlmesh.core.config import AutoCategorizationMode, CategorizerConfig, Config
332335
333336
config = Config(

docs/integrations/gitlab.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# GitLab Integration
2+
3+
SQLMesh can be integrated with GitLab to provide a seamless CI/CD experience for your data projects. This integration allows you to run SQLMesh commands directly from your GitLab merge requests, and to get feedback on the status of your data models.
4+
5+
## Setup
6+
7+
To set up the GitLab integration, you need to do the following:
8+
9+
1. **Create a GitLab private token.** You can create a private token in your GitLab user settings. The token needs to have the `api` scope.
10+
2. **Add the private token to your GitLab CI/CD variables.** Add the token as a masked variable named `GITLAB_PRIVATE_TOKEN` in your project's CI/CD settings.
11+
3. **Configure the SQLMesh bot.** You need to add a `cicd_bot` section to your `config.yml` file. The following is an example configuration:
12+
13+
```yaml
14+
cicd_bot:
15+
type: gitlab
16+
project_id: <your-project-id>
17+
enable_deploy_command: true
18+
```
19+
20+
## Usage
21+
22+
Once the GitLab integration is set up, you can use the following commands in your merge request comments:
23+
24+
- `/deploy`: Deploys the changes in the merge request to the production environment.
25+
26+
## CI/CD Pipeline
27+
28+
The following is an example of a GitLab CI/CD pipeline that uses the SQLMesh GitLab integration:
29+
30+
```yaml
31+
stages:
32+
- test
33+
- deploy
34+
35+
sqlmesh_checks:
36+
stage: test
37+
script:
38+
- sqlmesh gitlab run-all-checks --token $GITLAB_PRIVATE_TOKEN
39+
40+
sqlmesh_deploy:
41+
stage: deploy
42+
script:
43+
- sqlmesh gitlab run-deploy-command --token $GITLAB_PRIVATE_TOKEN
44+
rules:
45+
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE == "note"'
46+
```

docs/integrations/overview.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ SQLMesh supports integrations with the following tools:
66
* [dbt](dbt.md)
77
* [dlt](dlt.md)
88
* [GitHub Actions](github.md)
9+
* [GitLab](gitlab.md)
910
* [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli)
1011

1112
## Execution engines

0 commit comments

Comments
 (0)