Skip to content

Commit badd74b

Browse files
committed
feat: add helm chart and docker compose for hybrid executors
1 parent e0f922f commit badd74b

7 files changed

Lines changed: 593 additions & 57 deletions

File tree

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Tobiko Cloud Hybrid Executors - Docker Compose Setup
2+
3+
This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. Hybrid executors enable your own infrastructure to connect to Tobiko Cloud for operations while keeping your data warehouse credentials within your environment.
4+
5+
## What This Setup Provides
6+
7+
The hybrid executors connect your data warehouse to Tobiko Cloud in a secure way:
8+
9+
- **Apply Executor**: Executes operations that change state, like creating and updating models
10+
- **Run Executor**: Runs read-only operations like planning and analysis
11+
12+
## Prerequisites
13+
14+
- Access to a data warehouse (e.g., Postgres, Snowflake, BigQuery)
15+
- Docker and Docker Compose
16+
- A Tobiko Cloud account with client ID and client secret
17+
18+
![add_oath_client](./scheduler/add_oath_client.png)
19+
20+
## Quick Start Guide
21+
22+
1. **Get docker-compose file**:
23+
24+
Download the [docker-compose file](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) to a local directory.
25+
26+
2. **Create your environment file**:
27+
28+
```bash
29+
cp .env.example .env
30+
```
31+
32+
3. **Edit the .env file** with your specific configuration:
33+
34+
- Set your Tobiko Cloud organization, project, client ID, and client secret
35+
- Configure your database connection details
36+
- Adjust resource limits if needed
37+
38+
4. **Start the executors**:
39+
40+
```bash
41+
docker compose up -d
42+
```
43+
44+
5. **Check the logs**:
45+
46+
```bash
47+
docker compose logs -f
48+
```
49+
50+
## Configuration Options
51+
52+
### Database Configuration
53+
54+
The default configuration is set up for Postgre, but you can use any supported database by adjusting the connection parameters in your `.env` file:
55+
56+
### Multiple Gateways
57+
58+
To configure multiple gateways, edit the docker compose.yml file to add additional environment variables for each gateway:
59+
60+
```yaml
61+
environment:
62+
# First gateway
63+
SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
64+
# ... other GATEWAY_A configuration ...
65+
66+
# Second gateway
67+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake
68+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT}
69+
# ... other GATEWAY_B configuration ...
70+
```
71+
72+
## Health Checking
73+
74+
You can verify executor health by running:
75+
76+
```bash
77+
docker compose exec apply-executor /app/pex executor apply --check
78+
docker compose exec run-executor /app/pex executor run --check
79+
```
80+
81+
Check for exit code 0 using `echo $?` to confirm the executors are healthy.
82+
Example successful output:
83+
84+
```bash
85+
> docker compose exec apply-executor /app/pex executor apply --check
86+
2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025)
87+
2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180
88+
```
89+
90+
## Stopping the Executors
91+
92+
To stop the executors:
93+
94+
```bash
95+
docker compose down
96+
```
97+
98+
## Troubleshooting
99+
100+
If you encounter issues:
101+
102+
1. Check the logs: `docker compose logs -f`
103+
2. Verify your connection settings in the `.env` file
104+
3. Ensure your client ID and client secret are correct
105+
4. Check that your database is accessible from the Docker containers
106+
107+
## Security Considerations
108+
109+
- The `.env` file contains sensitive information. Never commit it to version control.
110+
- Consider using Docker secrets or a secrets management solution in production environments.
111+
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.

0 commit comments

Comments
 (0)