Skip to content

Commit fc161b5

Browse files
committed
feat: add helm chart and docker compose for hybrid executors
1 parent badba08 commit fc161b5

7 files changed

Lines changed: 541 additions & 46 deletions

File tree

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Tobiko Cloud Hybrid Executors - Docker Compose Setup
2+
3+
This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. Hybrid executors enable your own infrastructure to connect to Tobiko Cloud for operations while keeping your data warehouse credentials within your environment.
4+
5+
## What This Setup Provides
6+
7+
The hybrid executors connect your data warehouse to Tobiko Cloud in a secure way:
8+
9+
- **Apply Executor**: Executes operations that change state, like creating and updating models
10+
- **Run Executor**: Runs read-only operations like planning and analysis
11+
12+
## Prerequisites
13+
14+
- Docker and Docker Compose
15+
- A Tobiko Cloud account with client ID and client secret
16+
- Access to a data warehouse (e.g., Postgres, Snowflake, BigQuery)
17+
18+
## Quick Start Guide
19+
20+
1. **Get docker-compose file**:
21+
22+
Download the [docker-compose file](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) to a local directory.
23+
24+
2. **Create your environment file**:
25+
26+
```bash
27+
cp .env.example .env
28+
```
29+
30+
3. **Edit the .env file** with your specific configuration:
31+
32+
- Set your Tobiko Cloud organization, project, client ID, and client secret
33+
- Configure your database connection details
34+
- Adjust resource limits if needed
35+
36+
4. **Start the executors**:
37+
38+
```bash
39+
docker compose up -d
40+
```
41+
42+
5. **Check the logs**:
43+
44+
```bash
45+
docker compose logs -f
46+
```
47+
48+
## Configuration Options
49+
50+
### Database Configuration
51+
52+
The default configuration is set up for Postgre, but you can use any supported database by adjusting the connection parameters in your `.env` file:
53+
54+
### Multiple Gateways
55+
56+
To configure multiple gateways, edit the docker compose.yml file to add additional environment variables for each gateway:
57+
58+
```yaml
59+
environment:
60+
# First gateway
61+
SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
62+
# ... other GATEWAY_A configuration ...
63+
64+
# Second gateway
65+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake
66+
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT}
67+
# ... other GATEWAY_B configuration ...
68+
```
69+
70+
## Health Checking
71+
72+
You can verify executor health by running:
73+
74+
```bash
75+
docker compose exec apply-executor executor apply --check
76+
docker compose exec run-executor executor run --check
77+
```
78+
79+
## Stopping the Executors
80+
81+
To stop the executors:
82+
83+
```bash
84+
docker compose down
85+
```
86+
87+
## Troubleshooting
88+
89+
If you encounter issues:
90+
91+
1. Check the logs: `docker compose logs -f`
92+
2. Verify your connection settings in the `.env` file
93+
3. Ensure your client ID and client secret are correct
94+
4. Check that your database is accessible from the Docker containers
95+
96+
## Security Considerations
97+
98+
- The `.env` file contains sensitive information. Never commit it to version control.
99+
- Consider using Docker secrets or a secrets management solution in production environments.
100+
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.

0 commit comments

Comments
 (0)