|
| 1 | +# Tobiko Cloud Hybrid Executors - Docker Compose Setup |
| 2 | + |
| 3 | +This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. Hybrid executors enable your own infrastructure to connect to Tobiko Cloud for operations while keeping your data warehouse credentials within your environment. |
| 4 | + |
| 5 | +## What This Setup Provides |
| 6 | + |
| 7 | +The hybrid executors connect your data warehouse to Tobiko Cloud in a secure way: |
| 8 | + |
| 9 | +- **Apply Executor**: Executes operations that change state, like creating and updating models |
| 10 | +- **Run Executor**: Runs read-only operations like planning and analysis |
| 11 | + |
| 12 | +## Prerequisites |
| 13 | + |
| 14 | +- Access to a data warehouse (e.g., Postgres, Snowflake, BigQuery) |
| 15 | +- Docker and Docker Compose |
| 16 | +- A Tobiko Cloud account with client ID and client secret |
| 17 | + |
| 18 | + |
| 19 | + |
| 20 | +## Quick Start Guide |
| 21 | + |
| 22 | +1. **Get docker-compose file**: |
| 23 | + |
| 24 | + Download the [docker-compose file](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) to a local directory. |
| 25 | + |
| 26 | +2. **Create your environment file**: |
| 27 | + |
| 28 | + ```bash |
| 29 | + cp .env.example .env |
| 30 | + ``` |
| 31 | + |
| 32 | +3. **Edit the .env file** with your specific configuration: |
| 33 | + |
| 34 | + - Set your Tobiko Cloud organization, project, client ID, and client secret |
| 35 | + - Configure your database connection details |
| 36 | + - Adjust resource limits if needed |
| 37 | + |
| 38 | +4. **Start the executors**: |
| 39 | + |
| 40 | + ```bash |
| 41 | + docker compose up -d |
| 42 | + ``` |
| 43 | + |
| 44 | +5. **Check the logs**: |
| 45 | + |
| 46 | + ```bash |
| 47 | + docker compose logs -f |
| 48 | + ``` |
| 49 | + |
| 50 | +## Configuration Options |
| 51 | + |
| 52 | +### Database Configuration |
| 53 | + |
| 54 | +The default configuration is set up for Postgre, but you can use any supported database by adjusting the connection parameters in your `.env` file: |
| 55 | + |
| 56 | +### Multiple Gateways |
| 57 | + |
| 58 | +To configure multiple gateways, edit the docker compose.yml file to add additional environment variables for each gateway: |
| 59 | + |
| 60 | +```yaml |
| 61 | +environment: |
| 62 | + # First gateway |
| 63 | + SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres} |
| 64 | + # ... other GATEWAY_A configuration ... |
| 65 | + |
| 66 | + # Second gateway |
| 67 | + SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake |
| 68 | + SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT} |
| 69 | + # ... other GATEWAY_B configuration ... |
| 70 | +``` |
| 71 | + |
| 72 | +## Health Checking |
| 73 | + |
| 74 | +You can verify executor health by running: |
| 75 | + |
| 76 | +```bash |
| 77 | +docker compose exec apply-executor /app/pex executor apply --check |
| 78 | +docker compose exec run-executor /app/pex executor run --check |
| 79 | +``` |
| 80 | + |
| 81 | +Check for exit code 0 using `echo $?` to confirm the executors are healthy. |
| 82 | +Example successful output: |
| 83 | + |
| 84 | +```bash |
| 85 | +> docker compose exec apply-executor /app/pex executor apply --check |
| 86 | +2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025) |
| 87 | +2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180 |
| 88 | +``` |
| 89 | +
|
| 90 | +## Stopping the Executors |
| 91 | +
|
| 92 | +To stop the executors: |
| 93 | +
|
| 94 | +```bash |
| 95 | +docker compose down |
| 96 | +``` |
| 97 | +
|
| 98 | +## Troubleshooting |
| 99 | +
|
| 100 | +If you encounter issues: |
| 101 | +
|
| 102 | +1. Check the logs: `docker compose logs -f` |
| 103 | +2. Verify your connection settings in the `.env` file |
| 104 | +3. Ensure your client ID and client secret are correct |
| 105 | +4. Check that your database is accessible from the Docker containers |
| 106 | +
|
| 107 | +## Security Considerations |
| 108 | +
|
| 109 | +- The `.env` file contains sensitive information. Never commit it to version control. |
| 110 | +- Consider using Docker secrets or a secrets management solution in production environments. |
| 111 | +- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options. |
0 commit comments