Skip to content

Commit 7cca1ff

Browse files
authored
Docs: Trey edits to hybrid deployment docs (#4122)
1 parent badd74b commit 7cca1ff

6 files changed

Lines changed: 296 additions & 223 deletions

File tree

626 KB
Loading
Lines changed: 35 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,42 @@
11
# Tobiko Cloud Hybrid Executors - Docker Compose Setup
22

3-
This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. Hybrid executors enable your own infrastructure to connect to Tobiko Cloud for operations while keeping your data warehouse credentials within your environment.
3+
This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose.
44

5-
## What This Setup Provides
5+
Hybrid executors allow you to run operations on your own infrastructure while leveraging Tobiko Cloud for orchestration.
66

7-
The hybrid executors connect your data warehouse to Tobiko Cloud in a secure way:
7+
## What this setup provides
88

9-
- **Apply Executor**: Executes operations that change state, like creating and updating models
10-
- **Run Executor**: Runs read-only operations like planning and analysis
9+
This setup deploys two hybrid executors that pass work tasks from Tobiko Cloud to your data warehouse in a secure way:
10+
11+
- **Apply Executor**: Handles applying changes to the data warehouse
12+
- **Run Executor**: Handles scheduled model execution
13+
14+
Both executors must be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse.
1115

1216
## Prerequisites
1317

14-
- Access to a data warehouse (e.g., Postgres, Snowflake, BigQuery)
18+
- Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery)
1519
- Docker and Docker Compose
16-
- A Tobiko Cloud account with client ID and client secret
20+
- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials)
1721

18-
![add_oath_client](./scheduler/add_oath_client.png)
19-
20-
## Quick Start Guide
22+
## Quick start guide
2123

2224
1. **Get docker-compose file**:
2325

24-
Download the [docker-compose file](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) to a local directory.
26+
Download the [docker-compose.yml](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) files to a local directory.
2527

2628
2. **Create your environment file**:
2729

30+
Copy the downloaded example environment file into a new `.env` file:
31+
2832
```bash
2933
cp .env.example .env
3034
```
3135

32-
3. **Edit the .env file** with your specific configuration:
36+
3. **Edit the .env file** with your project's configuration:
3337

3438
- Set your Tobiko Cloud organization, project, client ID, and client secret
35-
- Configure your database connection details
39+
- Configure your gateway connection details
3640
- Adjust resource limits if needed
3741

3842
4. **Start the executors**:
@@ -47,47 +51,48 @@ The hybrid executors connect your data warehouse to Tobiko Cloud in a secure way
4751
docker compose logs -f
4852
```
4953

50-
## Configuration Options
54+
## Configuration options
5155

52-
### Database Configuration
56+
### Gateway configuration
5357

54-
The default configuration is set up for Postgre, but you can use any supported database by adjusting the connection parameters in your `.env` file:
58+
The default configuration in the `docker-compose.yml` file uses Postgres, but you can use [any supported SQL engine](../../../integrations/overview.md#execution-engines) by adjusting the connection parameters in your `.env` file.
5559

56-
### Multiple Gateways
60+
#### Multiple gateways
5761

58-
To configure multiple gateways, edit the docker compose.yml file to add additional environment variables for each gateway:
62+
To configure multiple gateways, add additional environment variables for each gateway the `docker-compose.yml` file:
5963

6064
```yaml
6165
environment:
6266
# First gateway
6367
SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
6468
# ... other GATEWAY_A configuration ...
65-
69+
6670
# Second gateway
6771
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake
6872
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT}
6973
# ... other GATEWAY_B configuration ...
7074
```
7175

72-
## Health Checking
76+
## Health checking
7377

74-
You can verify executor health by running:
78+
Verify the health of your executors by running these commands:
7579

7680
```bash
7781
docker compose exec apply-executor /app/pex executor apply --check
7882
docker compose exec run-executor /app/pex executor run --check
7983
```
8084

8185
Check for exit code 0 using `echo $?` to confirm the executors are healthy.
86+
8287
Example successful output:
8388

8489
```bash
8590
> docker compose exec apply-executor /app/pex executor apply --check
8691
2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025)
87-
2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180
92+
2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180)
8893
```
8994

90-
## Stopping the Executors
95+
## Stopping the executors
9196

9297
To stop the executors:
9398

@@ -102,10 +107,13 @@ If you encounter issues:
102107
1. Check the logs: `docker compose logs -f`
103108
2. Verify your connection settings in the `.env` file
104109
3. Ensure your client ID and client secret are correct
105-
4. Check that your database is accessible from the Docker containers
110+
4. Check that your SQL engine is accessible from the Docker containers
111+
112+
## Security considerations
113+
114+
!!! warning "Never commit .env to version control"
106115

107-
## Security Considerations
116+
The `.env` file contains sensitive information. Never commit it to version control.
108117

109-
- The `.env` file contains sensitive information. Never commit it to version control.
110118
- Consider using Docker secrets or a secrets management solution in production environments.
111-
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.
119+
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.

0 commit comments

Comments
 (0)