feat: add helm chart and docker compose for hybrid executors

Author: eakmanrq

docs/cloud/features/scheduler/scheduler.md

@@ -165,20 +165,40 @@ This gives you complete control over data security and network access while stil
 
 Self-hosted executors as the name indicates are self-hosted workers that take on the responsibility of "executing" changes to the data warehouse. While Tobiko Cloud schedules and plans changes, the executors are responsible for executing those changes. Executors are docker containers that are configured to connect to Tobiko Cloud as well as your data warehouse. Connected to both, the executor pulls work from the cloud, whether it's a plan or scheduled background work from a run, and execute it on your data warehouse.
 
+### Deployment Options
+
+You can deploy the executor containers using any method that works for your infrastructure and operational requirements. The executors are standard Docker containers that can be deployed in any container environment as long as they're properly configured with the required environment variables.
+
+For your convenience, we provide two reference implementations:
+
+1. **Kubernetes with Helm Chart**: For production environments, we provide a [Helm chart](./scheduler/hybrid-executors/README.md) that includes robust configurability, secret management, and scaling options.
+
+2. **Docker Compose**: For simpler environments or testing, we offer a [Docker Compose setup](./scheduler/hybrid-executors-docker-compose.md) to quickly deploy executors on any machine with Docker.
+
+You're free to adapt these reference implementations or create your own deployment method that fits your specific needs. The only requirement is that two executor instances must be running (one for run operations and one for apply operations) and properly configured.
+
 ### Configuration
 
-Exact configuration is left to the user and will vary based on the infrastructure and setup of the user, for example the executors could be run on a Kubernetes cluster or as a standalone pair of Docker containers depending on the user's infrastructure. The following details are an example of how this is done for a Postgres data warehouse and a pair of local containers running in docker.
+The following covers basic configuration concepts for hybrid executors. For detailed configuration options, please refer to the specific documentation for your chosen deployment method above.
+
+Tobiko Cloud requires 2 executor instances to be running:
 
-Tobiko Cloud requires 2 docker instances to be running, one to pick up runs and one for plan applications. The entrypoint for both is `executor run` and `executor apply` respectively. The executor container can be found on [Docker Hub](https://hub.docker.com/r/tobikodata/tcloud). In addition to running the container, the user will need to configure the executor with environment variables that point to Tobiko Cloud as well as the data warehouse.
+1. **Run Executor**: Handles scheduled model execution
+2. **Apply Executor**: Handles applying changes to the data warehouse
 
-To connect to the Tobiko Cloud, the user will need to provide the following environment variables, replaced with the user's own values.
+Both executors need to be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse.
+
+#### Basic Environment Variables
+
+To connect to Tobiko Cloud, you'll need to provide:
 
 ```env
 TCLOUD_URL=https://cloud.tobikodata.com/sqlmesh/acme/analytics_project
-TCLOUD_TOKEN=your_token
+TCLOUD_CLIENT_ID=your_client_id
+TCLOUD_CLIENT_SECRET=your_client_secret
 ```
 
-In addition to the above variables, a gateway is needed to provide a connection to a data warehouse. The following details are an example of how this is done for a Postgres data warehouse where the gateway is named `GATEWAY_A`. For more details on how to configure a gateway, see the details on other [engines](../../../integrations/overview.md#execution-engines) and how to [overide variables](../../../guides/configuration.md#overrides) as done below. 
+To connect to your data warehouse, configure a gateway:
 
 ```env
 SQLMESH__DEFAULT_GATEWAY=GATEWAY_A
@@ -190,7 +210,7 @@ SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__USER=example_user
 SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__PASSWORD=example_password
 ```
 
-**Note**: If there are multiple gateways, each gateway will need to have its own set of environment variables. For example, if there are two gateways, `GATEWAY_A` and `GATEWAY_B`, the environment variables will need to be set for both gateways.
+**Note**: For multiple gateways, each gateway needs its own set of environment variables:
 
 ```env
 SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE=<connection type>  
@@ -199,14 +219,9 @@ SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE=<connection type>
 # <Gateway B connection settings>  
 ```
 
-Once you have set up both sets of environment variables in a file named `local.env`, you can run the following command to start the executor:
-
-```shell
-docker run -d --env-file local.env tobikodata/tcloud:latest -- executor run
-docker run -d --env-file local.env tobikodata/tcloud:latest -- executor apply
-```
+For more details, including secure secret management options, refer to the [Helm chart](./scheduler/hybrid-executors/README.md) or [Docker Compose](./scheduler/hybrid-executors-docker-compose.md) documentation.
 
-After the executors are properly configured, they will appear in the cloud UI where they can be used to apply plans and execute scheduled runs.
+After the executors are properly configured and running, they will appear in the cloud UI where they can be used to apply plans and execute scheduled runs.
 
 ![executors](../scheduler/executors.png)
 
@@ -224,43 +239,18 @@ While the exact requirements for executors vary depending on the customer's spec
 
 Environment variables may also be passed to SQLMesh by mounting a `.env` file into the docker image and specifying it's full path with the environment variable `TCLOUD_ENV_FILE`. 
 
-**Note** These variables are only read at SQLMesh run time so variables that are used by `tcloud` such as `TCLOUD_URL`, `TCLOUD_TOKEN`, etc. must be passed as variables. 
+**Note** These variables are only read at SQLMesh run time so variables that are used by `tcloud` such as `TCLOUD_URL`, `TCLOUD_CLIENT_ID`, `TCLOUD_CLIENT_SECRET`, etc. must be passed as variables. 
 
 ### Health checks
 
-In production settings, we recommend setting up health checks as well, as these can be helpful in understanding the health of executors. When calling via the entrypoint, the health checks are as follows:
+In production settings, we recommend setting up health checks to monitor the status of your executors. Health checks help ensure your executors are operating correctly and can identify issues before they impact your workflows.
 
-```shell
-docker run -d --env-file local.env tobikodata/tcloud:latest -- executor run --check
-docker run -d --env-file local.env tobikodata/tcloud:latest -- executor apply --check
-```
+For detailed information on implementing health checks:
 
-For environments that ignore the container's entrypoint, for example in the case of Kubernetes healthchecks, the health check is invoked as follows:
-
-```yaml
-# For the run executor
-readinessProbe:
-  exec:
-    command:
-    - "/app/pex"
-    - "executor"
-    - "run"
-    - "--check"
-  timeoutSeconds: 5
-```
+- **Kubernetes/Helm**: See the [Hybrid Executors Helm Chart documentation](./scheduler/hybrid-executors/README.md#verifying-the-installation) for information on health check configuration in Kubernetes.
 
-```yaml
-# For the apply executor
-readinessProbe:
-  exec:
-    command:
-    - "/app/pex"
-    - "executor"
-    - "apply"
-    - "--check"
-  timeoutSeconds: 5
-```
+- **Docker Compose**: Refer to the [Docker Compose setup documentation](./scheduler/hybrid-executors-docker-compose.md#health-checking) for health check implementation with Docker Compose.
 
-Each executor type (run or apply) should have its own health check implemented to ensure proper monitoring of both components.
+Both executor types (run and apply) should have appropriate health checks implemented to ensure proper monitoring and reliability.
 
-**Note** Please note the specified timeout. Depending on the allocated resources, sometimes the default timeout is too aggressive.
+**Note** When configuring health checks, ensure timeouts are set appropriately based on your resources. Default timeouts can sometimes be too aggressive depending on the allocated resources.

docs/cloud/features/scheduler/scheduler/.env.example

@@ -0,0 +1,24 @@
+# Tobiko Cloud Configuration
+ORGANIZATION=your-organization
+PROJECT=your-project
+TCLOUD_CLIENT_ID=your-client-id
+TCLOUD_CLIENT_SECRET=your-client-secret
+
+# Database Configuration
+DEFAULT_GATEWAY=GATEWAY_A
+DB_TYPE=postgres
+DB_HOST=your-database-host
+DB_PORT=5432
+DB_NAME=your-database-name
+DB_USER=your-database-user
+DB_PASSWORD=your-database-password
+
+# Optional: Resource Limits
+APPLY_MEMORY_LIMIT=4g
+APPLY_CPU_LIMIT=2
+APPLY_MEMORY_REQUEST=2g
+APPLY_CPU_REQUEST=1
+PLAN_MEMORY_LIMIT=4g
+PLAN_CPU_LIMIT=2
+PLAN_MEMORY_REQUEST=2g
+PLAN_CPU_REQUEST=1 

docs/cloud/features/scheduler/scheduler/docker-compose.yml

@@ -0,0 +1,72 @@
+version: '3.8'
+
+services:
+  apply-executor:
+    image: tobikodata/tcloud:latest
+    command: executor apply
+    restart: unless-stopped
+    environment:
+      # Tobiko Cloud connection
+      TCLOUD_URL: https://cloud.tobikodata.com/sqlmesh/${ORGANIZATION}/${PROJECT}
+      TCLOUD_CLIENT_ID: ${TCLOUD_CLIENT_ID}
+      TCLOUD_CLIENT_SECRET: ${TCLOUD_CLIENT_SECRET}
+      
+      # SQLMesh configuration
+      SQLMESH__DEFAULT_GATEWAY: ${DEFAULT_GATEWAY:-GATEWAY_A}
+      
+      # Example database configuration for Postgres (adjust for your database)
+      # All database parameters below should be customized for your specific setup
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__HOST: ${DB_HOST}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__PORT: ${DB_PORT:-5432}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__DATABASE: ${DB_NAME}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__USER: ${DB_USER}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__PASSWORD: ${DB_PASSWORD}
+    volumes:
+      # Optional volume for persistent storage if needed
+      - apply-executor-data:/app/data
+    deploy:
+      resources:
+        limits:
+          memory: ${APPLY_MEMORY_LIMIT:-4g}
+          cpus: ${APPLY_CPU_LIMIT:-2}
+        reservations:
+          memory: ${APPLY_MEMORY_REQUEST:-2g}
+          cpus: ${APPLY_CPU_REQUEST:-1}
+
+  run-executor:
+    image: tobikodata/tcloud:latest
+    command: executor run
+    restart: unless-stopped
+    environment:
+      # Tobiko Cloud connection
+      TCLOUD_URL: https://cloud.tobikodata.com/sqlmesh/${ORGANIZATION}/${PROJECT}
+      TCLOUD_CLIENT_ID: ${TCLOUD_CLIENT_ID}
+      TCLOUD_CLIENT_SECRET: ${TCLOUD_CLIENT_SECRET}
+      
+      # SQLMesh configuration
+      SQLMESH__DEFAULT_GATEWAY: ${DEFAULT_GATEWAY:-GATEWAY_A}
+      
+      # Example database configuration for Postgres (adjust for your database)
+      # All database parameters below should be customized for your specific setup
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__HOST: ${DB_HOST}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__PORT: ${DB_PORT:-5432}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__DATABASE: ${DB_NAME}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__USER: ${DB_USER}
+      SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__PASSWORD: ${DB_PASSWORD}
+    volumes:
+      # Optional volume for persistent storage if needed
+      - run-executor-data:/app/data
+    deploy:
+      resources:
+        limits:
+          memory: ${PLAN_MEMORY_LIMIT:-4g}
+          cpus: ${PLAN_CPU_LIMIT:-2}
+        reservations:
+          memory: ${PLAN_MEMORY_REQUEST:-2g}
+          cpus: ${PLAN_CPU_REQUEST:-1}
+
+volumes:
+  apply-executor-data: {}
+  run-executor-data: {}