diff --git a/docs/cloud/features/scheduler/airflow.md b/docs/cloud/features/scheduler/airflow.md index f997da39da..11e82769dd 100644 --- a/docs/cloud/features/scheduler/airflow.md +++ b/docs/cloud/features/scheduler/airflow.md @@ -52,7 +52,7 @@ $ pip install tobiko-cloud-scheduler-facade[airflow] ### Connect Airflow to Tobiko Cloud -First, provision an OAuth Client for Airflow to use by following the guide on how to [provision client credentials](../single_sign_on.md#provisioning-client-credentials). +First, provision an OAuth Client for Airflow to use by following the guide on how to [provision client credentials](../security/single_sign_on.md#provisioning-client-credentials). After provisioning the credentials, you can obtain the `Client ID` and `Client Secret` values for Airflow to use to connect to Tobiko Cloud. diff --git a/docs/cloud/features/scheduler/dagster.md b/docs/cloud/features/scheduler/dagster.md index 367863aea6..338bed2572 100644 --- a/docs/cloud/features/scheduler/dagster.md +++ b/docs/cloud/features/scheduler/dagster.md @@ -57,7 +57,7 @@ Dagster recommends [injecting secret values using Environment Variables](https:/ On this page, we demonstrate the secrets method Dagster recommends for **local development**. -First, provision an OAuth Client for Dagster to use by following the guide on how to [provision client credentials](../single_sign_on.md#provisioning-client-credentials). +First, provision an OAuth Client for Dagster to use by following the guide on how to [provision client credentials](../security/single_sign_on.md#provisioning-client-credentials). After provisioning the credentials, you can obtain the `Client ID` and `Client Secret` values for Dagster to use to connect to Tobiko Cloud. @@ -374,8 +374,8 @@ customer_revenue_by_day = AssetKey(["postgres", "sushi", "customer_revenue_by_da | Option | Description | Type | Required | |----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----:|:--------:| | `url` | The Base URL to your Tobiko Cloud instance | str | Y | -| `oauth_client_id` | OAuth Client ID of the credentials you [provisioned](../single_sign_on.md#provisioning-client-credentials) for Dagster | str | N | -| `oauth_client_secret` | OAuth Client Secret of the credentials you [provisioned](../single_sign_on.md#provisioning-client-credentials) for Dagster | str | N | +| `oauth_client_id` | OAuth Client ID of the credentials you [provisioned](../security/single_sign_on.md#provisioning-client-credentials) for Dagster | str | N | +| `oauth_client_secret` | OAuth Client Secret of the credentials you [provisioned](../security/single_sign_on.md#provisioning-client-credentials) for Dagster | str | N | | `dagster_graphql_host` | Hostname of the Dagster Webserver GraphQL endpoint | str | N | | `dagster_graphql_port` | Port of the Dagster Webserver GraphQL endpoint | int | N | | `dagster_graphql_kwargs` | Extra args to pass to the [DagsterGraphQLClient](https://docs.dagster.io/api/python-api/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) class when it is instantiated | dict | N | diff --git a/docs/cloud/features/scheduler/hybrid_executors_docker_compose.md b/docs/cloud/features/scheduler/hybrid_executors_docker_compose.md index f5bac11818..e3bd072752 100644 --- a/docs/cloud/features/scheduler/hybrid_executors_docker_compose.md +++ b/docs/cloud/features/scheduler/hybrid_executors_docker_compose.md @@ -19,7 +19,7 @@ Both executors must be properly configured with environment variables to connect - Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery) - Docker and Docker Compose -- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials) +- A Tobiko Cloud account with [client ID and client secret](../security/single_sign_on.md#provisioning-client-credentials) ## Quick start guide diff --git a/docs/cloud/features/scheduler/hybrid_executors_helm.md b/docs/cloud/features/scheduler/hybrid_executors_helm.md index 7c5d8ace4d..b945ad6bd6 100644 --- a/docs/cloud/features/scheduler/hybrid_executors_helm.md +++ b/docs/cloud/features/scheduler/hybrid_executors_helm.md @@ -17,7 +17,7 @@ Both executors must be properly configured with environment variables to connect - Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery) - Helm 3.8+ -- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials) +- A Tobiko Cloud account with [client ID and client secret](../security/single_sign_on.md#provisioning-client-credentials) ## Quick start guide @@ -267,7 +267,7 @@ run: ## Defining Custom Environment Variables -If there are additional environment variables that are required to run your project, you will want to define them for both the apply and run executors. +If there are additional environment variables that are required to run your project, you will want to define them for both the apply and run executors. ```yaml apply: diff --git a/docs/cloud/features/scheduler/hybrid_executors_overview.md b/docs/cloud/features/scheduler/hybrid_executors_overview.md index 6bc2a5524a..ae07cfc364 100644 --- a/docs/cloud/features/scheduler/hybrid_executors_overview.md +++ b/docs/cloud/features/scheduler/hybrid_executors_overview.md @@ -57,7 +57,7 @@ One important type of environment variable is the `TCLOUD` variables used for co The first required `TCLOUD` variable is a unique Tobiko Cloud URL for your project, which your Solutions Architect will provide after your project is created. -You also need the Client ID and Client Secret variables, which are generated when you [create an OAuth Client](../single_sign_on.md#provisioning-client-credentials) in the Tobiko Cloud UI. +You also need the Client ID and Client Secret variables, which are generated when you [create an OAuth Client](../security/single_sign_on.md#provisioning-client-credentials) in the Tobiko Cloud UI. Specify the URL, Client ID, and Client Secret in these environment variables: diff --git a/docs/cloud/features/security/security.md b/docs/cloud/features/security/security.md index bbcccd1a6f..59b2149432 100644 --- a/docs/cloud/features/security/security.md +++ b/docs/cloud/features/security/security.md @@ -1,15 +1,15 @@ # Security Overview -At Tobiko, we treat security as a first-class citizen because we know how valuable your data assets are. Our team follows and executes security best practices across each layer of our product. +At Tobiko, we treat security as a first-class citizen because we know how valuable your data assets are. Our team follows and executes security best practices across each layer of our product. ## Tobiko Cloud Standard Deployment -Our standard Tobiko Cloud deployment consists of several components that are each responsible for different parts of the product. +Our standard Tobiko Cloud deployment consists of several components that are each responsible for different parts of the product. -Below is a diagram of the components along with their descriptions. +Below is a diagram of the components along with their descriptions. -![tobiko_cloud_standard_deployment](./tcloud_standard_deployment.png){ width=80% height=60% style="display: block; margin: 0 auto" } +![tobiko_cloud_standard_deployment](./security/tcloud_standard_deployment.png){ width=80% height=60% style="display: block; margin: 0 auto" } - **Scheduler**: Orchestrates schedule cadence and hosts state metadata (code versions, logs, cost) - **Executor**: Applies code changes and runs SQL queries (actual data processing in SQL Engine) and Python models in proper DAG order. @@ -18,21 +18,21 @@ Below is a diagram of the components along with their descriptions. ## Tobiko Cloud Hybrid Deployment -For some customers, our hybrid deployment option is a great fit. It provides a seamless experience with Tobiko Cloud but within your own VPC and infrastructure. +For some customers, our hybrid deployment option is a great fit. It provides a seamless experience with Tobiko Cloud but within your own VPC and infrastructure. -In a hybrid deployment, Tobiko Cloud does not execute tasks directly with the engine. Instead, it passes tasks to the executors hosted in your environment, which then execute the tasks with the engine. +In a hybrid deployment, Tobiko Cloud does not execute tasks directly with the engine. Instead, it passes tasks to the executors hosted in your environment, which then execute the tasks with the engine. Executors are Docker containers that connect to both Tobiko Cloud and your SQL engine. They pull work tasks from the Tobiko Cloud scheduler and execute them with your SQL engine. This is a pull-only mechanism authenticated through an OAuth Client ID/Secret. Whitelist IPs in your network to allow reaching Tobiko Cloud IPs from the executor: 34.28.17.91, 34.136.27.153, 34.136.131.20 -Below is a diagram of the components along with their description. +Below is a diagram of the components along with their description. -![tobiko_cloud_hybrid_deployment](./tcloud_hybrid_deployment.png){ width=80% height=60% style="display: block; margin: 0 auto" } +![tobiko_cloud_hybrid_deployment](./security/tcloud_hybrid_deployment.png){ width=80% height=60% style="display: block; margin: 0 auto" } - **Scheduler**: Orchestrates schedule cadence and hosts state metadata (code versions, logs, cost). **Never pushes** instructions to executor. - **Executor**: Appplies code changes and runs SQL queries and Python models in proper DAG order (actual data processing in SQL Engine) - **Gateway**: Stores authentication credentials for SQL Engine. Secured through your secrets manager or Kubernetes Secrets. - **SQL Engine**: Processes and stores data based on the above instructions -- **Executor -> Scheduler**: A pull-only mechanism for obtaining work tasks. +- **Executor -> Scheduler**: A pull-only mechanism for obtaining work tasks. - **Helm Chart**: For production environements, we provide a [Helm chart](../scheduler/hybrid_executors_helm.md) that includes robust configurability, secret management, and scaling options. - **Docker Compose**: For simpler environments or testing, we offer a [Docker Compose setup](../scheduler/hybrid_executors_docker_compose.md) to quickly deploy executors on any machine with Docker. @@ -40,7 +40,7 @@ Below is a diagram of the components along with their description. ## Internal Code Practices -We enforce coding standards throughout Tobiko to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust. +We enforce coding standards throughout Tobiko to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust. A few key components of our internal code requirements: @@ -49,19 +49,19 @@ A few key components of our internal code requirements: - We sign commits and register the key with GitHub ([Github Docs](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)). - Binaries are signed using cosign and OIDC for keyless ([Signing docs](https://docs.sigstore.dev/cosign/signing/overview/)). - Attestations are created to certify an image, enforced with GCP Binary Authorization ([Attestation docs](https://cloud.google.com/binary-authorization/docs/key-concepts#attestations)). -- Encryption is a key feature of our security posture and is enforced at each stage of access. For example, the state database automatically encrypts all data. Credentials are also securely encrypted and stored. +- Encryption is a key feature of our security posture and is enforced at each stage of access. For example, the state database automatically encrypts all data. Credentials are also securely encrypted and stored. - We back up each state database nightly and before upgrades. These backups are stored for 14 days. ## Penetration Testing At least once a year, Tobiko engages a third-party security firm to perform a penetration test. This test evaluates our systems by identifying and attempting to exploit known vulnerabilities, focusing on critical external and/or internal assets. A detailed report is available upon request. - -## Asset and Access Management + +## Asset and Access Management ### How do we protect PGP keys? -If an employee loses their laptop, we don't need to get the old PGP key back because we can invalidate the key directly. +If an employee loses their laptop, we don't need to get the old PGP key back because we can invalidate the key directly. We use GitHub to sign code commits. At the time the code was committed, the PGP key was valid. When an employee loses their laptop, we will invalidate it, and they will regenerate a new key to use in future commits. The old commits are still valid because the PGP key was valid at the time the commit was made. @@ -77,4 +77,3 @@ We would revoke access for the GitHub user account associated with the compromis - We follow a formal IT asset disposal procedure to prevent key compromise through improper hardware disposal. - See above for PGP key protection. - Binaries are signed using Cosign and OIDC for keyless signing. - diff --git a/docs/cloud/features/security/tcloud_hybrid_deployment.png b/docs/cloud/features/security/security/tcloud_hybrid_deployment.png similarity index 100% rename from docs/cloud/features/security/tcloud_hybrid_deployment.png rename to docs/cloud/features/security/security/tcloud_hybrid_deployment.png diff --git a/docs/cloud/features/security/tcloud_standard_deployment.png b/docs/cloud/features/security/security/tcloud_standard_deployment.png similarity index 100% rename from docs/cloud/features/security/tcloud_standard_deployment.png rename to docs/cloud/features/security/security/tcloud_standard_deployment.png diff --git a/docs/cloud/features/single_sign_on.md b/docs/cloud/features/security/single_sign_on.md similarity index 93% rename from docs/cloud/features/single_sign_on.md rename to docs/cloud/features/security/single_sign_on.md index 726c1a8365..716fb10589 100644 --- a/docs/cloud/features/single_sign_on.md +++ b/docs/cloud/features/security/single_sign_on.md @@ -1,17 +1,17 @@ -# SSO (Single Sign-On) +# SSO (Single Sign-On) ## Overview -Tobiko Cloud supports single sign-on (SSO) through OpenID and SAML 2.0 providers. +Tobiko Cloud supports single sign-on (SSO) through OpenID and SAML 2.0 providers. -This makes it easy to provision access to users and simplifies authentication. +This makes it easy to provision access to users and simplifies authentication. -## Setup & Prerequsites +## Setup & Prerequsites -You must have an active Tobiko Cloud instance with SSO enabled. Please contact your account team to ensure this is enabled. +You must have an active Tobiko Cloud instance with SSO enabled. Please contact your account team to ensure this is enabled. -If your Tobiko Cloud instance is setup to require SSO, then you won't need to provide a token in your `tcloud.yml` configuration. +If your Tobiko Cloud instance is setup to require SSO, then you won't need to provide a token in your `tcloud.yml` configuration. Below is an example of a `tcloud.yml` configuration: ```yaml @@ -27,7 +27,7 @@ default_project: ## Identity Providers -Tobiko Cloud currently supports OpenID and SAML 2.0. +Tobiko Cloud currently supports OpenID and SAML 2.0. ### OpenID @@ -37,17 +37,17 @@ to login with most OAuth2 login providers. There are two ways to use OpenID Providers. The first is a if you use a shared provider like Google, Github, -Microsoft, etc. +Microsoft, etc. #### Google OAuth -To enable Google OAuth, all we need is your domain (ex: `yourname@companyname.com`, `companyname.com` is the domain). From here, we can switch SSO on with Google OAuth. +To enable Google OAuth, all we need is your domain (ex: `yourname@companyname.com`, `companyname.com` is the domain). From here, we can switch SSO on with Google OAuth. -The login flow will look like the following if you access [cloud.tobikodata.com/auth/login](https://cloud.tobikodata.com/auth/login) directly from your browser. If authenticating through CLI see [here](https://sqlmesh.readthedocs.io/en/stable/cloud/features/single_sign_on/#status) for more details. +The login flow will look like the following if you access [cloud.tobikodata.com/auth/login](https://cloud.tobikodata.com/auth/login) directly from your browser. If authenticating through CLI see [here](../security/single_sign_on.md#status) for more details.
-#### Other OAuth Providers +#### Other OAuth Providers If you use Okta and other custom OpenID/OAuth2 providers you need to add us as an Application or Client (terms differ across providers). @@ -69,9 +69,9 @@ We will need the following information from you once you set us up: |---------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| | Client ID | The random ID we use to communicate with their OAuth service | `` | | Client Secret | The random secret we use to authentication with their OAuth service | `` | -| Open ID Configuration URL | This is the URL we use to gather the rest of their OpenID Configuration. We can often find this on our own and don't need to request it from them, check with the onboarding engineer to make sure we know this. | +| Open ID Configuration URL | This is the URL we use to gather the rest of their OpenID Configuration. We can often find this on our own and don't need to request it from them, check with the onboarding engineer to make sure we know this. | -Once we have the above information, we can enable SSO on your account. You will then follow the login flow through your provider such as logging in through Okta. +Once we have the above information, we can enable SSO on your account. You will then follow the login flow through your provider such as logging in through Okta. ### SAML V2.0 @@ -107,34 +107,34 @@ a provider named `acme`: ### Okta Integration -The following instructions will walk you through configuring Okta as your identity provider. +The following instructions will walk you through configuring Okta as your identity provider. Log into your Okta account. Navigate to Application and create a new app. You will want to select SAML 2.0 ![okta_setup_1](./single_sign_on/okta_setup_1.png) -Next, name your app "Tobiko Cloud". You can add the app logo by downloading the image [here](https://avatars.githubusercontent.com/u/113925670?s=200&v=4). +Next, name your app "Tobiko Cloud". You can add the app logo by downloading the image [here](https://avatars.githubusercontent.com/u/113925670?s=200&v=4). ![okta_setup_2](./single_sign_on/okta_setup_2.png) #### SAML Configurations and Settings -1. We now need to fill in the SAML Settings. Please enter the following values: +1. We now need to fill in the SAML Settings. Please enter the following values: - - **Single sign-on URL**: `https://cloud.tobikodata.com/auth/saml/callback/acme` + - **Single sign-on URL**: `https://cloud.tobikodata.com/auth/saml/callback/acme` - **Audience URI (SP Entity ID)**: `https://cloud.tobikodata.com/auth/saml/metadata/acme` ![okta_setup_3](./single_sign_on/okta_setup_3.png) -2. Fill in the Attribute Statements section with email, firstName, and lastName: These are required to properly map to your users. +2. Fill in the Attribute Statements section with email, firstName, and lastName: These are required to properly map to your users. ![okta_setup_4](./single_sign_on/okta_setup_4.png) -3. Click next and now you are on the last step. Check off the box `Contact app vendor` and hit `Finish`. Now you're all set! +3. Click next and now you are on the last step. Check off the box `Contact app vendor` and hit `Finish`. Now you're all set! ![okta_setup_5](./single_sign_on/okta_setup_5.png) -Here is what you will see if you are accessing Tobiko Cloud via Okta. Click on the Tobiko Cloud icon to be redirected to the application. +Here is what you will see if you are accessing Tobiko Cloud via Okta. Click on the Tobiko Cloud icon to be redirected to the application. ![sso_okta](./single_sign_on/sso_okta.png) @@ -142,7 +142,7 @@ Here is what you will see if you are accessing Tobiko Cloud via Okta. Click on t ### Status -You can see what the status of your session is with the `status` command: +You can see what the status of your session is with the `status` command: ``` bash $ tcloud auth status @@ -192,11 +192,11 @@ Not currently authenticated ![tcloud_logout](./single_sign_on/tcloud_logout.png) -Otherwise, you will be logged out automatically when the SSO session expires (every 24 hours). +Otherwise, you will be logged out automatically when the SSO session expires (every 24 hours). ## OAuth Clients -Sometimes, you want to grant an external service access to your Tobiko Cloud project. For example, the external service could be the [CICD bot](../../integrations/github.md) or a [scheduler integration](./scheduler/airflow.md). +Sometimes, you want to grant an external service access to your Tobiko Cloud project. For example, the external service could be the [CICD bot](../../../integrations/github.md) or a [scheduler integration](../scheduler/airflow.md). These services take `Client ID` and `Client Secret` credentials. diff --git a/docs/cloud/features/single_sign_on/oauth_client_1.png b/docs/cloud/features/security/single_sign_on/oauth_client_1.png similarity index 100% rename from docs/cloud/features/single_sign_on/oauth_client_1.png rename to docs/cloud/features/security/single_sign_on/oauth_client_1.png diff --git a/docs/cloud/features/single_sign_on/oauth_client_2.png b/docs/cloud/features/security/single_sign_on/oauth_client_2.png similarity index 100% rename from docs/cloud/features/single_sign_on/oauth_client_2.png rename to docs/cloud/features/security/single_sign_on/oauth_client_2.png diff --git a/docs/cloud/features/single_sign_on/okta_setup_1.png b/docs/cloud/features/security/single_sign_on/okta_setup_1.png similarity index 100% rename from docs/cloud/features/single_sign_on/okta_setup_1.png rename to docs/cloud/features/security/single_sign_on/okta_setup_1.png diff --git a/docs/cloud/features/single_sign_on/okta_setup_2.png b/docs/cloud/features/security/single_sign_on/okta_setup_2.png similarity index 100% rename from docs/cloud/features/single_sign_on/okta_setup_2.png rename to docs/cloud/features/security/single_sign_on/okta_setup_2.png diff --git a/docs/cloud/features/single_sign_on/okta_setup_3.png b/docs/cloud/features/security/single_sign_on/okta_setup_3.png similarity index 100% rename from docs/cloud/features/single_sign_on/okta_setup_3.png rename to docs/cloud/features/security/single_sign_on/okta_setup_3.png diff --git a/docs/cloud/features/single_sign_on/okta_setup_4.png b/docs/cloud/features/security/single_sign_on/okta_setup_4.png similarity index 100% rename from docs/cloud/features/single_sign_on/okta_setup_4.png rename to docs/cloud/features/security/single_sign_on/okta_setup_4.png diff --git a/docs/cloud/features/single_sign_on/okta_setup_5.png b/docs/cloud/features/security/single_sign_on/okta_setup_5.png similarity index 100% rename from docs/cloud/features/single_sign_on/okta_setup_5.png rename to docs/cloud/features/security/single_sign_on/okta_setup_5.png diff --git a/docs/cloud/features/single_sign_on/sso_okta.png b/docs/cloud/features/security/single_sign_on/sso_okta.png similarity index 100% rename from docs/cloud/features/single_sign_on/sso_okta.png rename to docs/cloud/features/security/single_sign_on/sso_okta.png diff --git a/docs/cloud/features/single_sign_on/tcloud_auth.png b/docs/cloud/features/security/single_sign_on/tcloud_auth.png similarity index 100% rename from docs/cloud/features/single_sign_on/tcloud_auth.png rename to docs/cloud/features/security/single_sign_on/tcloud_auth.png diff --git a/docs/cloud/features/single_sign_on/tcloud_auth_browser_login.png b/docs/cloud/features/security/single_sign_on/tcloud_auth_browser_login.png similarity index 100% rename from docs/cloud/features/single_sign_on/tcloud_auth_browser_login.png rename to docs/cloud/features/security/single_sign_on/tcloud_auth_browser_login.png diff --git a/docs/cloud/features/single_sign_on/tcloud_auth_browser_success.png b/docs/cloud/features/security/single_sign_on/tcloud_auth_browser_success.png similarity index 100% rename from docs/cloud/features/single_sign_on/tcloud_auth_browser_success.png rename to docs/cloud/features/security/single_sign_on/tcloud_auth_browser_success.png diff --git a/docs/cloud/features/single_sign_on/tcloud_login.png b/docs/cloud/features/security/single_sign_on/tcloud_login.png similarity index 100% rename from docs/cloud/features/single_sign_on/tcloud_login.png rename to docs/cloud/features/security/single_sign_on/tcloud_login.png diff --git a/docs/cloud/features/single_sign_on/tcloud_logout.png b/docs/cloud/features/security/single_sign_on/tcloud_logout.png similarity index 100% rename from docs/cloud/features/single_sign_on/tcloud_logout.png rename to docs/cloud/features/security/single_sign_on/tcloud_logout.png diff --git a/mkdocs.yml b/mkdocs.yml index 6f20b9d844..9bcfeeafa3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -114,7 +114,6 @@ nav: - "Alerts & Notifications": cloud/features/alerts_notifications.md - cloud/features/debugger_view.md - cloud/features/data_catalog.md - - cloud/features/single_sign_on.md - Scheduler: - "Cloud": cloud/features/scheduler/scheduler.md - "Cloud Hybrid Deployments": @@ -124,8 +123,9 @@ nav: - cloud/features/scheduler/airflow.md - cloud/features/scheduler/dagster.md - Security: - - "Security Overview": cloud/features/security/security.md - - "Incident Reporting": cloud/features/incident_reporting.md + - cloud/features/security/security.md + - cloud/features/security/single_sign_on.md + - cloud/features/incident_reporting.md - cloud/features/xdb_diffing.md # - Observability: # - cloud/features/observability/overview.md