You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-**Scheduler**: Orchestrates schedule cadence and hosts state metadata(code versions, logs, cost). Never **pushes** instructions to executor.
32
-
-**Executor**: Appplies code changes and runs SQL queries and python models in proper DAG order (actual data processing in SQL Engine)
33
-
-**Gateway**: Stores authentication to SQL Engine. Secured through your secrets manager or Kubernetes Secrets.
31
+
-**Scheduler**: Orchestrates schedule cadence and hosts state metadata(code versions, logs, cost). **Never pushes** instructions to executor.
32
+
-**Executor**: Appplies code changes and runs SQL queries and Python models in proper DAG order (actual data processing in SQL Engine)
33
+
-**Gateway**: Stores authentication credentials for SQL Engine. Secured through your secrets manager or Kubernetes Secrets.
34
34
-**SQL Engine**: Processes and stores data based on the above instructions
35
-
-**Executor -> Scheduler**: This is a pull-only mechanism as stated abvoe.
35
+
-**Executor -> Scheduler**: A pull-only mechanism for obtaining work tasks.
36
36
-**Helm Chart**: For production environements, we provide a [Helm chart](../scheduler/hybrid_executors_helm.md) that includes robust configurability, secret management, and scaling options.
37
-
-**Docker Compose**: For simpler environments or testing, we offer a [Docker Compose setup](../scheduler/hybrid_executors_docker_compose) to quickly deploy executors on any machine with Docker.
37
+
-**Docker Compose**: For simpler environments or testing, we offer a [Docker Compose setup](../scheduler/hybrid_executors_docker_compose.md) to quickly deploy executors on any machine with Docker.
38
38
39
39
40
40
41
41
## Internal Code Practices
42
42
43
-
Our coding standards are guidelines we enforce throughout the organization to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust.
43
+
We enforce coding standards throughout Tobiko to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust.
44
44
45
-
Below you will find a few examples of our internal code requirements.
45
+
A few key components of our internal code requirements:
46
46
47
-
- We used signed commits, required approvers, and signed Docker artifacts.
48
-
- Each commit to main must be approved by someone other than the author.
49
-
- We follow the standard of signing commits and registering the key with GitHub. [Github Docs](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)
50
-
-Binary is signed using cosign and OIDC for keyless. [Signing docs](https://docs.sigstore.dev/cosign/signing/overview/)
51
-
- Attestations are created to certify an image. We use GCP Binary Authorization to enforce this. [Attestation docs](https://cloud.google.com/binary-authorization/docs/key-concepts#attestations)
47
+
- We used signed Git commits, required approvers, and signed Docker artifacts.
48
+
- Each commit to a `main` branch must be approved by someone other than the author.
49
+
- We sign commits and register the key with GitHub ([Github Docs](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits))
50
+
-Binaries are signed using cosign and OIDC for keyless ([Signing docs](https://docs.sigstore.dev/cosign/signing/overview/)
51
+
- Attestations are created to certify an image, enforced with GCP Binary Authorization ([Attestation docs](https://cloud.google.com/binary-authorization/docs/key-concepts#attestations))
52
52
- Encryption is a key feature of our security posture and is enforced at each stage of access. For example, the state database automatically encrypts all data. Credentials are also securely encrypted and stored.
53
53
- We back up each state database nightly and before upgrades. These backups are stored for 14 days.
54
54
@@ -61,11 +61,13 @@ At least once a year, Tobiko engages a third-party security firm to perform a pe
61
61
62
62
### How do we protect PGP keys?
63
63
64
-
If an employee loses their laptop, we don't need to get the old PGP key back. We use GitHub to sign code. At the time the code was committed, the PGP key was valid. When an employee loses their laptop, we will invalidate it, and they will regenerate a new key. The old commits are valid because the PGP key was valid at the time.
64
+
If an employee loses their laptop, we don't need to get the old PGP key back because we can invalidate the key directly.
65
+
66
+
We use GitHub to sign code commits. At the time the code was committed, the PGP key was valid. When an employee loses their laptop, we will invalidate it, and they will regenerate a new key to use in future commits. The old commits are still valid because the PGP key was valid at the time the commit was made.
65
67
66
68
### How do we invalidate PGP keys if someone did steal it and could potentially use it?
67
69
68
-
Revoke access for the GitHub user account associated with the compromised key and not give them access again until the old PGP key is deprecated and issue a new PGP key.
70
+
Revoke access for the GitHub user account associated with the compromised key and not give them access again until the old PGP key is deprecated and a new key issued.
69
71
70
72
### If someone steals a laptop, what's our continuity plan in protecting code?
0 commit comments