Adapt documentation for mTLS and KMS

docs/architecture/concepts/external-key-management.md

@@ -0,0 +1,43 @@
+---
+title: "External Key Management"
+description: "Data-at-rest encryption with external key management systems, enabling separation of duty, rotation, and audit."
+weight: 30220
+---
+
+Volume encryption protects data at rest by ciphering every block written to a logical volume. To encrypt data, the
+cipher itself needs a key. However, the question of *where that key lives and who controls it* is the responsibility of
+the key management layer.
+
+By default, simplyblock manages encryption keys internally. For environments with stricter security policies, such as
+regulated environments or any deployment that separates storage and security duties, where the team operating the
+storage cluster must not be in possession of the long-lived key material, the key-encryption keys can be offloaded to an
+external Key Management Service (KMS).
+
+Simplyblock supports storing keys in external KMS solutions. Currently supported KMS backends are:
+- [HashiCorp Vault](https://www.vaultproject.io/){:target="_blank" rel="noopener"}
+- [OpenBao](https://openbao.org/){:target="_blank" rel="noopener"}
+
+## Two-Layer Key Model
+
+When an external KMS is configured, simplyblock applies a two-layer key model:
+
+- **Unseal Keys** are generated once and presented at the time of the KMS setup (for example, HashCorp Vault).
+  Typically, a certain number of all the unseal keys are required to unseal the KMS (e.g., 3 of 5 keys). These keys
+  should be stored in separate secure locations.
+- **Data Encryption Keys (DEKs)** are generated per volume and used to encrypt the at-rest data blocks of that volume.
+  These keys are short-lived in cluster memory and never stored in plaintext at rest. The wrapped DEKs are stored inside
+  the external KMS.
+- **Key Encryption Keys (KEKs)** live inside the KMS. The cluster asks the KMS to wrap each DEK on creation and to
+  unwrap it when the volume is brought online. The KEKs never leave the KMS.
+
+## Authentication and Trust
+
+The KMS authenticates simplyblock components using a client certificate issued by the
+`simplyblock-certificate-authority-issuer` ClusterIssuer, which the operator creates as part of its mTLS setup.
+Because the KMS depends on this CA, [mTLS](../../deployments/kubernetes/security.md#transport-layer-security-mutual-tls-mtls)
+must be configured on the control plane before an external KMS can be wired up.
+
+Operationally, this means the KMS team and the storage team share only the CA bundle and an agreed-upon DNS-name for
+the simplyblock client. No static passwords or long-lived tokens must be exchanged.
+
+For the setup steps, see [Securing the Control Plane: External KMS](../../deployments/kubernetes/security.md#external-key-management-kms).

docs/deployments/kubernetes/k8s-control-plane.md

@@ -32,17 +32,14 @@ helm upgrade --install simplyblock -n simplyblock simplyblock/spdk-csi \
 ```
 
 !!! important "TLS Encryption"
-    {{experimental}}
+    {{ experimental }}
 
-    Simplyblock has just added support for TLS encryption for all internal communication. At the moment, it's an
-    experimental feature only available when installed into OpenShift clusters.
+    All internal control plane traffic can be encrypted with TLS. On OpenShift, the cluster's built-in certificate
+    manager is used out of the box. Mutual TLS (mTLS), where components additionally authenticate each other with
+    client certificates, is only works with Cert-Manager. That means that on OpenShift, the Cert-Manager must be
+    installed to enable mTLS.
 
-    It generally is a good idea to install the operator with TLS support enabled on OpenShift to ensure that all
-    internal communication is encrypted and secure.
-
-    To enable TLS, add the `--set tls.enabled=true` flag to the `helm install` command.
-
-    In the future, the support will be extended to further Kubernetes distributions by enabling Cert-Manager support.
+    See [Securing the Control Plane](security.md#transport-layer-security-mutual-tls-mtls) for configuration.
 
 After installation, verify the operator is running:
 

docs/deployments/kubernetes/k8s-storage-plane.md

@@ -100,6 +100,12 @@ been registered, it has no storage nodes yet. Those are added in the next step.
     as NVMe-oF transport security, backup configuration, capacity thresholds, and more, are available at
     [Cluster Deployment Options](../cluster-deployment-options.md).
 
+!!! tip "External KMS"
+    If volumes in this cluster should offload their encryption keys to an external KMS, set
+    `spec.hashicorpVaultSettings.base_url` on the `StorageCluster` now. The setting can also be added later, but
+    configuring it upfront means encrypted volumes use the external KMS from day one. See
+    [Securing the Control Plane: External KMS](security.md#external-key-management-kms).
+
 ## Add Storage Nodes
 
 Now, Kubernetes worker nodes will be transformed into simplyblock storage nodes. To initiate the process, a

docs/deployments/kubernetes/security.md

@@ -0,0 +1,225 @@
+---
+title: "Securing the Control Plane"
+description: "Configure mTLS for simplyblock control plane communication and offload at-rest encryption keys to an external KMS (HashiCorp Vault or OpenBao)."
+weight: 30050
+---
+
+This page covers two security features for simplyblock on Kubernetes: transport-layer encryption and mutual
+authentication for the control plane (mTLS), and offloading volume encryption keys to an external Key Management
+Service (KMS).
+
+mTLS must be configured before an external KMS can be wired up: the KMS authenticates simplyblock components using a
+certificate issued by the operator-managed certificate authority, which is only provisioned when mTLS is active.
+
+## Transport Layer Security (Mutual TLS / mTLS)
+
+{{ experimental }}
+
+Internal control-plane traffic between the control plane, the operator, and the storage-node handlers can be encrypted
+with TLS. Additionally, when mutual TLS is enabled, every component must present a valid client certificate, which
+means components authenticate each other rather than relying on network position alone.
+
+!!! important "Mutual TLS on OpenShift"
+    **mTLS is only supported using the Cert-Manager certificate provider.**
+
+    On OpenShift, the cluster's built-in certificate manager provides one-way TLS (server certificates) but does not
+    issue the client certificates required for mutual authentication. To enable mTLS on OpenShift,
+    [Cert-Manager](https://cert-manager.io/){:target="_blank" rel="noopener"} must be installed and the certificate
+    provider must be switched over.
+
+### Prerequisites
+
+- Cert-Manager must be installed in the cluster.
+- A `ClusterIssuer` (or namespaced `Issuer`) for Cert-Manager to be able to mint certificates must exist. Most
+  installations point this at an internal corporate certificate authority (CA) or at the cluster-local self-signed
+  issuer. Any issuer that simplyblock components trust via the CA is acceptable.
+
+### Enabling mTLS
+
+Mutual TLS (mTLS) is configured at Helm install time by setting four values on the operator chart. Either with setting
+the `tls` field directly in the values.yaml or via the `--set` flags on the Helm command line.
+
+```yaml title="Helm values for mTLS"
+tls:
+  enabled: true
+  mutual_enabled: true
+  provider: cert-manager
+  cert-manager:
+    issuer: my-cluster-issuer
+```
+
+Apply the values during the operator installation (see [Install Simplyblock Operator](k8s-control-plane.md)):
+
+```bash title="Install the operator with mTLS"
+helm upgrade --install simplyblock -n simplyblock simplyblock/spdk-csi \
+    --create-namespace \
+    --set operator.enabled=true \
+    --set tls.enabled=true \
+    --set tls.mutual_enabled=true \
+    --set tls.provider=cert-manager \
+    --set tls.cert-manager.issuer=my-cluster-issuer
+```
+
+Replace `my-cluster-issuer` with the name of the `ClusterIssuer` the operator should use to obtain its certificates.
+
+### What the Operator Provisions
+
+When mTLS is enabled, the operator creates a dedicated `ClusterIssuer` named
+`simplyblock-certificate-authority-issuer` and issues all internal component certificates signed with the configured
+certificate authority. The same issuer can be used to mint certificates for other workloads that need to talk to
+simplyblock. These workloads specifically include external key management systems (KMS), as described in the next
+section.
+
+!!! note "OpenShift"
+    On OpenShift, setting `tls.enabled=true` with the default `tls.provider=openshift` only activates one-way TLS using
+    OpenShift-managed certificates.
+
+    Mutual TLS is **not** available with the OpenShift default provider. To use `tls.mutual_enabled=true`
+    requires `tls.provider=cert-manager` regardless of the underlying Kubernetes distribution.
+
+## External Key Management (KMS)
+
+{{ experimental }}
+
+By default, simplyblock manages volume encryption keys internally. For environments that require stricter key handling,
+the cluster can be configured to keep the key-encryption material in an external KMS. This especially includes
+environments with strict separation of duty between storage administrators and key custodians, regular rotation, or
+audit trails.
+
+As of now, [HashiCorp Vault](https://www.vaultproject.io/){:target="_blank" rel="noopener"} and
+[OpenBao](https://openbao.org/){:target="_blank" rel="noopener"} are supported. The configuration is identical for
+either of them.
+
+### Prerequisites
+
+- [mTLS configured](#transport-layer-security-mutual-tls-mtls) is required, because the vault is authenticated to the cluster via
+  a certificate issued by the operator's `simplyblock-certificate-authority-issuer`.
+- A Vault or OpenBao instance reachable from the simplyblock namespace. The instance must be initialized and unsealed
+  before configuring authentication.
+
+### Issue a TLS Certificate via Vault
+
+Create a Cert-Manager `Certificate` resource that uses the operator-managed issuer. The resulting secret holds the
+TLS material that Vault serves to clients and is trusted by the simplyblock components because it chains to the same
+CA.
+
+```yaml title="vault-tls.yaml"
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: vault-tls
+  namespace: vault
+spec:
+  secretName: vault-tls
+  issuerRef:
+    name: simplyblock-certificate-authority-issuer
+    kind: ClusterIssuer
+  commonName: vault
+  dnsNames:
+    - vault
+    - vault.vault
+    - vault.vault.svc
+    - vault.vault.svc.cluster.local
+```
+
+Mount the resulting `vault-tls` secret into the Vault deployment as its serving certificate. Mount the issuer's CA
+bundle (typically `ca.crt`) at a path the Vault can read. The example below assumes `/vault/tls/ca.crt` for Vault
+and `/bao/tls/ca.crt` for OpenBao.
+
+### Deploy the Vault
+
+Install Vault or OpenBao using their upstream Helm chart and expose it inside the cluster. For the rest of this guide
+the in-cluster service is assumed to be `vault.vault:8200`. Adjust the URL to match the actual deployment.
+
+### Configure Auth, Policy, and Secret Engines
+
+Configure the vault with a policy that grants simplyblock access to the `transit` and `kv` backends, enable the
+certificate authentication bound to the simplyblock CA, and enable the required secret engines. The script below
+works for both Vault (`vault`) and OpenBao (`bao`). Assign the appropriate CLI to the `$CLI` variable.
+
+```bash title="Configure the vault for simplyblock"
+CLI=vault   # or: CLI=bao
+
+# Policy granting access to the transit and kv backends
+$CLI policy write webappapi-policy - <<EOF
+path "transit/keys/*" {
+  capabilities = ["create", "update", "read", "delete"]
+}
+
+path "transit/datakey/plaintext/*" {
+  capabilities = ["create", "update"]
+}
+
+path "transit/datakey/wrapped/*" {
+  capabilities = ["create", "update"]
+}
+
+path "transit/decrypt/*" {
+  capabilities = ["create", "update"]
+}
+
+path "kv/*" {
+  capabilities = ["create", "read", "update", "delete"]
+}
+EOF
+
+# Certificate authentication, bound to the simplyblock cluster CA
+$CLI auth enable cert
+$CLI write auth/cert/certs/webappapi \
+    certificate=@/${CLI}/tls/ca.crt \
+    allowed_dns_sans="simplyblock-webappapi" \
+    token_policies=webappapi-policy \
+    token_ttl=10m \
+    token_max_ttl=30m
+
+# Secret engines used by simplyblock
+$CLI secrets enable transit
+$CLI secrets enable -version=1 kv
+```
+
+- The **policy** grants simplyblock the minimum capabilities it needs: managing keys, performing envelope
+  encryption on the `transit` backend, and storing per-volume key material on the `kv` backend.
+- The **cert auth** role only accepts clients that present a certificate chaining to the simplyblock CA *and* whose
+  DNS SAN is `simplyblock-webappapi`. Tokens are short-lived (10 min, 30 min maximum), so a compromised token expires
+  quickly.
+- **Transit** is used for wrapping data-encryption keys. The **kv** version 1 is used as the per-volume metadata store.
+
+### Point the StorageCluster to Vault
+
+Set `spec.hashicorpVaultSettings.base_url` on the `StorageCluster` resource:
+
+```yaml title="StorageCluster with external KMS"
+apiVersion: storage.simplyblock.io/v1alpha1
+kind: StorageCluster
+metadata:
+  name: simplyblock-cluster
+  namespace: simplyblock
+spec:
+  clusterName: production
+  mgmtIfname: eth0
+  fabricType: tcp
+  haType: ha
+  hashicorpVaultSettings:
+    base_url: "https://vault.vault:8200/"
+```
+
+This setting is automatically picked up by the operator during the next reconcilation cycle. From that point on, volume
+encryption keys for this cluster are wrapped against the vault's transit backend instead of being held inside the
+cluster.
+
+!!! warning "Important Note"
+    Only encryption keys for volumes that are created after the vault is wired up are wrapped and stored in the vault.
+    Existing volumes are not affected.
+
+### Verification
+
+Once configured, check the operator and webappapi pod logs for vault connection messages and watch the cluster
+status:
+
+```bash title="Verify the KMS connection"
+kubectl get storagecluster -n simplyblock
+kubectl logs -n simplyblock deploy/simplyblock-operator
+```
+
+Creating a new encrypted volume after the vault is wired up exercises the path end-to-end. The volume's encryption key
+material is then stored in the vault rather than alongside the cluster.

docs/deployments/kubernetes/volume-encryption.md

@@ -1,7 +1,72 @@
 ---
 title: "Volume Encryption"
-description: "Configure encrypted Simplyblock volumes on Kubernetes using CSI, including key generation, Kubernetes Secret setup, and encrypted StorageClass/PVC examples."
+description: "Encrypt simplyblock logical volumes at rest using the AES_XTS crypto bdev. Keys are managed by the cluster and can optionally be offloaded to an external KMS."
 weight: 40000
 ---
 
-{% include 'kubernetes-csi-encryption.md' %}
+Simplyblock supports encryption of logical volumes at rest, ensuring that sensitive data remains protected across the
+distributed storage cluster. Internally, simplyblock uses the industry-proven
+[crypto bdev](https://spdk.io/doc/bdev.html){:target="_blank" rel="noopener"} provided by SPDK, with an AES_XTS
+variable-length block cipher.
+
+Encryption is enabled per StorageClass and applies to every volume provisioned from it.
+
+!!! warning
+    Encryption must be specified at the time of volume creation. Existing logical volumes cannot be retroactively
+    encrypted.
+
+## Enabling Encryption on a StorageClass
+
+To enable encryption, set the `encryption` parameter on the StorageClass to `"True"`. Every PersistentVolumeClaim
+that references the StorageClass is then provisioned as an encrypted volume.
+
+```yaml title="Encrypted StorageClass"
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: my-encrypted-volumes
+provisioner: csi.simplyblock.io
+parameters:
+  encryption: "True"
+  # ... other parameters
+reclaimPolicy: Delete
+volumeBindingMode: Immediate
+allowVolumeExpansion: true
+```
+
+A PersistentVolumeClaim using this StorageClass is then encrypted automatically:
+
+```yaml title="Encrypted PersistentVolumeClaim"
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: my-encrypted-volume-claim
+spec:
+  storageClassName: my-encrypted-volumes
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 200Gi
+```
+
+## Key Management
+
+Encryption keys are generated and managed by the simplyblock cluster. No user-supplied keys, per-PVC Secrets, or
+annotations are required to encrypt a volume.
+
+!!! warning "Migration from earlier versions"
+    Previous releases required a user-managed Kubernetes Secret (containing `crypto_key1` and `crypto_key2`) to be
+    referenced from each PVC via the `simplyblock.io/secret-name` (or legacy `simplybk/secret-name`) annotation.
+    That mechanism is **no longer used** for new volumes. Existing encrypted volumes provisioned with user-supplied
+    keys continue to work, but new PVCs should not set those annotations.
+
+## Hardening Key Storage with an External KMS
+
+For environments that require stricter handling of key material — separation of duty between storage and key
+custodians, regular rotation, or audit trails — the cluster can be configured to keep encryption keys in an external
+Hashicorp Vault or Openbao instance. The setup is configured once per `StorageCluster` and applies to every encrypted
+volume in that cluster.
+
+See [Securing the Control Plane: External KMS](security.md#external-key-management-kms) for the full setup, or
+[External Key Management](../../architecture/concepts/external-key-management.md) for the architectural background.

docs/maintenance-operations/security/encryption-kubernetes-secrets.md

@@ -1,6 +1,6 @@
 ---
 title: "Encrypting with Kubernetes Secrets"
-description: "Encrypting with Kubernetes Secrets: {% include 'kubernetes-csi-encryption.md' %}."
+description: "Encrypting with Kubernetes Secrets in simplyblock"
 weight: 30200
 ---