Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions environment-setup/configuration.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Configuration"
description: "Customize your tracebloc workspace — environment variables, cluster management, GPU support, and manual Helm deployment."

Check warning on line 3 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L3

Did you really mean 'tracebloc'?
---

The installer uses sensible defaults. This page covers everything you can change — from cluster naming and port mapping to GPU configuration, manual Helm deployment, and day-to-day cluster management.
Expand All @@ -15,6 +15,7 @@
| `SERVERS` | `1` | Number of control-plane nodes |
| `AGENTS` | `1` | Number of worker nodes |
| `K8S_VERSION` | `v1.29.4-k3s1` | k3s image tag |
| `TB_NAMESPACE` | `tracebloc` | Kubernetes namespace + Helm release name for the client |

Check warning on line 18 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L18

Did you really mean 'namespace'?
| `HTTP_PORT` | `80` | Host port mapped to cluster HTTP ingress |
| `HTTPS_PORT` | `443` | Host port mapped to cluster HTTPS ingress |
| `HOST_DATA_DIR` | `~/.tracebloc` | Persistent data directory on host |
Expand Down Expand Up @@ -42,10 +43,10 @@

### View logs

The jobs manager is the main tracebloc process. Check its logs when debugging connectivity or job execution issues:

Check warning on line 46 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L46

Did you really mean 'tracebloc'?

```bash
kubectl logs -n <workspace> -l app=tracebloc-jobs-manager
kubectl logs -n tracebloc -l app=tracebloc-jobs-manager
```

### Useful commands
Expand All @@ -55,9 +56,9 @@
```bash
kubectl get nodes -o wide # Node status and IPs
kubectl get pods -A # All pods across namespaces
kubectl get pods -n <workspace> # Pods in your workspace
kubectl get pvc -n <workspace> # Persistent volume claims
kubectl get services -n <workspace> # Services and endpoints
kubectl get pods -n tracebloc # Pods in your workspace
kubectl get pvc -n tracebloc # Persistent volume claims
kubectl get services -n tracebloc # Services and endpoints
```

Install logs are saved to `~/.tracebloc/install-*.log`.
Expand All @@ -70,7 +71,7 @@

Fully automatic. The installer:

1. Detects NVIDIA GPUs via `nvidia-smi` or `lspci`

Check warning on line 74 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L74

Did you really mean 'GPUs'?
2. Installs drivers if missing (Ubuntu, RHEL/CentOS, Arch)
3. Installs the NVIDIA Container Toolkit and configures Docker
4. Deploys the NVIDIA k8s device plugin into the cluster
Expand All @@ -80,11 +81,11 @@

### AMD (Linux)

Auto-detected. ROCm is installed automatically on Ubuntu and RHEL/CentOS. A logout/login may be needed for full GPU access.

Check warning on line 84 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L84

Did you really mean 'ROCm'?

### macOS

CPU only. Docker Desktop on macOS does not support GPU passthrough. For GPU workloads, deploy on a Linux machine with NVIDIA GPUs or use [AWS (EKS)](/environment-setup/eks-client-deployment-guide).

Check warning on line 88 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L88

Did you really mean 'passthrough'?

Check warning on line 88 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L88

Did you really mean 'GPUs'?

### Windows

Expand Down Expand Up @@ -242,7 +243,7 @@

#### Docker Registry

The chart pulls the client image from a container registry — credentials are required in production. Use a token, not a plaintext password.

Check warning on line 246 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L246

Did you really mean 'plaintext'?

```yaml
dockerRegistry:
Expand Down Expand Up @@ -279,11 +280,11 @@
timeout: "10m"
```

The CronJob's ServiceAccount is bound to the built-in `cluster-admin` ClusterRole because the chart templates cluster-scoped resources (PriorityClass, StorageClass, ClusterRoleBinding, optionally Namespace). Disable if you need a manual approval gate on upgrades.

Check warning on line 283 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L283

Did you really mean 'Namespace'?

#### NetworkPolicy hardening for training pods

Training pods run untrusted ML code. The chart can apply a NetworkPolicy that denies ingress and restricts egress to DNS + external HTTPS only — blocking pod-to-pod, MySQL, and Kubernetes API access from the training pod.

Check warning on line 287 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L287

Did you really mean 'untrusted'?

```yaml
networkPolicy:
Expand All @@ -309,12 +310,12 @@
Leave `enabled: false` on clusters without an enforcing CNI — silently having no protection is worse than explicitly disabling it.

<Warning>
The chart's training-pod egress lockdown only blocks traffic if your CNI enforces NetworkPolicy. Verify your CNI before relying on it.

Check warning on line 313 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L313

Did you really mean 'lockdown'?
</Warning>

#### Resource Monitor and node-agents namespace

Check warning on line 316 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L316

Did you really mean 'namespace'?

The `tracebloc-resource-monitor` DaemonSet collects node-level CPU/memory metrics. It mounts `hostPath` volumes (`/proc`, `/sys`) which Pod Security Admission's `restricted` profile bans — so the chart isolates it in a dedicated **privileged** namespace (default `tracebloc-node-agents`).

Check warning on line 318 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L318

Did you really mean 'namespace'?

```yaml
resourceMonitor: true # set false on clusters where metrics-server cannot be installed
Expand All @@ -324,7 +325,7 @@
name: tracebloc-node-agents
```

When `create: false`, create the namespace yourself with the required PSA labels:

Check warning on line 328 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L328

Did you really mean 'namespace'?

```bash
kubectl create namespace tracebloc-node-agents
Expand All @@ -338,7 +339,7 @@

#### Pod Security Admission labels

Training Jobs run untrusted user-supplied ML code. The chart can create the release namespace with Pod Security Admission `warn`/`audit`/`enforce` labels at the `restricted` profile for defense-in-depth:

Check warning on line 342 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L342

Did you really mean 'untrusted'?

Check warning on line 342 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L342

Did you really mean 'namespace'?

```yaml
namespace:
Expand All @@ -349,10 +350,10 @@
enforce: restricted # set "" for bare-metal hostPath installs
```

When `create: false` (default) and you want PSA labels on an existing namespace:

Check warning on line 353 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L353

Did you really mean 'namespace'?

```bash
kubectl label namespace <workspace> \
kubectl label namespace <namespace> \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/enforce=restricted
Expand All @@ -374,7 +375,7 @@

#### PriorityClass and PodDisruptionBudgets

The chart pins the MySQL pod with a `tracebloc-data-plane` PriorityClass (value `1000000`) so it survives node-level OOM and scheduling pressure, and applies PDBs to MySQL and the jobs manager. Override only if you run a multi-replica MySQL externally:

Check warning on line 378 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L378

Did you really mean 'PDBs'?

```yaml
priorityClass:
Expand All @@ -389,11 +390,11 @@

### Deploy

Install the chart into a new namespace:

Check warning on line 393 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L393

Did you really mean 'namespace'?

```bash
helm upgrade --install <workspace> tracebloc/client \
--namespace <workspace> \
helm upgrade --install <namespace> tracebloc/client \
--namespace <namespace> \
--create-namespace \
--values values.yaml
```
Expand All @@ -404,8 +405,8 @@

```bash
helm repo update
helm upgrade <workspace> tracebloc/client \
--namespace <workspace> \
helm upgrade <namespace> tracebloc/client \
--namespace <namespace> \
--reset-then-reuse-values \
--values values.yaml
```
Expand All @@ -417,14 +418,14 @@
### Uninstall

```bash
helm uninstall <workspace> -n <workspace>
helm uninstall <namespace> -n <namespace>
```

PVCs and the PriorityClass are annotated `helm.sh/resource-policy: keep` so your data and shared cluster resources survive uninstall. To remove them too:

Check warning on line 424 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L424

Did you really mean 'PVCs'?

```bash
kubectl delete pvc --all -n <workspace>
kubectl delete namespace <workspace>
kubectl delete pvc --all -n <namespace>
kubectl delete namespace <namespace>
```

### Migrating from legacy charts
Expand All @@ -440,10 +441,10 @@

## Security

Tracebloc is designed so your data never has to leave your network. Here's how:

Check warning on line 444 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L444

Did you really mean 'Tracebloc'?

- **Data stays local.** Training data never leaves your infrastructure. Only metadata and metrics are shared with the platform.
- **Encrypted.** All communication between client and platform is TLS-encrypted.
- **Isolated.** Training runs in containers with restricted system access. Kubernetes namespaces separate workloads from each other.

Check warning on line 448 in environment-setup/configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/configuration.mdx#L448

Did you really mean 'namespaces'?
- **Scanned.** Submitted models are analyzed for vulnerabilities before execution on your infrastructure.
- **Minimal footprint.** The installer only modifies `~/.tracebloc/` and Docker. No system-wide changes.
22 changes: 13 additions & 9 deletions environment-setup/setup-guide.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Setup Guide"
description: "Detailed walkthrough for deploying a tracebloc workspace — requirements, installer internals, GPU support, and verification."

Check warning on line 3 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L3

Did you really mean 'walkthrough'?

Check warning on line 3 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L3

Did you really mean 'tracebloc'?
---

Everything the installer does, explained. This page walks through the full setup process — what each step does, what to expect, and how to verify it worked.
Expand All @@ -23,7 +23,7 @@

**Supported platforms:** macOS (Intel & Apple Silicon) · Linux (x86_64 & arm64) · Windows (x86_64 & arm64)

**Outbound access needed:** The installer downloads container images and connects to the tracebloc platform. Make sure your network allows traffic to `*.docker.io`, `*.tracebloc.io`, `github.com`, and `pypi.org`.

Check warning on line 26 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L26

Did you really mean 'tracebloc'?

---

Expand All @@ -33,7 +33,7 @@

## 2. Register a Client

A **client** is your workspace's identity on the platform. It ties a specific machine to your account and controls what data and use cases are accessible from it.

Check warning on line 36 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L36

Did you really mean 'workspace's'?

Open the [client page](https://ai.tracebloc.io/clients) and click **"+"**.

Expand Down Expand Up @@ -98,27 +98,28 @@
Provisions a lightweight local Kubernetes cluster inside Docker. First run takes 1–2 minutes to download components.

**Step 3/4 — Install tracebloc client**
Prompts for a **workspace name** (e.g. `berlin-team`, `vision-lab`, `ml-mardan`). This identifies the client on your machine and becomes the Kubernetes namespace.
Deploys the tracebloc client into the cluster — no input required. The client runs in a fixed local namespace (`tracebloc`), and one client runs per machine.

Check warning on line 101 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L101

Did you really mean 'tracebloc'?

Check warning on line 101 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L101

Did you really mean 'namespace'?

**Step 4/4 — Connect to tracebloc network**
Prompts for your **Client ID** and **password** from step 2 above. This links your secure local environment to the tracebloc platform so vendors can submit models for evaluation.

Check warning on line 104 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L104

Did you really mean 'tracebloc'?

When it finishes you'll see a summary like:

```
tracebloc client installed successfully

Workspace : <workspace>
Mode : CPU # or GPU
Workspace : tracebloc
Version : 1.4.4 # the client version you're running
Mode : CPU # or GPU

Logs: ~/.tracebloc/
Data: /tracebloc/<workspace>
Data: /tracebloc/tracebloc
```

Install logs are kept in `~/.tracebloc/` if you need to debug anything.

<Note>
To upgrade a one-liner install later, run `helm upgrade <workspace> tracebloc/client -n <namespace> --reset-then-reuse-values` (append `--version <version-number>` to pin). See [Configuration → Upgrade](/environment-setup/configuration#upgrade) for details — `--reset-then-reuse-values` is required so the values applied by the installer are preserved.
To upgrade a one-liner install later, run `helm upgrade tracebloc tracebloc/client -n tracebloc --reset-then-reuse-values` (append `--version <version-number>` to pin). See [Configuration → Upgrade](/environment-setup/configuration#upgrade) for details — `--reset-then-reuse-values` is required so the values applied by the installer are preserved.

Check warning on line 122 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L122

Did you really mean 'tracebloc'?

Check warning on line 122 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L122

Did you really mean 'tracebloc'?
</Note>

### GPU Support
Expand All @@ -136,17 +137,20 @@
After the installer finishes, confirm that your workspace is running:

```bash
kubectl get pods -n <workspace>
kubectl get pods -n tracebloc
```

You should see two pods in `Running` state:
You should see these pods in `Running` state:

| Pod | Role |
|-----|------|
| `mysql-...` | Local metadata store — tracks jobs, metrics, and configuration |
| `tracebloc-jobs-manager-...` | The client — executes training jobs and communicates with the platform |
| `mysql-client-...` | Local metadata store — tracks jobs, metrics, and configuration |
| `tracebloc-jobs-manager-...` | The client — runs training jobs and communicates with the platform |
| `tracebloc-requests-proxy-...` | Routes the client's outbound traffic to the platform |

A `tracebloc-resource-monitor` DaemonSet also runs in the `tracebloc-node-agents` namespace, reporting node capacity.

Check warning on line 151 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L151

Did you really mean 'namespace'?

Then open [ai.tracebloc.io](https://ai.tracebloc.io) and check that your client status shows **Online**. This confirms the client has established a secure connection to the tracebloc backend.

Check warning on line 153 in environment-setup/setup-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/setup-guide.mdx#L153

Did you really mean 'tracebloc'?

<Tip>
Stuck on Offline? Check the [Troubleshooting](/environment-setup/troubleshooting) page — most issues resolve with a pod restart or credential check.
Expand Down
19 changes: 10 additions & 9 deletions environment-setup/troubleshooting.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
---
title: "Troubleshooting"
description: "Common issues and debugging commands for your tracebloc workspace."

Check warning on line 3 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L3

Did you really mean 'tracebloc'?
---

Most issues fall into a few categories: pods not starting, client not connecting, or resource limits being hit. Start with the quick checks below — they cover the most common problems.

For real-time cluster monitoring, try [k9s](https://k9scli.io/) — run `k9s -n <workspace>` to get a live view of pods, logs, and events.
For real-time cluster monitoring, try [k9s](https://k9scli.io/) — run `k9s -n <namespace>` to get a live view of pods, logs, and events.

## Quick Checks

| Symptom | Check | Fix |
|---------|-------|-----|
| Pods not starting | `kubectl describe pod <pod> -n <workspace>` | Check resource limits, Docker status |
| Client shows Offline | `kubectl logs -n <workspace> -l app=tracebloc-jobs-manager` | Verify client ID/password, check network |
| Pods not starting | `kubectl describe pod <pod> -n <namespace>` | Check resource limits, Docker status |
| Client shows Offline | `kubectl logs -n <namespace> -l app=tracebloc-jobs-manager` | Verify client ID/password, check network |
| Docker not running | `docker info` | Start Docker Desktop or daemon |
| Cluster not found | `k3d cluster list` | Re-run the installer |
| GPU not detected | `nvidia-smi` | Install NVIDIA drivers, reboot, re-run installer |
Expand All @@ -33,7 +33,8 @@

| Error Message | Description | Resolution |
|---------------|-------------|------------|
| ServiceBus connection error after Docker restart | When Docker overutilizes local resources and restarts, the ServiceBus connection may fail with `NoneType` errors. | Monitor resources via Docker Dashboard. Restart the jobs manager pod (e.g. in k9s, exit with Ctrl+D) to restore the connection. |

Check warning on line 36 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L36

Did you really mean 'overutilizes'?
| Installer stops: *"This machine already runs the tracebloc client '…'"* | A **different** client is already installed here. tracebloc runs one client per machine. | **Update** it: re-run with the same Client ID. **Switch** clients: `k3d cluster delete tracebloc` (wipes the current client + its local data), then re-run. **Run both:** use a separate machine. |

Check warning on line 37 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L37

Did you really mean 'tracebloc'?

Check warning on line 37 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L37

Did you really mean 'tracebloc'?

## Debugging Commands

Expand All @@ -42,9 +43,9 @@
### Pod status and logs

```bash
kubectl get pods -n <workspace>
kubectl logs <pod-name> -n <workspace>
kubectl describe pod <pod-name> -n <workspace>
kubectl get pods -n <namespace>
kubectl logs <pod-name> -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
```

### Resource usage
Expand All @@ -53,15 +54,15 @@

```bash
kubectl top nodes
kubectl top pods -n <workspace>
kubectl top pods -n <namespace>
```

### Storage

Check that persistent volume claims are bound and have enough capacity:

```bash
kubectl get pvc -n <workspace>
kubectl get pvc -n <namespace>
kubectl get pv
```

Expand All @@ -70,7 +71,7 @@
If pods fail with `ErrImagePull`, verify that the Docker registry secret exists:

```bash
kubectl get secret regcred -n <workspace>
kubectl get secret regcred -n <namespace>
```

## CPU and Memory Optimization
Expand All @@ -95,7 +96,7 @@
If training is slow, these are the factors to look at:

- **Batch size** — larger batches increase GPU utilization up to memory saturation
- **Model complexity** — transformer attention is O(seq_len² x hidden_dim); CNNs scale with kernel x feature map x filters

Check warning on line 99 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L99

Did you really mean 'hidden_dim'?

Check warning on line 99 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L99

Did you really mean 'CNNs'?
- **Precision** — FP16/BF16 can speed up training 2-3x on modern GPUs

Check warning on line 100 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L100

Did you really mean 'GPUs'?
- **Data pipeline** — slow CPU preprocessing (augmentation, tokenization) can bottleneck training

Check warning on line 101 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L101

Did you really mean 'tokenization'?
- **Parallelization** — data parallelism splits batches across GPUs; model parallelism splits the model itself

Check warning on line 102 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L102

Did you really mean 'Parallelization'?

Check warning on line 102 in environment-setup/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (tracebloc-develop) - vale-spellcheck

environment-setup/troubleshooting.mdx#L102

Did you really mean 'GPUs'?