docs: add working with controls tutorial - #154
Merged
Merged
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
pbeckham
commented
Apr 16, 2026
pbeckham
commented
Apr 16, 2026
pbeckham
commented
Apr 16, 2026
pbeckham
commented
Apr 16, 2026
…se, and dark controls paragraph
Contributor
|
Was this released yet? I cannot see this in Kosli 🤔 |
Contributor
Author
No this is a proposed feature. We are making this documentation ahead of development to facilitate discussions and get early feedback before we build it. |
Contributor
|
Okay, I assume we're not releasing this to docs yet then? I think it might confuse our customers. |
pbeckham
commented
Apr 16, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Update the working-with-controls tutorial to match what is actually built in the server and cli repos, replacing proposal-era guesses: - Mark the feature as beta (available) rather than in development. - Remove the non-existent `kosli create control` / `kosli list controls` CLI commands; document the New control form (identifier, name, description, links, tags) and the controls API instead. - Drop the unbuilt Active/Stale/No-decisions status table. - Fix `--compliant=true` (boolean flag) and add `--no-assert` to the `kosli evaluate` examples, which assert on deny by default. - Replace the imaginary top-level `controls:` policy key with the real `attestations:` rules using `type: decision` and `for_control:`. - Replace Decisions/Deployments/Coverage tabs with the real Decisions/Versions tabs and correct the decisions table columns. - Fix broken /reference/controls link. - Swap proposal mockups for real staging screenshots; add a decision detail screenshot; remove the now-orphaned coverage/deployments images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correct details surfaced by the controls project handovers, and switch to US spelling throughout: - Tags are not a create-time field — they are added later from the control detail page and do not version the control; note admin-only writes. - Clarify the validation asymmetry: recording a decision does not require the control to exist, but a policy referencing an unknown control identifier is rejected. - Note that multiple decisions can be recorded for a control on a trail and the most recent is used for enforcement. - Match the decision detail description to the shipped UI: evaluation result and params live in user-data; the Rego policy is the attached evidence file. - State that the feature is rolled out per organization (beta). - organisation -> organization, judgement -> judgment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Claude finished @pbeckham's task in 2m 45s —— View job Review: docs: add working with controls tutorial
SummaryThe tutorial is well-structured and thorough after its many rounds of review. It covers the full controls lifecycle (create → decide → assert → review) with clear CLI examples, a useful Mermaid sequence diagram, and real staging screenshots. No critical issues found. FindingsCriticalNone. Compliance checks (all pass)
Improvement (2)
Suggestion (2)
Overall assessmentThe tutorial is in good shape for a beta feature. Content is accurate against the shipped implementation, the PDP/PEP separation is clearly explained, and the evidence-attachment workflow with |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
pbeckham
marked this pull request as ready for review
June 15, 2026 14:50
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…xist Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jumboduck
approved these changes
Jun 17, 2026
Comment on lines
+123
to
+127
| kosli evaluate trail "$TRAIL_NAME" \ | ||
| --policy supply-chain-policy.rego \ | ||
| --flow "$FLOW_NAME" \ | ||
| --no-assert \ | ||
| --output json > eval-report.json |
Contributor
There was a problem hiding this comment.
Improvement: The --no-assert flag used here is not listed in the kosli evaluate trail CLI reference — that page documents --attestations, --flow, --output, --params, --policy, and --show-input, but no --no-assert. A reader following this example who checks the reference will find no matching flag.
Since the beta callout at the top already sets expectations for undocumented CLI behavior, this isn't blocking — but worth noting for when the reference pages are regenerated.
| --output json > eval-report.json | ||
|
|
||
| # Read the allow/deny result from the report | ||
| is_compliant=$(jq -r '.allow' eval-report.json) |
Contributor
There was a problem hiding this comment.
Suggestion: The jq -r '.allow' extraction assumes the --output json format from kosli evaluate trail includes a top-level allow field. The evaluate trails with OPA tutorial shows a different JSON output structure (result, violations). If the beta version of kosli evaluate has a different output shape than the GA version, a brief comment or note clarifying the expected JSON structure would help readers who cross-reference the two tutorials.
Comment on lines
+259
to
+264
| From here you can: | ||
|
|
||
| - Learn more about [controls](/understand_kosli/controls) | ||
| - Learn more about [environment policies](/getting_started/policies) | ||
| - Learn more about [attestations](/getting_started/attestations) | ||
| - [Evaluate trails with Rego policies](/tutorials/evaluate_trails_with_opa) to automate decision-making |
Contributor
There was a problem hiding this comment.
Suggestion (minor): Other tutorials in this site use <CardGroup> / <Card> components for "next steps" navigation (e.g., the Labs landing page). Replacing this bullet list with cards would give the reader visual targets and be consistent with the rest of the site — though this is polish, not critical.
dangrondahl
added a commit
that referenced
this pull request
Sep 3, 2026
Reviewed the last 20 non-bot PRs (14 touched docs) against what `doc-write` and `doc-review` actually ask for, then fixed what the record showed was wrong. No site content changes — this is all agent tooling. ## Why `doc-review` produced **0 Critical, 5 Improvement and ~20 Suggestion** findings across those 14 PRs. Its genuinely valuable catches were all cross-file consistency checks the skill never asked for: - a changelog entry documenting `kosli update attestation-type`, a command with no reference page (#371) - `template-reference/flow_template.md` out of sync with the schema the same PR regenerated (#296) - the one file an approvals-removal sweep missed, `understand_kosli/how_kosli_works.md:22` (#345) Its weakest findings were prose polish, some already fixed at branch head. Meanwhile its written checklist was dominated by things that always pass or are already enforced by `vale-spellcheck`. Two structural gaps the PR record made obvious: - **Placement was never questioned.** In #305 a pure reference page was authored into `integrations/`, and a human reviewer had to ask for the move — commit `9e7a121 docs: move GitHub Action reference into the Reference section`. That question should come from the review. - **Generated pages were handled inconsistently.** #375 got it right and said so. #374 wrote *"the durable fix is in the `kosli-dev/cli` generator — a hand-edit here is overwritten by the next release"* and then emitted a `Fix this` link scoped to `repo=kosli-dev/docs` telling an agent to edit the generated file anyway, plus 4 inline comments on regenerated files. ## What changed **`doc-review`** — promoted the three accidental wins to named checks, each carrying its precedent. Added placement, redirects and anchor stability. Added an explicit *what not to report* bar and an 8-finding cap. Stopped hand-checking spelling. Dropped the "what looks good" recital that the sticky comment re-renders on every push. **Generated pages** — split into three categories rather than one blanket ban: deterministically regenerated (edit is deleted), agent-synced (edit survives but drifts), and hand-authored despite the directory (`client_reference/overview.md`, `output_and_verbosity.md`). Includes the filename→source mapping, verified upstream: `kosli_attest_sonar.md` ← `cmd/kosli/attestSonar.go`. Also records that **`^` is the CLI's backtick convention** in Go long descriptions, substituted by `kosli docs`. That makes the `^jq^` defect #374 found a generator escaping bug in the Accordion-title path, not a typo in the Go string — so a reviewer can point at the right fix. Worth filing upstream separately. **`doc-write`** — added a Diátaxis→tab placement table so #305 can't recur, plus redirects, anchor stability, and the generated-paths table. **New `doc-structure` skill + monthly workflow** — audits navigation shape and changelog coverage, which per-PR review structurally cannot see, and files issues. Read-only against docs; issues are its only write. Capped at 8 issues, deduplicating against open issues first — its headline check reproduces the `/getting_started/attestations` summary gap, which is already open as #364. **New `scripts/audit_navigation.py` + 22 tests** — the audit's mechanical checks, extracted from an inline heredoc. Three payoffs: 1. `pr-quality.yml` already runs `pytest tests/`, so **CLAUDE.md core rule 2 is now enforced deterministically** — a page file with no `navigation` entry fails the build. No new job needed. 2. `doc-structure.yml` can allow `Bash(python3 scripts/audit_navigation.py:*)` instead of `Bash(python3:*)`, which was arbitrary code execution. 3. Determinism. The inline version had already shipped a bug: `sed 's|\.mdx\?$||'` is a no-op on BSD sed, so every page looked orphaned on macOS. That case is now a regression test. Integrity findings (orphans, dangling entries) are separated from shape findings (single-child groups, deep nesting, Title Case labels, oversized groups, inconsistent icons). Only integrity can fail a build — the script cannot tell a group that should be merged from one deliberately kept separate. `Reference ▸ CLI Reference` is exempt from shape checks: `update-cli-nav.py` generates it from the CLI's command tree, so a single-child `kosli allow` group is upstream truth. Without that exemption the audit reported 54 findings, 21 of them proposing to reshape generated navigation — the same mistake being fixed in `doc-review`. It now reports 33, all hand-maintained. **New `docs-restructure` issue template** — Markdown, not a YAML form: `gh issue create --body` doesn't apply templates mechanically and YAML forms can't be filled from the CLI, so a form would help humans and do nothing for the job. Its URL-impact section encodes the distinction that decides whether a restructure is safe — a group rename changes no URLs, a moved page file needs a `config/redirects.json` entry. `config.yml` keeps blank issues enabled. **`CLAUDE.md`** — documented the six automated PR checks, the generated-page source map, the audit script, and the three skills. Removed the pointer to a `changelog-creator` skill that doesn't exist in this repo. ## Anti-rot pass Swept all three skills for dated claims. Counts ("three of the last fourteen PRs") and current-state assertions ("link-rot reports skipping on most PRs") became durable rules. The link-rot guidance now routes through `gh pr checks` so it self-heals if Mintlify starts running it reliably. Past-tense `Precedent:` items were kept — they're what make the checks concrete. ## Verification - `python3 -m pytest tests/` — 44 passed (22 pre-existing + 22 new) - `python3 scripts/audit_navigation.py --check` — exit 0, integrity clean - `mint broken-links` — no new broken links - Both workflows parse; triggers and permissions confirmed ## Follow-ups, not in this PR - `tutorials/working_with_controls.mdx:24` links to `/getting_started/service-accounts`, which has never existed — the page is `/administration/authentication/service_accounts`. Broken on `main` since #154; `link-rot` reports `skipping`, which is why it went unseen. - The `mintlify-docs` plugin in `kosli-plugins` still ships near-duplicate `doc-writer`/`doc-reviewer` agents that say "update `docs.json` navigation" — the pre-`config/` layout. Needs its own PR there. - The reviews twice asked for `python3` in `doc-review.yml`'s `--allowedTools` to validate JSON payloads and fence balance. Left alone — `Bash(python3:*)` is a security-surface call worth making deliberately.
gsavage
pushed a commit
that referenced
this pull request
Sep 7, 2026
Reviewed the last 20 non-bot PRs (14 touched docs) against what `doc-write` and `doc-review` actually ask for, then fixed what the record showed was wrong. No site content changes — this is all agent tooling. ## Why `doc-review` produced **0 Critical, 5 Improvement and ~20 Suggestion** findings across those 14 PRs. Its genuinely valuable catches were all cross-file consistency checks the skill never asked for: - a changelog entry documenting `kosli update attestation-type`, a command with no reference page (#371) - `template-reference/flow_template.md` out of sync with the schema the same PR regenerated (#296) - the one file an approvals-removal sweep missed, `understand_kosli/how_kosli_works.md:22` (#345) Its weakest findings were prose polish, some already fixed at branch head. Meanwhile its written checklist was dominated by things that always pass or are already enforced by `vale-spellcheck`. Two structural gaps the PR record made obvious: - **Placement was never questioned.** In #305 a pure reference page was authored into `integrations/`, and a human reviewer had to ask for the move — commit `9e7a121 docs: move GitHub Action reference into the Reference section`. That question should come from the review. - **Generated pages were handled inconsistently.** #375 got it right and said so. #374 wrote *"the durable fix is in the `kosli-dev/cli` generator — a hand-edit here is overwritten by the next release"* and then emitted a `Fix this` link scoped to `repo=kosli-dev/docs` telling an agent to edit the generated file anyway, plus 4 inline comments on regenerated files. ## What changed **`doc-review`** — promoted the three accidental wins to named checks, each carrying its precedent. Added placement, redirects and anchor stability. Added an explicit *what not to report* bar and an 8-finding cap. Stopped hand-checking spelling. Dropped the "what looks good" recital that the sticky comment re-renders on every push. **Generated pages** — split into three categories rather than one blanket ban: deterministically regenerated (edit is deleted), agent-synced (edit survives but drifts), and hand-authored despite the directory (`client_reference/overview.md`, `output_and_verbosity.md`). Includes the filename→source mapping, verified upstream: `kosli_attest_sonar.md` ← `cmd/kosli/attestSonar.go`. Also records that **`^` is the CLI's backtick convention** in Go long descriptions, substituted by `kosli docs`. That makes the `^jq^` defect #374 found a generator escaping bug in the Accordion-title path, not a typo in the Go string — so a reviewer can point at the right fix. Worth filing upstream separately. **`doc-write`** — added a Diátaxis→tab placement table so #305 can't recur, plus redirects, anchor stability, and the generated-paths table. **New `doc-structure` skill + monthly workflow** — audits navigation shape and changelog coverage, which per-PR review structurally cannot see, and files issues. Read-only against docs; issues are its only write. Capped at 8 issues, deduplicating against open issues first — its headline check reproduces the `/getting_started/attestations` summary gap, which is already open as #364. **New `scripts/audit_navigation.py` + 22 tests** — the audit's mechanical checks, extracted from an inline heredoc. Three payoffs: 1. `pr-quality.yml` already runs `pytest tests/`, so **CLAUDE.md core rule 2 is now enforced deterministically** — a page file with no `navigation` entry fails the build. No new job needed. 2. `doc-structure.yml` can allow `Bash(python3 scripts/audit_navigation.py:*)` instead of `Bash(python3:*)`, which was arbitrary code execution. 3. Determinism. The inline version had already shipped a bug: `sed 's|\.mdx\?$||'` is a no-op on BSD sed, so every page looked orphaned on macOS. That case is now a regression test. Integrity findings (orphans, dangling entries) are separated from shape findings (single-child groups, deep nesting, Title Case labels, oversized groups, inconsistent icons). Only integrity can fail a build — the script cannot tell a group that should be merged from one deliberately kept separate. `Reference ▸ CLI Reference` is exempt from shape checks: `update-cli-nav.py` generates it from the CLI's command tree, so a single-child `kosli allow` group is upstream truth. Without that exemption the audit reported 54 findings, 21 of them proposing to reshape generated navigation — the same mistake being fixed in `doc-review`. It now reports 33, all hand-maintained. **New `docs-restructure` issue template** — Markdown, not a YAML form: `gh issue create --body` doesn't apply templates mechanically and YAML forms can't be filled from the CLI, so a form would help humans and do nothing for the job. Its URL-impact section encodes the distinction that decides whether a restructure is safe — a group rename changes no URLs, a moved page file needs a `config/redirects.json` entry. `config.yml` keeps blank issues enabled. **`CLAUDE.md`** — documented the six automated PR checks, the generated-page source map, the audit script, and the three skills. Removed the pointer to a `changelog-creator` skill that doesn't exist in this repo. ## Anti-rot pass Swept all three skills for dated claims. Counts ("three of the last fourteen PRs") and current-state assertions ("link-rot reports skipping on most PRs") became durable rules. The link-rot guidance now routes through `gh pr checks` so it self-heals if Mintlify starts running it reliably. Past-tense `Precedent:` items were kept — they're what make the checks concrete. ## Verification - `python3 -m pytest tests/` — 44 passed (22 pre-existing + 22 new) - `python3 scripts/audit_navigation.py --check` — exit 0, integrity clean - `mint broken-links` — no new broken links - Both workflows parse; triggers and permissions confirmed ## Follow-ups, not in this PR - `tutorials/working_with_controls.mdx:24` links to `/getting_started/service-accounts`, which has never existed — the page is `/administration/authentication/service_accounts`. Broken on `main` since #154; `link-rot` reports `skipping`, which is why it went unseen. - The `mintlify-docs` plugin in `kosli-plugins` still ships near-duplicate `doc-writer`/`doc-reviewer` agents that say "update `docs.json` navigation" — the pre-`config/` layout. Needs its own PR there. - The reviews twice asked for `python3` in `doc-review.yml`'s `--allowedTools` to validate JSON payloads and fence balance. Left alone — `Bash(python3:*)` is a security-surface call worth making deliberately.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tutorials/working_with_controls.mdx— a tutorial covering how to define controls in Kosli, record decisions against them withkosli attest decision, require controls in environment policies viafor_control, enforce them withkosli assert artifact --environment, and review a control's decisions and version history.config/navigation.jsonto include the new page under a new "Controls" group in the Tutorials section.Aligned with the shipped implementation
POST /api/v2/controls/{org}) — there is nokosli create control/kosli list controlsCLI.attestations:rules withtype: decisionandfor_control:(not a top-levelcontrols:key).kosli evaluateexamples use--no-assert(it asserts on deny by default);--compliantis shown as a boolean flag (--compliant=true).Closes kosli-dev/server#5355