Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 10 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -222,25 +222,24 @@ jobs:

- name: Run end-to-end smoke test
run: |
# This profile intentionally has HIGH security findings against it
# in CI (env-derived DB passwords look like hardcoded secrets to
# CDS-SEC-001/003/041, and cosign is not installed on this runner,
# so CDS-VER-001 also fires) — the same reason the "Verify
# unverifiable images fail closed" step below expects `cds
# security` to exit 1. `cds test` bundles security into its single
# exit code, so assert only that validate/plan/render passed and
# that the sole failing stage is "security", rather than requiring
# the whole command to exit 0.
# With the env-derived secret values set above, the security stage
# passes: since #979 the renderer references in initDbEnv.* and
# the waiver reason no longer false-positive as hardcoded secrets,
# and only the known MEDIUM gaps (CDS-SEC-080/081) remain, which
# do not fail the stage. Image verification is separate: cosign is
# not installed on this runner, so the "Verify unverifiable images
# fail closed" step below still expects `cds security
# --verify-images` to exit 1 with CDS-VER-* findings.
set +e
test_output="$(cds test local-dagster-postgres-superset 2>&1)"
test_status=$?
set -e
echo "$test_output"
grep -Eq "^\[PASS\] validate$" <<<"$test_output"
grep -Eq "^\[FAIL\] security$" <<<"$test_output"
grep -Eq "^\[PASS\] security$" <<<"$test_output"
grep -Eq "^\[PASS\] plan$" <<<"$test_output"
grep -Eq "^\[PASS\] render$" <<<"$test_output"
test "$test_status" -eq 1
test "$test_status" -eq 0

- name: Generate MVP plan
run: cds plan local-dagster-postgres-superset --output /tmp/plan.json
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,21 @@ The format is based on Keep a Changelog.

### Fixed

- Fixed `cds security` false positives on the reference production profile
(#979): the secret-value rules CDS-SEC-001/003/033 no longer flag values
that are entirely a renderer reference (`config.passwordFrom`,
`${secrets.x}`, ...), which the declarative engine sees unresolved in the
planned config but the renderer resolves to `${CDS_*}` placeholders at
render time (new opt-in `ignoreRendererReferences` rule flag, rule set
1.0.1); the CDS-SEC-013 secret-reuse check skips such references as well;
the waiver-reason entropy exemption now also matches the flattened
`security.waivers.*.reason` path (spec-level keys flatten without the
`spec.` prefix); and CDS-SEC-031 matches the env file path relative to the
project root (the same marker walk `resolve_env_file_path` uses, now shared
via `cli/security_common.py`), so a bare project-root `.env` never fires
when the CLI is invoked from another directory (e.g. with
`CDS_PROFILE_PATH` set).

- Fixed the `helm` render target lacking the provide-side contract-host
cross-check that the compose target got for issue #1010: the Kubernetes
renderer only validated the consume side (E084), so a provided
Expand Down
22 changes: 1 addition & 21 deletions cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
NIS2_ARTICLE_21_MEASURES,
SEVERITY_ORDER,
infer_profile_class,
resolve_project_root,
)
from .state import format_state_output, group_services_by_health, parse_compose_ps_json
from .up_runner import (
Expand Down Expand Up @@ -738,27 +739,6 @@ def resolve_profile_path(profile: str | None) -> str:
)


def resolve_project_root(profile_path: str) -> Path:
"""
Resolve a project root for output artifacts.

The resolver walks up from the selected profile location and picks the
first directory containing `.cds` (CDS's own state directory, checked
first so it takes priority over an unrelated ancestor repository),
`pyproject.toml`, or `.git`. If no marker is found, it falls back to the
current working directory.
"""
start = Path(profile_path).resolve().parent
for directory in [start, *start.parents]:
if (
(directory / ".cds").exists()
or (directory / "pyproject.toml").exists()
or (directory / ".git").exists()
):
return directory
return Path.cwd().resolve()


def resolve_env_file_path(profile_path: str) -> Path:
"""
Resolve the default .env location for a profile.
Expand Down
5 changes: 5 additions & 0 deletions cli/resources/rule-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,11 @@
"type": "boolean",
"default": true
},
"ignoreRendererReferences": {
"type": "boolean",
"default": false,
"description": "For rules that match on secret-like *values*: skip values that are entirely a renderer reference expression (bare 'config.x'/'secrets.x'/'bindings.x'/'service.x'/'certificates.x' or the ${...}-wrapped form). The declarative engine scans the planned config where those references are still unresolved strings, but the renderer resolves them before anything is executed (secrets.*/certificates.* become ${CDS_*} Compose placeholders), so they are pointers, not literal secrets (#979)."
},
"codeEnforced": {
"type": "boolean",
"default": true,
Expand Down
13 changes: 8 additions & 5 deletions cli/resources/rule-set.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "rule-schema.json",
"version": "1.0.0",
"version": "1.0.1",
"metadata": {
"name": "cds-core-security-rules",
"description": "Core security validation rules for cds validate --security",
Expand Down Expand Up @@ -30,7 +30,8 @@
"Move the value to environment variables or a secret backend.",
"Replace literal values with references."
],
"enabled": true
"enabled": true,
"ignoreRendererReferences": true
},
{
"id": "CDS-SEC-002",
Expand Down Expand Up @@ -81,7 +82,8 @@
"Store private keys in a secret backend or mounted secret file.",
"Do not keep cryptographic material directly in profile YAML."
],
"enabled": true
"enabled": true,
"ignoreRendererReferences": true
},
{
"id": "CDS-SEC-004",
Expand Down Expand Up @@ -439,7 +441,7 @@
},
{
"id": "CDS-SEC-031",
"$comment": "Only the project-root .env is covered by this repo's `/.env` .gitignore entry (leading slash pins it to the root). Any nested .env (profile-scoped, rendered/generated output, or otherwise) is NOT covered by that ignore rule and can be committed accidentally, so the pathPatterns below only flag paths with at least one directory component before the filename; a bare root '.env' is the expected, already-ignored location and must not fire here.",
"$comment": "Only the project-root .env is covered by this repo's `/.env` .gitignore entry (leading slash pins it to the root). Any nested .env (profile-scoped, rendered/generated output, or otherwise) is NOT covered by that ignore rule and can be committed accidentally, so the pathPatterns below only flag paths with at least one directory component before the filename; a bare root '.env' is the expected, already-ignored location and must not fire here. The engine matches the env file path relative to the project root (resolved from the profile path, same marker walk as resolve_env_file_path), not relative to the invocation cwd, so an absolute project-root .env (e.g. when the CLI runs from another directory with CDS_PROFILE_PATH set) is still seen as the bare '.env' and never matches (#979).",
"title": "Env file path appears repository-managed or unmanaged",
"severity": "medium",
"category": "file-output-safety",
Expand Down Expand Up @@ -512,7 +514,8 @@
"Use environment or secret-backend references for sensitive fields.",
"Keep profiles declarative and secret-free."
],
"enabled": true
"enabled": true,
"ignoreRendererReferences": true
},
{
"id": "CDS-SEC-040",
Expand Down
92 changes: 79 additions & 13 deletions cli/security.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@
resolved_secret_values,
)
from .secrets import load_secrets_from_env
from .security_common import SECRET_KEY_RE, SEVERITY_ORDER, infer_profile_class
from .security_common import (
SECRET_KEY_RE,
SEVERITY_ORDER,
infer_profile_class,
resolve_project_root,
)
from .utils import _compose_service_name

_PROFILE_SCOPES = {
Expand Down Expand Up @@ -192,29 +197,42 @@ def _flatten_env_secrets(secrets: dict[str, str]) -> list[tuple[str, str, Any]]:
return [("<env>", f"secrets.{key}", value) for key, value in secrets.items()]


def _normalize_scan_path(path: Path) -> str:
def _normalize_scan_path(path: Path, project_root: Path | None = None) -> str:
"""
Render a scanned file path relative to the project root when possible, so
the project-root `.env` is always the bare ".env" no matter which directory
the CLI was invoked from (e.g. with CDS_PROFILE_PATH set); fall back to a
cwd-relative path, then to the absolute path.
"""
resolved = path.resolve()
try:
return resolved.relative_to(Path.cwd().resolve()).as_posix()
except ValueError:
return resolved.as_posix()
for base in (project_root, Path.cwd().resolve()):
if base is None:
continue
try:
return resolved.relative_to(base).as_posix()
except ValueError:
continue
return resolved.as_posix()


def _flatten_env_inputs(
secrets: dict[str, str],
env_file: str | None,
project_root: Path | None = None,
) -> list[tuple[str, str, Any]]:
"""
Emit both loaded .env secrets and the env file path itself when present.

Some rules evaluate how a secret-bearing env file is located or managed
rather than inspecting individual secret values. Represent the file path as
a synthetic flat item so those rules can use the same matcher.
a synthetic flat item so those rules can use the same matcher. The path is
normalized against `project_root` (not just cwd) so a bare project-root
`.env` stays bare regardless of the invocation directory.
"""
items = _flatten_env_secrets(secrets)
env_path = Path(env_file) if env_file is not None else Path(".env")
if env_path.exists():
scan_path = _normalize_scan_path(env_path)
scan_path = _normalize_scan_path(env_path, project_root)
items.append(("<env>", scan_path, scan_path))
return items

Expand All @@ -223,7 +241,10 @@ def _is_windows() -> bool:
return os.name == "nt"


def _flatten_file_permissions(env_file: str | None) -> list[tuple[str, str, Any]]:
def _flatten_file_permissions(
env_file: str | None,
project_root: Path | None = None,
) -> list[tuple[str, str, Any]]:
"""
Emit the POSIX permission bits of the secret-bearing .env file as a
synthetic item whose value is the octal mode string (e.g. "0644").
Expand All @@ -240,7 +261,7 @@ def _flatten_file_permissions(env_file: str | None) -> list[tuple[str, str, Any]
mode = stat.S_IMODE(env_path.stat().st_mode)
except OSError:
return []
return [("<env>", f"{_normalize_scan_path(env_path)}.permissions", f"{mode:04o}")]
return [("<env>", f"{_normalize_scan_path(env_path, project_root)}.permissions", f"{mode:04o}")]


def _permission_mode(value: str) -> int | None:
Expand Down Expand Up @@ -329,6 +350,28 @@ def _is_secret_reference(value: Any) -> bool:
return isinstance(value, str) and bool(_SECRET_REF_RE.match(value))


# The renderer's reference vocabulary (cli/renderer.py's _resolve_expr), in
# both its bare form (e.g. "config.passwordFrom", "secrets.cache_password",
# used in module config positions the renderer resolves, such as initDbEnv)
# and its ${...}-wrapped form (e.g. "${config.passwordFrom}"). A value that
# is entirely such an expression is a pointer the renderer resolves at render
# time (secrets.*/certificates.* become ${CDS_*} Compose placeholders), never
# a literal secret, so secret-value rules that opt in via the rule's
# `ignoreRendererReferences` flag skip it (#979).
_RENDERER_REF_RE = re.compile(
r"^\$\{(?:secrets|certificates|config|bindings|service)\.[^}]+\}$" # ${config.x}
r"|^(?:secrets|certificates|config|bindings|service)\.[A-Za-z0-9_.]+$" # config.x
)

def _is_renderer_reference(value: Any) -> bool:
"""
Returns True if value is entirely a renderer reference expression
(bare "config.x"/"secrets.x"/... or "${config.x}"/"${secrets.x}"/...),
i.e. a value the renderer resolves at render time rather than a literal.
"""
return isinstance(value, str) and bool(_RENDERER_REF_RE.match(value))


# ---------------------------------------------------------------------------
# Profile class inference
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -394,9 +437,13 @@ def _redact(value: Any) -> str | None:
# secret heuristic. Unlike _NON_SECRET_PATH_SUFFIXES, these only suppress the
# check for the exact documented field, so an unrelated key that happens to
# end in "reason" (e.g. a genuine secret named "authFailureReason") is still
# scanned normally.
# scanned normally. Both the "spec."-prefixed and the bare form are listed:
# _flatten_profile_by_module emits spec-level keys without the "spec." prefix
# (e.g. "security.waivers.runtimePolicy.reason"), while callers that flatten
# the raw profile document use the prefixed form.
_NON_SECRET_PATH_PATTERNS = (
"spec.security.waivers.*.reason",
"security.waivers.*.reason",
)
def _eval_condition(
path: str,
Expand Down Expand Up @@ -498,6 +545,12 @@ def _check_secret_reuse(
for module_id, path, value in flat_items:
if not isinstance(value, str) or not value:
continue
# Renderer references (e.g. "secrets.backup_encryption_key") are
# pointers resolved at render time, not secret values: sharing one
# reference across fields is the intended indirection, not reuse of
# the underlying value (#979).
if _is_renderer_reference(value):
continue
if not SECRET_KEY_RE.search(path.split(".")[-1]):
continue
value_to_locations.setdefault(value, []).append((module_id, path))
Expand Down Expand Up @@ -535,12 +588,20 @@ def _rule_matches(

findings: list[dict[str, Any]] = []
match = rule["match"]
# Rules flagging secret-like *values* can opt out of renderer references
# (bare "config.x" or "${secrets.x}" expressions): the declarative engine
# scans the planned config where those references are still unresolved
# strings, but the renderer resolves them before anything is executed,
# and rendered output only ever contains ${CDS_*} placeholders (#979).
skip_renderer_refs = bool(rule.get("ignoreRendererReferences", False))

for module_id, path, value in flat_items:
# Skip unresolved ${secrets.*} references in profile config.
# They are intentional indirections, not real values.
if _is_secret_reference(value):
continue
if skip_renderer_refs and _is_renderer_reference(value):
continue

key = path.split(".")[-1] if path else ""

Expand Down Expand Up @@ -1382,8 +1443,13 @@ def run_security_validation(
secrets, secret_diags = load_secrets_from_env(env_file)

flat_profile = _flatten_profile_by_module(profile, profile_dir=profile_path.parent)
flat_env = _flatten_env_inputs(secrets, env_file)
flat_permissions = _flatten_file_permissions(env_file)
# Normalize scanned file paths (e.g. the .env path CDS-SEC-031 matches on)
# against the same project root resolve_env_file_path uses, so a bare
# project-root .env never matches regardless of the invocation directory
# (e.g. when CDS_PROFILE_PATH points at a profile while cwd is elsewhere).
scan_project_root = resolve_project_root(str(profile_path))
flat_env = _flatten_env_inputs(secrets, env_file, project_root=scan_project_root)
flat_permissions = _flatten_file_permissions(env_file, project_root=scan_project_root)

# Planning and rendering the profile is only useful when some enabled
# rule actually declares the "rendered-compose" scope -- e.g. a custom
Expand Down
22 changes: 22 additions & 0 deletions cli/security_common.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,32 @@
from __future__ import annotations

import re
from pathlib import Path
from typing import Any

SEVERITY_ORDER = {"high": 0, "medium": 1, "low": 2}


def resolve_project_root(profile_path: str) -> Path:
"""
Resolve a project root for output artifacts.

The resolver walks up from the selected profile location and picks the
first directory containing `.cds` (CDS's own state directory, checked
first so it takes priority over an unrelated ancestor repository),
`pyproject.toml`, or `.git`. If no marker is found, it falls back to the
current working directory.
"""
start = Path(profile_path).resolve().parent
for directory in [start, *start.parents]:
if (
(directory / ".cds").exists()
or (directory / "pyproject.toml").exists()
or (directory / ".git").exists()
):
return directory
return Path.cwd().resolve()

# Closed set of informational compliance control categories a rule-set.json
# rule can be tagged with (rule-schema.json's $defs.complianceCategory),
# each mapped to a short display label plus the rationale documented in
Expand Down
14 changes: 11 additions & 3 deletions docs/production-hardening-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,19 +178,27 @@ Expected: `cds state` shows every service healthy; the proxy's TLS entry
points answer (for example `curl -v https://<host>/`) and the backend ports
answer only through the proxy. `cds down` stops the stack.

## 4. Expected findings (snapshot 2026-10-26)
## 4. Expected findings (snapshot 2026-10-30)

`cds security team-dagster-postgres-superset --environment prod` currently
exits 1 with the following findings. The set is stable; a deviation in either
direction is the operator's signal to stop and investigate.

Renderer references (for example `config.passwordFrom` in `initDbEnv.*`,
`${secrets.*}` aliases) and the free-text waiver reason are no longer
flagged: since #979 the secret-value rules CDS-SEC-001/003/033 skip values
that are entirely a renderer reference (new `ignoreRendererReferences` rule
flag), the CDS-SEC-013 reuse check skips such references, the waiver-reason
entropy exemption covers the flattened `security.waivers.*.reason` path, and
CDS-SEC-031 matches the env file path relative to the project root so a bare
project-root `.env` never fires regardless of the invocation directory.
Rendered output still contains only `${CDS_*}` placeholders.

| Finding | Count | Classification | Action / tracking |
| --- | --- | --- | --- |
| `CDS-SEC-001`/`CDS-SEC-033` on `initDbEnv.*`; `CDS-SEC-003` on `initDbEnv.*` and on the waiver reason | 11 | False positive: the flagged values are renderer references (for example `config.passwordFrom`) or free-text waiver reasons, not secrets; rendered output contains only `${CDS_*}` placeholders | [#979](https://github.com/RonaldHensbergen/composable-data-stack/issues/979) |
| `CDS-SEC-074` on `services.dagster-webserver.ports[0]` and `services.superset.ports[0]` | 2 | True positive: the admin backends still publish directly to the host, allowing a proxy bypass | Remove the backend host publishes (proxy becomes the sole entry point), or set `spec.security.waivers.plaintextEndpointExposure.reason` |
| `CDS-SEC-081` on the admin-facing services | 3 | Known gap: no identity module is bound | [#370](https://github.com/RonaldHensbergen/composable-data-stack/issues/370) |
| `CDS-SEC-080` on `spec.modules` | 1 | Known gap: the backup policy is declared at profile level (`spec.backup`), which the rule does not yet recognize as contract consumption | [backup-restore.md](backup-restore.md) follow-ups ([#210](https://github.com/RonaldHensbergen/composable-data-stack/issues/210)) |
| `CDS-SEC-013` on `backup.encryption.keyFrom` / `backup.encryptionKeyFrom` | 2 | Accepted: one backup encryption key is deliberately referenced by both the module capability and the profile policy | None (single purpose, single key) |
| `W103` on `spec.security.waivers.runtimePolicy` | 1 | Informational: the documented non-root waiver for images that set `USER` in their Dockerfiles | Keep the waiver reason accurate |

Additional `CDS-SEC-004` (placeholder/weak secret) or `CDS-SEC-012` (too
Expand Down
Loading
Loading