Resolve secrets/vars expressions at dispatch + full detection state in the UI

CLAUDE.md

@@ -186,24 +186,47 @@ variables), and `environments` (distinct job `environment:` names, case-insensit
 deduped, slug-filtered so dynamic `${{ … }}` names are never stored). All scanners share
 `workflow_parse::scan_context_refs` (identifier `[A-Za-z_][A-Za-z0-9_]*` ≤200 B, dedupe,
 cap 100, sort). Two read endpoints compute "referenced but not configured" as a set
-difference at QUERY time (`db/workflows.rs::missing_secret_refs/workspace_var_refs/
-missing_environment_refs` — `jsonb_array_elements_text` LATERAL with a `jsonb_typeof`
-guard so pre-feature/failed-parse metadata reads as empty; anti-join semantics: a secret
-name counts as configured via workspace scope, same-repo repository scope, or any
-environment scope): `GET …/secrets/requirements` (`secrets.read`) and
-`GET …/environments/requirements` (`content.read`). Handlers re-filter names at read
-time against the configurable-name allow-lists (defense-in-depth against hand-edited
-metadata) and cap output (200 names, 20 refs/name, true `referenceCount`). YAML never
-mints rows — the UI offers one-click create through the ordinary RBAC'd endpoints:
-`DetectedRequirementsCard` (secrets page right rail; "Add" pre-fills `SecretFormDialog`
-via `presetName`/`presetRepository`, repo scope only when every reference shares one
-repo) and `DetectedEnvironmentsCard` (environments page; "Create" pre-fills
-`EnvironmentFormDialog` via `presetName`). Both requirements queries live under their
-feature's react-query prefix, so every mutation's existing invalidation clears satisfied
-warnings automatically. The environment detail page lists `boundWorkflows` (from
-`list_binding_environment`, case-insensitive) linking to the workflow detail pages.
-Detection refreshes on every repo sync; workflows synced before this feature light up
-after their next push/manual re-sync.
+difference at QUERY time (`db/workflows.rs::secret_ref_states/workspace_var_refs/
+environment_ref_states` — `jsonb_array_elements_text` LATERAL with a `jsonb_typeof`
+guard so pre-feature/failed-parse metadata reads as empty; configured semantics: a
+secret name counts as configured via workspace scope, same-repo repository scope, or
+any environment scope, and the LATERAL picks the highest-precedence match's id):
+`GET …/secrets/requirements` (`secrets.read`) and `GET …/environments/requirements`
+(`content.read`) return EVERY detected (name, repository) pair with
+`configured`/`configuredId`, not just missing ones. Handlers re-filter names at read
+time against charset allow-lists (defense-in-depth against hand-edited metadata;
+deliberately looser than the configurable-name rule so reserved-prefix refs like
+DOCKER_TOKEN stay visible — the client marks them unconfigurable via the shared
+`features/secrets/lib/secretNameRules.ts`) and cap output (200 entries, 20 refs each,
+true `referenceCount`). YAML never mints rows — the repo-grouped "Detected in
+workflows" cards (`DetectedRequirementsCard` on the secrets page,
+`DetectedEnvironmentsCard` on the environments page) show Configured entries as
+success badges linking to the detail pages and missing ones with one-click Add/Create
+through the ordinary RBAC'd endpoints (dialog `presetName`/`presetRepository`). Both
+requirements queries live under their feature's react-query prefix, so every
+mutation's existing invalidation updates the cards automatically. The workflow detail
+Metadata tab surfaces `secretRefs`/`varRefs`/`environments` (with a re-sync hint when
+the stored parse predates detection), the environment detail page lists
+`boundWorkflows` (from `list_binding_environment`, case-insensitive), and the pipeline
+Environment tab shows the job's `plan.environment` binding. Detection refreshes on
+every repo sync; `workflow_parse::PARSER_VERSION` is stamped into metadata and the
+sync skip-condition re-parses stored workflows once after any parser bump (bump it
+whenever parse output changes, or new metadata never reaches already-synced
+workflows).
+
+**Dispatch-time expression resolution.** `workflow_parse::substitute_context_refs`
+resolves GitHub-style `${{ secrets.NAME }}` / `${{ vars.NAME }}` blocks (dot or
+bracket form; the trimmed inner expression must be exactly one such ref — compound
+expressions and other contexts pass through untouched, it is a substitutor, not an
+evaluator). `scheduler::dispatch` applies it to plan env VALUES and step `run`
+strings after decrypting the job's secrets: known secrets resolve to their value,
+unknown secrets and all vars resolve to "" (GitHub's unset semantics; no variables
+store yet). Substitution happens in dispatch memory only — stored plans keep the
+literals, so API responses never carry resolved values and reruns re-substitute with
+current secrets. Mask registration runs after substitution, and secret plaintexts are
+always masks, so a substituted-into-run secret still masks in logs. Secrets also
+continue to inject as env vars under their own names (precedence: plan env <
+workspace < repository < environment).
 
 **Notification Center (operational inbox).** Notifications are a per-user, actionable
 PROJECTION of the immutable `audit_logs` ledger — the Activity Feed keeps the complete

backend/src/db/workflows.rs

@@ -243,14 +243,16 @@ pub struct PushRunnableWorkflow {
 }
 
 /// One workflow-YAML reference to a secret/var/environment name, with the
-/// repository and workflow it came from. Names only — never values.
+/// repository and workflow it came from and — when something satisfies it —
+/// the id of the configured row. Names and ids only, never values.
 #[derive(Debug, sqlx::FromRow)]
 pub struct RequirementRefRow {
     pub name: String,
     pub repository_id: Uuid,
     pub repository_name: String,
     pub workflow_id: Uuid,
     pub workflow_path: String,
+    pub configured_id: Option<Uuid>,
 }
 
 /// CTE expanding one metadata ref array across a workspace's workflows.
@@ -279,31 +281,37 @@ fn refs_cte(key: &str) -> String {
     )
 }
 
-/// Secret names referenced by workflow YAML with no configured secret that
-/// could satisfy them: a name counts as configured when a workspace-scoped
-/// secret, a repository-scoped secret on the referencing repo, or an
-/// environment-scoped secret in any environment carries it (which environment
-/// applies is a dispatch-time question — "any environment" keeps the rule
-/// simple and matches how precedence is explained in the UI).
-pub async fn missing_secret_refs(
+/// Every secret name referenced by workflow YAML, with the id of the secret
+/// that would satisfy it when one exists: a name counts as configured for a
+/// referencing repo when a workspace-scoped secret, a repository-scoped
+/// secret on that repo, or an environment-scoped secret in any environment
+/// carries it (which environment applies is a dispatch-time question). The
+/// LATERAL picks the highest-precedence match so the UI can link straight
+/// to it.
+pub async fn secret_ref_states(
     pool: &PgPool,
     workspace_id: Uuid,
 ) -> sqlx::Result<Vec<RequirementRefRow>> {
     let sql = refs_cte("secretRefs")
         + r#"
-        SELECT name, repository_id, repository_name, workflow_id, workflow_path
+        SELECT refs.name, refs.repository_id, refs.repository_name,
+               refs.workflow_id, refs.workflow_path, s.id AS configured_id
         FROM refs
-        WHERE NOT EXISTS (
-            SELECT 1 FROM secrets s
+        LEFT JOIN LATERAL (
+            SELECT s.id
+            FROM secrets s
             WHERE s.workspace_id = $1
               AND s.name = refs.name
               AND (
                    (s.repository_id IS NULL AND s.environment_id IS NULL)
                 OR s.repository_id = refs.repository_id
                 OR s.environment_id IS NOT NULL
               )
-        )
-        ORDER BY name, repository_name, workflow_path
+            ORDER BY (s.environment_id IS NOT NULL) DESC,
+                     (s.repository_id IS NOT NULL) DESC
+            LIMIT 1
+        ) s ON TRUE
+        ORDER BY refs.name, refs.repository_name, refs.workflow_path
         LIMIT 1000
         "#;
     sqlx::query_as::<_, RequirementRefRow>(&sql)
@@ -313,14 +321,15 @@ pub async fn missing_secret_refs(
 }
 
 /// Every `${{ vars.NAME }}` reference across the workspace — informational
-/// only (the platform doesn't manage plain variables), so no anti-join.
+/// only (the platform doesn't manage plain variables), never configured.
 pub async fn workspace_var_refs(
     pool: &PgPool,
     workspace_id: Uuid,
 ) -> sqlx::Result<Vec<RequirementRefRow>> {
     let sql = refs_cte("varRefs")
         + r#"
-        SELECT name, repository_id, repository_name, workflow_id, workflow_path
+        SELECT name, repository_id, repository_name, workflow_id, workflow_path,
+               NULL::uuid AS configured_id
         FROM refs
         ORDER BY name, repository_name, workflow_path
         LIMIT 1000
@@ -331,21 +340,21 @@ pub async fn workspace_var_refs(
         .await
 }
 
-/// Environment names bound by workflow YAML with no matching environment row
-/// (case-insensitive, like dispatch resolution).
-pub async fn missing_environment_refs(
+/// Every environment name bound by workflow YAML, with the id of the
+/// matching environment row when it exists (case-insensitive, like dispatch
+/// resolution).
+pub async fn environment_ref_states(
     pool: &PgPool,
     workspace_id: Uuid,
 ) -> sqlx::Result<Vec<RequirementRefRow>> {
     let sql = refs_cte("environments")
         + r#"
-        SELECT name, repository_id, repository_name, workflow_id, workflow_path
+        SELECT refs.name, refs.repository_id, refs.repository_name,
+               refs.workflow_id, refs.workflow_path, e.id AS configured_id
         FROM refs
-        WHERE NOT EXISTS (
-            SELECT 1 FROM environments e
-            WHERE e.workspace_id = $1 AND lower(e.name) = lower(refs.name)
-        )
-        ORDER BY name, repository_name, workflow_path
+        LEFT JOIN environments e
+          ON e.workspace_id = $1 AND lower(e.name) = lower(refs.name)
+        ORDER BY refs.name, refs.repository_name, refs.workflow_path
         LIMIT 1000
         "#;
     sqlx::query_as::<_, RequirementRefRow>(&sql)
@@ -363,7 +372,8 @@ pub async fn list_binding_environment(
 ) -> sqlx::Result<Vec<RequirementRefRow>> {
     let sql = refs_cte("environments")
         + r#"
-        SELECT name, repository_id, repository_name, workflow_id, workflow_path
+        SELECT name, repository_id, repository_name, workflow_id, workflow_path,
+               NULL::uuid AS configured_id
         FROM refs
         WHERE lower(refs.name) = lower($2)
         ORDER BY repository_name, workflow_path

backend/src/handlers/environments.rs

@@ -144,23 +144,24 @@ pub async fn summary(
 
 /// GET /api/workspaces/{workspace_id}/environments/requirements
 ///
-/// Environment names bound by workflow YAML (`environment:`) with no
-/// matching environment row — detected at sync time, computed as a set
-/// difference at query time. Deliberately never auto-created: YAML must not
-/// mint workspace resources past RBAC; the UI offers a one-click create
-/// through the ordinary RBAC'd endpoint instead.
+/// Every environment name bound by workflow YAML (`environment:`), detected
+/// at sync time, with its configured state — `configuredId` is the matching
+/// environment row's id when one exists. Missing names are deliberately
+/// never auto-created: YAML must not mint workspace resources past RBAC;
+/// the UI offers a one-click create through the ordinary RBAC'd endpoint
+/// instead.
 pub async fn requirements(
     State(state): State<AppState>,
     CurrentUser(user): CurrentUser,
     Path(workspace_id): Path<Uuid>,
 ) -> AppResult<Json<serde_json::Value>> {
     authz::require_permission(&state.pool, user.id, workspace_id, authz::CONTENT_READ).await?;
 
-    let missing = db::workflows::missing_environment_refs(&state.pool, workspace_id).await?;
+    let refs = db::workflows::environment_ref_states(&state.pool, workspace_id).await?;
     Ok(Json(json!({
         // Read-time re-filter (the secrets requirements pattern): only names
-        // an environment row could actually take surface as creatable.
-        "environments": super::secrets::group_requirements(missing, |name| {
+        // an environment row could actually take surface.
+        "environments": super::secrets::group_requirements(refs, |name| {
             validate_name(name).is_ok()
         }),
     })))

backend/src/handlers/secrets.rs

@@ -228,65 +228,91 @@ pub(crate) fn valid_ref_ident(name: &str) -> bool {
         && chars.all(|c| c.is_ascii_alphanumeric() || c == '_')
 }
 
-/// Fold requirement rows into per-name entries: names sorted (BTreeMap),
-/// capped at 200 names and 20 references each — `referenceCount` carries the
-/// true total. `keep` is the read-time allow-list filter.
+/// Fold requirement rows into per-(repository, name) entries — the shape the
+/// repo-grouped detection cards render directly. Sorted by repository then
+/// name (BTreeMap key), capped at 200 entries and 20 workflow references
+/// each with `referenceCount` carrying the true total. `keep` is the
+/// read-time charset allow-list. `configuredId` is the covering secret's or
+/// matching environment's id (null when missing). Names and ids only, never
+/// values.
 pub(crate) fn group_requirements(
     rows: Vec<db::workflows::RequirementRefRow>,
     keep: impl Fn(&str) -> bool,
 ) -> Vec<serde_json::Value> {
-    const MAX_NAMES: usize = 200;
-    const MAX_REFS_PER_NAME: usize = 20;
-
-    let mut grouped: std::collections::BTreeMap<String, (i64, Vec<serde_json::Value>)> =
+    const MAX_ENTRIES: usize = 200;
+    const MAX_REFS_PER_ENTRY: usize = 20;
+
+    struct Entry {
+        repository_id: Uuid,
+        configured_id: Option<Uuid>,
+        count: i64,
+        references: Vec<serde_json::Value>,
+    }
+    let mut grouped: std::collections::BTreeMap<(String, String), Entry> =
         std::collections::BTreeMap::new();
     for row in rows {
         if !keep(&row.name) {
             continue;
         }
-        if !grouped.contains_key(&row.name) && grouped.len() >= MAX_NAMES {
+        let key = (row.repository_name.clone(), row.name.clone());
+        if !grouped.contains_key(&key) && grouped.len() >= MAX_ENTRIES {
             continue;
         }
-        let entry = grouped.entry(row.name).or_default();
-        entry.0 += 1;
-        if entry.1.len() < MAX_REFS_PER_NAME {
-            entry.1.push(json!({
-                "repositoryId": row.repository_id,
-                "repositoryName": row.repository_name,
+        let entry = grouped.entry(key).or_insert_with(|| Entry {
+            repository_id: row.repository_id,
+            configured_id: None,
+            count: 0,
+            references: Vec::new(),
+        });
+        entry.count += 1;
+        if entry.configured_id.is_none() {
+            entry.configured_id = row.configured_id;
+        }
+        if entry.references.len() < MAX_REFS_PER_ENTRY {
+            entry.references.push(json!({
                 "workflowId": row.workflow_id,
                 "workflowPath": row.workflow_path,
             }));
         }
     }
     grouped
         .into_iter()
-        .map(|(name, (count, references))| {
-            json!({ "name": name, "referenceCount": count, "references": references })
+        .map(|((repository_name, name), entry)| {
+            json!({
+                "name": name,
+                "repositoryId": entry.repository_id,
+                "repositoryName": repository_name,
+                "configured": entry.configured_id.is_some(),
+                "configuredId": entry.configured_id,
+                "referenceCount": entry.count,
+                "references": entry.references,
+            })
         })
         .collect()
 }
 
 /// GET /api/workspaces/{workspace_id}/secrets/requirements
 ///
-/// Workflow-declared requirements detected at sync time: secret names the
-/// YAML references (`${{ secrets.X }}`) with no configured secret that could
-/// satisfy them, plus `${{ vars.X }}` references (informational — the
-/// platform doesn't manage plain variables). Names only, never values.
+/// Workflow-declared requirements detected at sync time: every secret name
+/// the YAML references (`${{ secrets.X }}`) with its configured state and,
+/// when satisfied, the id of the covering secret; plus `${{ vars.X }}`
+/// references (informational — the platform doesn't manage plain
+/// variables). The charset filter is deliberately loose (`valid_ref_ident`,
+/// not the configurable-name rule) so reserved-prefix refs like
+/// DOCKER_TOKEN stay VISIBLE — the client marks them unconfigurable instead
+/// of silently hiding them. Names and ids only, never values.
 pub async fn requirements(
     State(state): State<AppState>,
     CurrentUser(user): CurrentUser,
     Path(workspace_id): Path<Uuid>,
 ) -> AppResult<Json<serde_json::Value>> {
     authz::require_permission(&state.pool, user.id, workspace_id, authz::SECRETS_READ).await?;
 
-    let missing = db::workflows::missing_secret_refs(&state.pool, workspace_id).await?;
+    let refs = db::workflows::secret_ref_states(&state.pool, workspace_id).await?;
     let vars = db::workflows::workspace_var_refs(&state.pool, workspace_id).await?;
 
     Ok(Json(json!({
-        // Only names that could actually become overup secrets surface as
-        // missing — a lowercase or GITHUB_*-reserved ref is unconfigurable
-        // here (GitHub folds case; we don't) and would be a dead-end button.
-        "secrets": group_requirements(missing, |name| validate_name(name).is_ok()),
+        "secrets": group_requirements(refs, valid_ref_ident),
         "vars": group_requirements(vars, valid_ref_ident),
     })))
 }