feat(orchestration): refuse an unoffered skill_triggered target at resolution; read plugin manifests as Claude Code does

validate_plugins now fails plan when a skill_triggered skill_name is not among
the offered skills, so a misnamed skill no longer costs a full run before it
finishes ERROR (a ${row...} name is checked on its expanded row; the checker
gate stays for detached grades of recorded runs).

Plugin scanning now matches Claude Code's plugins reference and a CLI 2.1.273
spike: a manifest skills path ADDS to the default skills/ instead of replacing
it, a declared path may name one skill, a root holding SKILL.md is a
single-skill plugin, and a skill is named by its SKILL.md frontmatter name.
The deliberately invalid skill_not_offered fixture is deleted; unit tests cover
the refusal on plan and resolution.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: uipreliga

.claude/notes/agents.md

@@ -648,22 +648,32 @@ adapter used to scan the authored path its own way. claude-code loaded nothing f
 skills directory, with no error, so an activation suite scored recall 0.0 and read exactly
 like a skill that never triggers.
 
-- **Both authored layouts are accepted.** For each root, the manifest-declared skill dirs
-  that exist are scanned (default `skills/`). If none exists, the root is a bare skills
-  directory. The manifest's `skills` field is read, not hardcoded, so a plugin that
-  relocates its skills keeps working.
+- **A plugin root is read the way Claude Code reads it.** The default `skills/` is always
+  scanned, and each path the manifest's `skills` field declares ADDS to it; a declared path
+  may parent skills or be one skill (it holds `SKILL.md`). A root holding `SKILL.md` is a
+  single-skill plugin. If a root yields nothing that way, it is read as a bare skills
+  directory. A skill's name is its frontmatter `name`, else its directory name: Claude
+  Code invokes the frontmatter name, so a gate keyed on the directory name would refuse a
+  skill that loads. Confirmed by the plugins reference ("Adds to the default: `skills`")
+  and a CLI 2.1.273 spike on 2026-09-16; the moved reader had treated the manifest as a
+  REPLACEMENT, which dropped the default `skills/` of any plugin that declared extras.
 - **Only skills are staged.** A plugin's agents, hooks, commands and MCP servers are
   dropped on every harness, claude-code included. That also removes a confound: a project
   subagent beside `skills/` can no longer answer the request the skill should answer.
-- **The staged manifest is `{"name": "coder-eval-plugins"}` and nothing else.** A spike
-  with `claude -p --plugin-dir` showed that a manifest declaring `"skills": ["skills"]`
-  loaded no skill, symlinked or copied. A name-only manifest loads the `skills/` default.
+- **The staged manifest is `{"name": "coder-eval-plugins"}` and nothing else.** A
+  2026-09-17 spike with `claude -p --plugin-dir` showed a staged root whose manifest declared
+  `"skills": ["skills"]` load no skill; a 2026-09-16 spike on CLI 2.1.273 loaded a real
+  `["./skills"]` fine. The name-only manifest loads the `skills/` default either way.
 - **Refusal is at resolution.** `validate_plugins` runs in `validate_resolved_task`, so a
-  path with no skill, an unresolvable path, or one skill name from two sources fails
-  `plan`. A run can no longer measure the model WITHOUT the skill under test and look normal.
+  path with no skill, an unresolvable path, one skill name from two sources, or a
+  `skill_triggered` `skill_name` the plugins do not offer fails `plan` before the run is paid
+  for. A name still holding a `${row...}` placeholder is checked on its expanded row. The
+  cost: a task whose skill under test comes from a template or `setting_sources` while it
+  also sets `agent.plugins` is refused, because only plugin skills are offered.
 - **`skills_offered` is recorded** in `environment_info` and passed to the checker.
-  `skill_triggered` raises `CheckerMisuseError` when its `skill_name` is not offered: the
-  positive control cannot run, so the row escalates instead of scoring 0.0.
+  `skill_triggered` still raises `CheckerMisuseError` when its `skill_name` is not offered.
+  Resolution catches every new run first; the checker gate remains for a detached grade of
+  a recorded run, where resolution does not re-run.
 
 Delivery, per harness:
 

.claude/notes/contracts.md

@@ -191,7 +191,8 @@ judge's author-written `files:` entry, because it names one file of the solution
 compared against and traversal out of the staged copy is always a mistake.
 
 `skill_triggered` escalates the same way when its `skill_name` is not among the skills
-`agent.plugins` offered (`CheckContext.skills_offered`). The agent was never offered the
+`agent.plugins` offered (`CheckContext.skills_offered`). Resolution refuses the same task
+first (`validate_plugins`), so this gate fires only on a detached grade of a recorded run. The agent was never offered the
 skill, so the positive control cannot run. Scored as 0.0, every positive row of an
 activation suite would read as a skill that never triggers. The gate applies only when the
 task sets plugins: with `skills_offered` `None` the criterion scores as before, so a skill

docs/TASK_DEFINITION_GUIDE.md

@@ -1358,7 +1358,7 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab
 
 **Requires agent telemetry.** This criterion reads `turn_records`, so it only works against a real agent run (not a static check). With no turn records it reports `score=0.0` and an `error`.
 
-**The skill must be offered.** When the task sets `agent.plugins`, coder-eval stages the skills those paths offer (a plugin root or a bare skills directory) and records their names in `environment_info.skills_offered`. A `skill_name` that is not among them makes the criterion finish `ERROR`, not `0.0`: the positive control cannot run. This applies even when the skill reaches the agent another way (for example a template's `.claude/skills/`): with `agent.plugins` set, put the skill under test in a plugin path. A plugin path that offers no skill fails `coder-eval plan`. See [Plugin staging](agents/HARNESS_PARITY.md#plugin-staging).
+**The skill must be offered.** When the task sets `agent.plugins`, coder-eval stages the skills those paths offer (a plugin root or a bare skills directory) and records their names (the `SKILL.md` frontmatter `name`) in `environment_info.skills_offered`. A `skill_name` that is not among them fails `coder-eval plan` before the run is paid for: the positive control cannot run. A `skill_name` taken from a dataset row is checked on each expanded row. This applies even when the skill reaches the agent another way (for example a template's `.claude/skills/`): with `agent.plugins` set, put the skill under test in a plugin path. Re-grading a recorded run whose `skill_name` was not offered finishes `ERROR`, not `0.0`. A plugin path that offers no skill also fails `coder-eval plan`. See [Plugin staging](agents/HARNESS_PARITY.md#plugin-staging).
 
 **Classification metrics.** `skill_triggered` returns a `ClassificationCriterionResult`, so on a [dataset-backed task](#dataset) the suite aggregator computes accuracy / precision / recall / F1 / confusion matrix across all rows. Gate the suite with `suite_thresholds` using any of: `accuracy`, `macro_f1`, `weighted_f1`, `micro_f1`, or per-label `precision.<label>` / `recall.<label>` / `f1.<label>` (labels are `yes` / `no`). The run exits non-zero if any listed metric falls below its minimum.
 

docs/agents/HARNESS_PARITY.md

@@ -659,8 +659,12 @@ whose cap fires should not look like a task whose harness hung.
 
 ## Plugin staging
 
-Each `agent.plugins[].path` names a plugin root (`<path>/skills/<name>/SKILL.md`) or a bare
-skills directory (`<path>/<name>/SKILL.md`). Both layouts work on every harness. Before the
+Each `agent.plugins[].path` names a plugin root or a bare skills directory
+(`<path>/<name>/SKILL.md`). A plugin root is read as Claude Code reads it: the default
+`skills/` plus every path its `.claude-plugin/plugin.json` `skills` field declares (a
+declared path may be one skill), or the root itself when it holds `SKILL.md`. A skill's name
+is its `SKILL.md` frontmatter `name`, else its directory name. Every layout works on every
+harness. Before the
 agent starts, coder-eval stages the skills into one root, `<run_dir>/plugin_root`: a
 `.claude-plugin/plugin.json` that names `coder-eval-plugins`, and one `skills/<name>` symlink
 per skill. Each harness receives that root in its native way. Only skills are staged: a
@@ -669,9 +673,10 @@ included. Claude Code names staged skills `coder-eval-plugins:<skill>`, not `<pl
 Files beside the skills also stay behind: a skill that reads `${CLAUDE_PLUGIN_ROOT}/scripts/`
 or a shared `references/` directory at the plugin root cannot find it. Keep a skill's files
 inside its own `<name>/` directory.
-A path that offers no skill, or two paths that offer the same skill name, fail `coder-eval plan`.
-`environment_info.skills_offered` records the staged skill names. A `skill_triggered` criterion
-whose `skill_name` is not in that list finishes `ERROR`, not 0.0.
+A path that offers no skill, two paths that offer the same skill name, or a `skill_triggered`
+criterion whose `skill_name` the plugins do not offer fail `coder-eval plan`, before the run is
+paid for. `environment_info.skills_offered` records the staged skill names; re-grading a
+recorded run whose `skill_name` is not in that list finishes `ERROR`, not 0.0.
 
 ## Reproducing
 

docs/tutorials/07-plugin-in-claude-code.md

@@ -160,8 +160,7 @@ covers telling them apart with `/doctor` and `/context`.
 | No `/coder-eval:` commands after installing | Check `/plugin`; re-run the install |
 | A skill offers to install the CLI, or Bash reports `command not found` | The CLI isn't installed or isn't on `PATH` — accept the offer, or install it yourself |
 | `coder-eval run` matches nothing | Wrong directory — use the path `init` reported in step 2 |
-| Step 5 stops with a config error about `agent.plugins` | `SKILL_SOURCE_PATH` is unset, or points at a directory that holds no `<name>/SKILL.md` |
-| A `skill_triggered` row finishes `ERROR` | The skill name is not among the skills the plugin path offers |
+| Step 5 stops with a config error about `agent.plugins` | `SKILL_SOURCE_PATH` is unset, points at a directory that holds no `<name>/SKILL.md`, or the suite's `skill_name` is not the frontmatter `name` of a skill that path offers |
 
 To update after the marketplace moves, `/plugin marketplace update coder-eval`; to
 remove it, `/plugin uninstall`.

plugins/coder-eval/skills/check-skill/SKILL.md

@@ -176,8 +176,9 @@ export SKILL_SOURCE_PATH="$(pwd)/.claude"
 
 Keep it an environment variable rather than baking an absolute path into the YAML — the
 suite is committed and re-run on other machines. If the variable is unset, or the path holds
-no skill, `coder-eval plan` fails with a config error. If `skill_name` is not among the skills
-the path offers, every `skill_triggered` row finishes `ERROR` rather than scoring 0.
+no skill, `coder-eval plan` fails with a config error. It fails the same way when `skill_name`
+is not among the skills the path offers (their `SKILL.md` frontmatter `name`), before any
+run is paid for.
 
 `skill_name` must be the bare name even when the skill comes from a plugin and is
 invoked as `plugin:skill` — the checker strips the namespace before comparing. A

src/coder_eval/orchestration/plugin_staging.py

@@ -2,8 +2,11 @@
 
 The canonical root is ``<root>/.claude-plugin/plugin.json`` plus
 ``<root>/skills/<name>`` (a symlink to the authored skill directory, or a copy where
-symlinks fail). Both authored layouts are accepted: a plugin root whose manifest (or
-default ``skills/``) parents the skills, and a bare skills directory.
+symlinks fail). A plugin root is read the way Claude Code reads it: the default
+``skills/`` plus every manifest-declared path, where a path may parent skills or be one
+skill, and a root holding ``SKILL.md`` is a single-skill plugin. A bare skills
+directory is accepted too. A skill's name is its ``SKILL.md`` frontmatter ``name``,
+else its directory name.
 
 Rationale: .claude/notes/agents.md § Skills, per harness
 """
@@ -17,6 +20,9 @@
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
+import yaml
+
+from coder_eval.models import SkillTriggeredCriterion
 from coder_eval.orchestration.harness_contract import TaskResolutionError
 from coder_eval.utils import expand_env_vars
 
@@ -60,7 +66,7 @@ def resolve_plugin_path(raw: str) -> Path:
     return root
 
 
-def _declared_skill_dirs(root: Path) -> list[Path]:
+def _declared_skill_paths(root: Path) -> list[Path]:
     manifest = root.joinpath(*_MANIFEST_RELPATH)
     declared: list[str] = []
     if manifest.is_file():
@@ -74,34 +80,55 @@ def _declared_skill_dirs(root: Path) -> list[Path]:
                 declared = [value]
             elif isinstance(value, list):
                 declared = [entry for entry in value if isinstance(entry, str)]
-    if not declared:
-        declared = [_DEFAULT_SKILLS_SUBDIR]
     return [(root / relative).resolve() for relative in declared]
 
 
-def scan_plugin_skills(plugins: Sequence[LocalPluginConfig]) -> dict[str, Path]:
-    """Skill name -> its directory, over every entry, both authored layouts.
+def _skill_name(skill_dir: Path) -> str:
+    """The frontmatter ``name`` of ``<skill_dir>/SKILL.md``, else the directory name."""
+    text = (skill_dir / _SKILL_FILE).read_text(encoding="utf-8", errors="replace")
+    if text.startswith("---"):
+        front, _, _rest = text[3:].partition("\n---")
+        try:
+            data: Any = yaml.safe_load(front)
+        except yaml.YAMLError:
+            data = None
+        if isinstance(data, dict) and isinstance(data.get("name"), str) and data["name"].strip():
+            return data["name"].strip()
+    return skill_dir.name
+
+
+def _skill_dirs(root: Path) -> list[Path]:
+    """Every skill directory one ``plugins:`` root offers, in Claude Code's reading order."""
+    candidates = [root / _DEFAULT_SKILLS_SUBDIR, *_declared_skill_paths(root)]
+    found: list[Path] = []
+    for candidate in candidates:
+        if (candidate / _SKILL_FILE).is_file():
+            found.append(candidate)
+        elif candidate.is_dir():
+            found += [skill_file.parent for skill_file in sorted(candidate.glob(f"*/{_SKILL_FILE}"))]
+    if found:
+        return found
+    if (root / _SKILL_FILE).is_file():
+        return [root]
+    return [skill_file.parent for skill_file in sorted(root.glob(f"*/{_SKILL_FILE}"))]
+
 
-    For each root, the manifest-declared skill directories that exist are scanned;
-    if none exists, the root itself is a bare skills directory.
+def scan_plugin_skills(plugins: Sequence[LocalPluginConfig]) -> dict[str, Path]:
+    """Skill name -> its directory, over every entry and every accepted layout.
 
     Raises:
         PluginStagingError: an unresolvable path, a skill name from two sources, or no skill at all.
     """
     skills: dict[str, Path] = {}
     for plugin in plugins:
-        root = resolve_plugin_path(plugin["path"])
-        candidates = [directory for directory in _declared_skill_dirs(root) if directory.is_dir()] or [root]
-        for candidate in candidates:
-            for skill_file in sorted(candidate.glob(f"*/{_SKILL_FILE}")):
-                skill_dir = skill_file.parent
-                previous = skills.get(skill_dir.name)
-                if previous is not None and previous != skill_dir.resolve():
-                    raise PluginStagingError(
-                        f"ambiguous skill name {skill_dir.name!r}: agent.plugins offers it from both "
-                        + f"{previous} and {skill_dir.resolve()}"
-                    )
-                skills[skill_dir.name] = skill_dir.resolve()
+        for skill_dir in _skill_dirs(resolve_plugin_path(plugin["path"])):
+            name, source = _skill_name(skill_dir), skill_dir.resolve()
+            previous = skills.get(name)
+            if previous is not None and previous != source:
+                raise PluginStagingError(
+                    f"ambiguous skill name {name!r}: agent.plugins offers it from both {previous} and {source}"
+                )
+            skills[name] = source
     if not skills:
         paths = [plugin["path"] for plugin in plugins]
         raise PluginStagingError(
@@ -114,12 +141,25 @@ def scan_plugin_skills(plugins: Sequence[LocalPluginConfig]) -> dict[str, Path]:
 def validate_plugins(task: TaskDefinition) -> None:
     """Resolution-time refusal; no-op when plugins is unset or empty.
 
+    Also refuses a ``skill_triggered`` criterion whose ``skill_name`` the plugins do
+    not offer, before the run is paid for. A name still holding a ``${row...}``
+    placeholder is checked on its expanded row instead.
+
     Raises:
-        PluginStagingError: see ``scan_plugin_skills``.
+        PluginStagingError: see ``scan_plugin_skills``, or a ``skill_triggered`` target not offered.
     """
     plugins = task.agent.plugins if task.agent is not None else None
-    if plugins:
-        scan_plugin_skills(plugins)
+    if not plugins:
+        return
+    offered = scan_plugin_skills(plugins)
+    targets = {c.skill_name for c in task.success_criteria if isinstance(c, SkillTriggeredCriterion)}
+    missing = sorted(name for name in targets if "${" not in name and name not in offered)
+    if missing:
+        raise PluginStagingError(
+            f"skill_triggered names skill(s) {missing} but agent.plugins offers only {sorted(offered)}: "
+            + "the positive control cannot run. With agent.plugins set, the skill under test must come "
+            + "from a plugin path (a skill from a template or setting_sources is not offered)."
+        )
 
 
 def link_or_copy(source: Path, target: Path) -> None: