Merge branch 'main' into perf/openapi-index-lookups

The branch forked before #328, #356 and #373, so three of the four files
needed more than a textual resolution:

- The test helpers it calls were consolidated while it was open. ymap,
  yscalar, yalias and ymerge became ynode.Map/Scalar/Alias/Merge in #373,
  and yamlNode became openapitest.YAMLNode in #328. The bodies are
  identical, so these are renames.
- TestRawChildNode_IsNotTheMergeAwareView asserted that a repeated key
  resolves to opposite ends in the two readers. #356 made RawChildNode
  take the last pair, as the parser does, so they now agree. The case is
  rewritten to assert the agreement rather than deleted: a reader
  drifting back to first-wins is worth failing on.
- nodeview.go kept main's DocumentPath, walkPointer and tokenless, and
  main's ynode.MergeTag over the local const this branch predates. The
  index is additive to all of it.

Author: OmarAlJarrah

.golangci.yml

@@ -10,6 +10,7 @@ linters:
     - gocritic
     - misspell
     - nilerr
+    - nolintlint # a suppression outliving what it suppressed is invisible without this
     - prealloc
     - revive
     - unconvert
@@ -22,6 +23,19 @@ linters:
       lines: 70
       statements: -1
       ignore-comments: true
+    nolintlint:
+      # A directive that suppresses nothing reads as a live constraint on the code
+      # under it, and nothing else in the gate can tell the two apart: the tree
+      # stays green whether the finding is real or long gone. Each of these makes
+      # one way of writing an inert directive fail instead.
+      #
+      # allow-unused only reaches directives naming an ENABLED linter. One naming a
+      # disabled or nonexistent linter is ignored by the nolint processor and
+      # reported by nothing — which is how //nolint:forcetypeassert stood here
+      # without ever suppressing anything. That gap is GitHub #306.
+      allow-unused: false # it no longer suppresses anything — delete it
+      require-specific: true # bare //nolint hides findings nobody chose to accept
+      require-explanation: true # the rationale is what a later reader re-checks against
     gocognit:
       # Calibrated against the finished tree, whose worst function scores 21.
       # Raising this is how a function that should have been split stays whole,

CLAUDE.md

@@ -20,13 +20,23 @@ Pipeline: **compilers** (spec → IR) → **IR passes** (IR → IR) → **emitte
 
 ## The documents are the spec — read them first
 
+`ls docs/*.md` is the set; what follows is why each one matters, not how many there are.
+
 - **`docs/ir-design.md` is normative.** Field names and struct shapes in it are the contract;
   receiver methods and helpers are not. When implementing the IR, match its shapes exactly.
+- **`docs/emitter-design.md` is normative for the emitter half**, the way `ir-design.md` is for the
+  IR. Nothing under `emitters/` exists yet, so read it before writing the first one.
 - `docs/architecture.md` — pipeline stages, package layout, layering rules, milestones.
 - `docs/ir-spec-matrix.md` — the union of source-format capabilities the IR is designed against.
 - `docs/prior-art.md` — the evidence base (oagen, Kiota, TypeSpec/TCGC) and the specific mistakes
   each Morphic decision is designed to avoid. Read this before proposing IR changes; most
   "simplifications" that come to mind are failure modes already rejected here.
+- `docs/reference-learnings.md` — the same evidence base widened to every shipped generator it
+  surveys (its header names them), each finding carrying a verdict on a Morphic decision and
+  citing the repo it came from.
+- `docs/micro-compiler-design.md` and `docs/micro-compiler-plan.md` — the restructuring that
+  produced today's `compilers/openapi`. Both record work that has landed; read them for why the
+  package boundaries fall where they do, not as a proposal or a backlog.
 
 ## Invariants that must not be violated
 
@@ -35,7 +45,7 @@ claim (lossless, spec-agnostic, many-target). Before changing any of them, re-re
 in the docs.
 
 1. **The IR is the ABI.** Compilers and emitters never see each other. A compiler's only output is
-   an IR document + diagnostics; a emitter's only input is an IR document + its own options.
+   an IR document + diagnostics; an emitter's only input is an IR document + its own options.
 2. **Lossless by default, lowered late.** Compilers never flatten (no `allOf` merging, no
    union-to-optional-fields collapse, no primary-response selection). Composition, unions,
    visibility, discriminators, encodings, streaming stay in source-semantic form. Lowering to what
@@ -73,8 +83,13 @@ in the docs.
 ## Go representation conventions the design mandates
 
 - **Closed sums = sealed interfaces**: unexported marker method (`typeDef()`), one concrete struct
-  per kind, a `Kind()` accessor for switch-dispatch, and a generated switch-completeness test over
-  the kind enum (the `assertNever` lesson). JSON encodes sums with an adjacent `kind` tag.
+  per kind, a `Kind()` accessor for switch-dispatch, and a switch-completeness test over the kind
+  enum (the `assertNever` lesson). None of it is code-generated — the module has no `go:generate`
+  at all (`grep -rn go:generate --include='*.go' .` is empty). `TestTypeDef_KindDispatchIsComplete`
+  iterates a hand-written `allKinds`, and `TestTypeDef_HandWrittenKindListsAreComplete` holds that
+  list and every other hand-written kind list to the kinds the `ir` sources declare, so adding a
+  kind without updating a list reddens there instead of quietly narrowing what a test covers. JSON
+  encodes sums with an adjacent `kind` tag.
 - **No `float64` anywhere in the IR.** Numeric values, defaults, and constraints use arbitrary-
   precision decimal strings (`BigVal`). This is a hard rule (the TypeSpec `Numeric` lesson).
 - **Values are a separate channel from types** (`Value`/`ValueKind`), per the TypeSpec Type-vs-Value
@@ -96,17 +111,20 @@ compilers/   Layer 1 — the Compiler contract and the format-keyed registry. Im
              compilers, compilers/compile, its own internal/* (+ own format libs); never each
              other, never emitters/engine.
     internal/  Layer 1 — that compiler's own packages, each with its own allowlist rather than
-             the compiler's. openapi has thirteen; the ordering among them is real and enforced,
-             from diag (reaches only ir) up to operation. None may reach the compiler above it.
+             the compiler's. The ordering among them is real and enforced, from diag (reaches
+             only ir) up to operation. None may reach the compiler above it.
 pass/        Layer 1 — IR → IR passes. Imports ir only.
 emitters/*   Layer 2 — imports ir + emitter contract; never compiler. (Not built yet.)
 engine/      Layer 3 — orchestration; imports everything below.
-cmd/morphic/ Layer 4 — CLI; imports engine.
+cmd/morphic/ Layer 4 — CLI; imports ir + engine.
+cmd/morphic-harness/
+             Layer 4 — sweeps a spec or directory through the oracles; imports internal/harness.
 internal/    Test/tooling infrastructure, outside the pipeline (harness, archtest, testspec).
 ```
 
 The layering is enforced by `internal/archtest`, and **its `rules` map is the source of truth** —
-this diagram is prose, and deliberately does not name the thirteen. Read them off the tree:
+this diagram is prose, and deliberately neither names nor counts a compiler's internal packages.
+Read them off the tree:
 
 ```bash
 git ls-files '*/*.go' | xargs -n1 dirname | sort -u
@@ -132,20 +150,30 @@ These all exist already — extend them rather than building a parallel mechanis
   Corpus under `testdata/golden/`.
 - **Capability conformance corpus** (`testdata/conformance/`): one minimal spec per
   `ir-spec-matrix.md` row per format that can express it, asserting lossless capture. This is what
-  keeps "lossless by default" honest.
+  keeps "lossless by default" honest. The row↔spec mapping is machine-read, not prose: matrix rows
+  carry stable keys, each case names the keys it witnesses, and
+  `compilers/openapi/conformance_matrix_test.go` requires every expressible row to be witnessed or
+  listed with a reason. What it cannot check is whether a spec that *names* a row exercises that
+  capability — that claim is read by a reviewer, so weigh it like any other.
 - **Oracles**: `internal/harness` drives a spec through no-panic → no error diagnostic →
-  `irverify` invariants → JSON round-trip → determinism. `irverify` is the structural-invariant
-  checker (stable IDs, no dangling refs, neutral naming, routable `Unmodeled`, in-range
-  provenance); its findings are `Violation` values — *our* bugs — deliberately a channel separate
-  from `ir.Diagnostic`, which reports problems in the source spec.
+  `irverify` invariants → JSON round-trip → determinism → order-invariance, stopping at the first
+  one that fires. `harness.Check` is the list — read it there rather than trusting this sentence;
+  `go run ./cmd/morphic-harness <file|dir>` runs them over one spec or a whole tree. `irverify` is
+  the structural-invariant checker (stable IDs, no dangling refs, neutral naming, routable
+  `Unmodeled`, in-range provenance); its findings are `Violation` values — *our* bugs —
+  deliberately a channel separate from `ir.Diagnostic`, which reports problems in the source spec.
 - **Architecture test**: `internal/archtest`, per the layering section above.
 
 Beyond those, "verify by executing" below has consequences specific enough to write down as
 assertion shapes:
 
 - **Order-dependence needs a two-order diff.** Compile the same source twice with the declaration
   order swapped and `cmp.Diff` the two documents. A single-order test passes on *both* orders of a
-  colliding lowering, which is why the pointer collisions survived the suite.
+  colliding lowering, which is why the pointer collisions survived the suite. The order-invariance
+  oracle above is the general form of this and already runs it across the corpus under `testdata/`,
+  so the usual way to cover a new construct is to add a spec the sweep reaches, not to hand-roll
+  the diff. A targeted case still earns its place when it pins one lowering, because the oracle
+  proves order-independence only for the constructs its inputs happen to contain.
   - A fixture's own declaration order is part of the test. A golden that happens to declare things
     in the order the *correct* lowering already produced cannot see the fix at all: reverting it
     leaves the golden green, and only the two-order oracle reddens. If a single-order case is meant
@@ -200,9 +228,12 @@ below are the ones most likely to bite in this codebase — the full guide gover
 - **Docs:** GoDoc on every exported symbol starting with its name, complete sentences; package
   comment on every package; comments explain *why*, not what.
 - **Serialization:** explicit JSON struct tags on every field; `omitempty` only on optional
-  fields; custom `MarshalJSON`/`UnmarshalJSON` for special forms (the IR's sum types and
-  `BigVal` do this); never `float64` for money — and in this repo, never in the IR at all, per the
-  representation conventions above.
+  fields; custom codecs for special forms, but not symmetrically — each member of the IR's
+  `TypeDef` sum has a `MarshalJSON` that writes its adjacent `kind` tag, and *decoding* is
+  centralized in `(*TypeRegistry).UnmarshalJSON`, which reads that tag; no member unmarshals
+  itself, and `BigVal` has no codec at all because it *is* a string type and already marshals as
+  a JSON string. `grep -rnE 'func .*(Unm|M)arshalJSON' ir/` is the current list; never `float64`
+  for money — and in this repo, never in the IR at all, per the representation conventions above.
 - **Logging:** `log/slog` only, injected — but note the stronger repo invariant: pipeline
   stages don't log at all; they return diagnostics.
 

README.md

@@ -93,40 +93,77 @@ go build -o morphic ./cmd/morphic
 ### CLI
 
 `morphic compile` lowers one OpenAPI 3.x spec into Morphic IR JSON on stdout, and writes
-diagnostics to stderr.
+diagnostics to stderr. Stdout is indented for reading; a file written with `-o` is compact, which
+is about half the bytes, unless `--pretty` asks for the indented form. `morphic validate` runs the
+same pipeline over the same spec for the diagnostics and the exit code alone, writing no IR
+anywhere.
 
 ```bash
 morphic compile openapi.yaml                 # IR JSON to stdout
 morphic compile openapi.yaml -o api.ir.json  # ...or to a file
+morphic validate openapi.yaml                # diagnostics and exit code only
 ```
 
 ```
 usage:
   morphic <command> [flags]
   morphic compile <spec-file> [flags]
+  morphic validate <spec-file> [flags]
 ```
 
 `morphic`, `morphic help`, and `morphic` with a help flag (`-h`, `--help` or `-help`) print the
-command list. `morphic help compile` and `morphic compile --help` print a command's flags. Help
-always prints to stdout and exits `0`.
+command list. `morphic help <command>` and `morphic <command> --help` print a command's flags.
+Help always prints to stdout and exits `0`.
 
-The flags below are `compile`'s:
+| Flag | Commands | Meaning |
+|---|---|---|
+| `--fail-on error\|warning` | both | Exit non-zero when a diagnostic at or above this severity is emitted (default `error`). |
+| `--skip-validate` | both | Skip the referential-integrity `validate` pass. |
+| `-o <file>` | `compile` | Write IR JSON to `<file>` instead of stdout, compact rather than indented. |
+| `--pretty` | `compile` | Indent the JSON `-o` writes; stdout is indented either way. |
+| `--explain <json-pointer>` | `compile` | Report what compiling produced at this source coordinate instead of writing the document. |
+| `--opt <key>=<value>` | both | Set one option on the compiler the spec selects. Repeatable; a repeated key is refused. |
+
+Diagnostics print one per line as `<severity> <code> <location>: <message>`, where `<location>` is
+`<path>#<pointer>` for a finding in a spec file, a bare pointer for one an IR pass made about the
+document, and absent for one raised before any document existed.
+
+Both commands use the same exit codes: `0` clean (and for any help request); `1` the spec has
+problems — a diagnostic reached the `--fail-on` threshold, or it could not be lowered at all, which
+covers an undecodable file, an unrecognized or unsupported format, and a version no compiler claims;
+`2` the invocation or the filesystem was wrong — a bad flag or argument, a spec that could not be
+read, an output that could not be written. Nothing about the spec's own contents reaches `2`.
+
+#### Compiler options
+
+`--opt` names an option in the vocabulary of whichever compiler recognizes the spec — morphic
+itself knows none of them, and an unknown name is refused by the compiler rather than ignored.
+The OpenAPI compiler accepts:
+
+| Option | Values | Meaning |
+|---|---|---|
+| `grouping` | `tags` (default), `path-prefix` | How operations are grouped into operation groups. |
+| `allow-external-refs` | `true`, `false` (default) | Let `$ref` resolution leave the source document, reading files and fetching URLs. |
+| `overlay` | a file path | Apply an [OpenAPI Overlay](https://spec.openapis.org/overlay/latest.html) document to the source before lowering. |
+| `overlay-lax` | `true`, `false` (default) | Do not refuse when an overlay action's selector matches nothing. |
 
-| Flag | Meaning |
-|---|---|
-| `-o <file>` | Write IR JSON to `<file>` instead of stdout. |
-| `--fail-on error\|warning` | Exit non-zero when a diagnostic at or above this severity is emitted (default `error`). |
-| `--skip-validate` | Skip the referential-integrity `validate` pass. |
-| `--explain <json-pointer>` | Report what compiling produced at this source coordinate instead of writing the document. |
+```bash
+morphic compile openapi.yaml --opt grouping=path-prefix --opt overlay=patch.yaml
+```
 
-Diagnostics print one per line as `<severity> <code> <path>#<pointer>: <message>`. Exit codes:
-`0` clean (and for any help request), `1` a diagnostic reached the `--fail-on` threshold (or the
-spec could not be lowered), `2` a usage or I/O error.
+`1` and `2` can both be earned by one run — a spec that reached the threshold whose `-o`
+destination then refused the write. The verdict on the spec wins, so `1` means what it says
+whatever `-o` pointed at, and `2` means the run failed for a reason outside the spec. The write
+error is printed on stderr either way. Note that `-o` publishes by rename, so a destination whose
+directory will not take a temp file — `/dev/null`, a read-only directory — cannot be written to at
+all.
 
 ### Library
 
 The same pipeline is available as a package. `engine.New` builds the default registry (OpenAPI
-compiler + `validate` pass); `Run` sniffs the format, compiles, and runs passes.
+compiler + `validate` pass); `Run` asks the registered compilers which of them recognizes the
+source, compiles, and runs passes. A Go caller can set compiler options as a typed value through
+`RunOptions.FormatOptions` instead of as text through `RunOptions.CompilerOptions`.
 
 ```go
 eng, err := engine.New()

cmd/morphic/args.go

@@ -0,0 +1,80 @@
+package main
+
+import (
+	"flag"
+	"strings"
+)
+
+// flagTerminator is the end-of-flags marker every POSIX utility accepts: the
+// arguments after it are operands however they are spelled, which is the only
+// way to name a file that begins with "-".
+const flagTerminator = "--"
+
+// boolFlag is the flag package's own test for a flag set by its own presence
+// rather than by the argument after it. The interface is unexported there, so
+// it is restated rather than reached for; every flag registered by BoolVar
+// satisfies it.
+type boolFlag interface{ IsBoolFlag() bool }
+
+// cutTerminator returns args with a leading flagTerminator removed, and reports
+// whether one was there. It is the whole of terminator handling for an argument
+// list that defines no flags: there is nothing to stop parsing, so the marker's
+// only job is to say that what follows is not a flag.
+func cutTerminator(args []string) ([]string, bool) {
+	if len(args) > 0 && args[0] == flagTerminator {
+		return args[1:], true
+	}
+	return args, false
+}
+
+// splitAtTerminator splits args at the first flagTerminator standing as an
+// argument of its own, returning what precedes it and the operands that follow.
+// A "--" some flag asked for is that flag's value and not a marker, so the scan
+// steps over it — which is the whole reason it needs fs rather than a plain
+// search for the token.
+func splitAtTerminator(fs *flag.FlagSet, args []string) (before, operands []string) {
+	for i := 0; i < len(args); i++ {
+		if args[i] == flagTerminator {
+			return args[:i], args[i+1:]
+		}
+		if takesNextValue(fs, args[i]) {
+			i++
+		}
+	}
+	return args, nil
+}
+
+// takesNextValue reports whether arg is a flag fs defines that reads its value
+// from the following argument: spelled without an inline "=value" and not
+// boolean. An argument fs does not define is left alone, since Parse will
+// reject it and the split cannot change that.
+func takesNextValue(fs *flag.FlagSet, arg string) bool {
+	name, ok := flagName(arg)
+	if !ok {
+		return false
+	}
+	f := fs.Lookup(name)
+	if f == nil {
+		return false
+	}
+	b, isBool := f.Value.(boolFlag)
+	return !isBool || !b.IsBoolFlag()
+}
+
+// flagName returns the flag name arg spells, and whether arg is a flag whose
+// value would come from the next argument at all. It mirrors the flag package's
+// own syntax: one or two leading dashes, a name that starts with neither "-"
+// nor "=", and no inline "=value".
+func flagName(arg string) (string, bool) {
+	if len(arg) < 2 || arg[0] != '-' {
+		return "", false
+	}
+	name := strings.TrimPrefix(arg[1:], "-")
+	if name == "" || name[0] == '-' || name[0] == '=' {
+		return "", false
+	}
+	if strings.Contains(name, "=") {
+		return "", false
+	}
+	return name, true
+}