feat(eval): Phase 4 — CI gate, triage, K∧E∧T coverage, an L3 build oracle, and the three defects it found

[dynamics365ninja]

Phase 4 of docs/KNOWLEDGE_AUDIT_PLAN.md — the eval loop to predecessor parity (audit finding R7).

What landed

4.1 — CI gate. An eval job replays the whole catalog (eval run --all, 51/51 green) and runs eval coverage --check on every PR.

4.2 — L3 build oracle. d365fo eval verify-build provisions every reviewed golden into a throwaway model, compiles it with xppc.exe, attributes each diagnostic to the case whose golden produced the named object, and persists eval/golden-build-verification.json. EvalScoreCard gained BuildClean/BuildErrors, null wherever no compiler ran.

Four assumptions were wrong until it ran against a real installation:

• a descriptor good enough for this repo's extractor kills the compiler before it compiles anything (missing <ModelModule>, Layer is an ordinal, string collections live in the DataContract arrays namespace);
• referencing ApplicationSuite alone leaves the standard Name EDT and ModuleAxapta unresolvable;
• provisioning the fixture as its own module does not make its tables visible — five goldens were reported as referencing a nonexistent FmVehicle;
• XppcDiagnostics knew five severity prefixes and the compiler emits three more, so a run logging Errors: 8 parsed as zero diagnostics and the harness called every golden clean.

48 of 51 goldens compile clean. The three that don't are real shipping defects no offline gate could see, now ranked as TOOL_DEFECT clusters with corpus provenance:

Case(s)	Defect
L1-query-basic, L1-query-join	every generated AxQuery crashes the metadata reader — <SourceCode><Methods><Method><Name>classDeclaration is mandatory, and each data source needs <DynamicFields>Yes</DynamicFields> in contract position
L2-event-handler-basic	the subscriber method is written inside <Declaration>; the provider needs it as an XML <Method> with a <Name>

Fixing them is scaffolder work and belongs to the improver's next PR — which is the loop behaving as designed. Same reason the mini-AOT fixture gained an EDT and an enum but not a query: a generated one would make the fixture module itself uncompilable and block the oracle.

4.4 — Triage, clusters, knowledge feedback. EvalTriage derives a hypothesis for replay runs (no model in the loop, so MODEL_ERROR is impossible by construction) and deliberately refuses to for agent runs, where the same scorecard is equally consistent with a tool defect and the agent's own mistake. eval clusters gained --actionable/--top, carries run ids and failing dimensions, and exempts known-reference-gap cases. d365fo eval knowledge turns KNOWLEDGE_GAP/MODEL_ERROR runs into skills/_source proposals naming the literal that links each topic to the case (finding K4).

4.5 — K ∧ E ∧ T coverage. eval/COVERAGE.md is derived from ObjectTypeRegistry, a new GenerateSurface, the case catalog and the embedded corpus: 42 of 70 leaves complete, with the real gaps listed (no topic names AxMap, AxView or the extension families). GenerateSurface is gated in both directions against the CLI's own help output so it cannot become the fifth drifting registry.

What did not land

4.3 — the L4 runtime oracle. It needs a SysTestRunner result parser (test run returns a raw output tail) and per-run fixture provisioning inside a live model. One prerequisite defect was fixed on the way: test run --suite X passed "--suite X" as a single argv element, so the flag could never have worked.

Verification

• dotnet test — 1004 passed, 0 failed
• d365fo eval run --all — 51/51
• d365fo eval coverage --check, d365fo knowledge audit --verify, scripts/emit-skills.py drift — all clean
• d365fo eval verify-build on a D365FO installation — 48 clean / 3 errors / 0 skipped

🤖 Generated with Claude Code

[dynamics365ninja]

Follow-up commit: the scaffolder defects the oracle found are now fixed in this PR rather than deferred.

Scaffolder	Defect	Ground truth
generate query	metadata reader died with a bare KeyNotFoundException before compiling anything	300/300 shipped ApplicationSuite queries carry a classDeclaration method — it is mandatory. Data sources also need <DynamicFields>Yes</DynamicFields> (contract position: after ConcurrencyModel), since a scaffolded data source selects no fields
generate event-handler	handler method written inline in <Declaration>, so the provider found a method with no name in the XML	now an XML <Method> with its own <Name> — what CocExtension and every other class scaffolder here already emitted
generate migration-script	loop counter named count, an X++ keyword: declaration rejected, rest of the method mis-parsed	surfaced only after the first two were fixed — the metadata crashes had been aborting the compile before IL generation

Also in this commit:

• known-reference-gap cases are exempt from build classification and clustering. Their X++ names a sibling artifact the case's single golden doesn't include, or a standard object the fixture can't contain; the compiler rejects those names for exactly the reason validate references does. Filing them as TOOL_DEFECT would park three permanent false clusters at the top of the improver's queue.
• Fixture: the query goes back in now that generate query produces a readable one, plus an OnInitialized delegate on FmVehicleService — delegateStr() is compile-time checked, so without it the event-handler case couldn't compile however correct the scaffolder was.

Baseline is unchanged at 48/51 clean, but it means something different now: the 3 failures are the documented known-reference-gap cases, not defects.

dotnet test 1007 passed · eval run --all 51/51 · eval coverage --check, knowledge audit --verify, skills drift all clean.

[dynamics365ninja]

Fourth defect fixed — and it exposed that the oracle itself was under-reporting.

generate business-event

The [BusinessEvents] attribute was wrong in three ways at once. Shape read off shipped events (BenefitAnnualSalaryChangeBusinessEvent, BackgroundOperationCancelledBusinessEvent):

[BusinessEvents(classStr(<Contract>), '<name>', '<description>', ModuleAxapta::<module>)]

vs. what we emitted — the event class first, a second classStr where the name belongs, and a plain string where the enum belongs (Cannot implicitly convert from type 'str' to type 'Extensible Enumeration(ModuleAxapta)'). --category now means that enum, validated against its 40 real values, defaulting to NotApplicable.

The generated factory also called parmId(), which exists on no business event — BusinessEventsBase declares only parmUserId. It now keeps the source record the way shipped events do: a private member behind a generated parm method.

The oracle was hiding findings

Two bugs, both in code this PR added:

• The metadata validator reports in its own shape — Metadata Error: AxForm/<object>/Design/Controls/…/DataGroup: … — no dynamics:// URL, no source location, and for a form the member path is the whole control tree. XppcDiagnostics didn't recognise it, so those lines fell through to the message-only fallback with no object to blame.
• The attribution key was the golden's file stem, which truncates NoYes.Extension at the dot.

Baseline is now 36/51 clean with zero unattributed diagnostics (was a flattering 48/3). The number went down because the oracle got more honest, not because the tool got worse — the same failure mode as a validator that passes because it never ran.

What that leaves

12 actionable TOOL_DEFECT clusters — the real generation-depth queue, all Phase 2-shaped rather than eval-loop work:

• reports (L1-report-basic, L1-report-parameters) — missing every mandatory AX_* framework parameter, no page size, a dataset with no fields
• data entity (L2-virtual-entity-basic) — public entity with no key and no primary key
• forms (3 pattern cases + L2-datasource-method-basic) — DataGroup bound to field groups the table doesn't declare; one nests an AxFormActionPaneTabControl under a tab page, which the validator forbids
• L1-workflow-basic, L2-security-policy-basic, L1-custom-service-basic, L2-security-duty-extension — empty mandatory properties / a service class that doesn't exist
• L2-enum-extension — extends NoYes, which isn't extensible on this platform; that's a case-authoring bug, not a scaffolder one

dotnet test 1014 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

Reports and the data entity fixed — both were producing invalid artifacts on every invocation.

generate report

Three problems, all grounded against a shipped auto-design report (QMSDispBatchProdReport):

Problem	What the validator said
DefaultParameterGroup emitted only when --parameter was passed	"Report default parameter group is not found", plus one error per missing framework parameter — AX_PartitionKey, AX_CompanyName, AX_UserContext, AX_RenderingCulture, AX_ReportContext
auto design had no PageSize / InteractiveSize / Margin	"Invalid page size. The page height and the page width cannot be less than or equal to 0."
--extra-dataset Name:DPClass produced a dataset with no fields	"The dataset X has no fields specified" + "Table2: Missing Data fields" + an empty temp table

The parameter group is now always emitted (framework parameters first, caller's appended), the design carries portrait-Letter geometry in contract order after DataRegions, and extra datasets take fields inline (Name:DPClass:F1,F2) or inherit --field — so the dataset, its temp table, the DP getter and the design region all describe the same columns.

generate entity

IsPublic was hard-coded Yes while Keys stayed optional and nothing ever passed keys, so every generated entity failed "There must be a key defined for a public data entity" and both PrimaryKey checks.

The command now derives the EntityKey from the table's alternate-key index — that's what a data entity key mirrors — or takes a new --key, or falls back to explicitly mandatory fields, and asks instead of guessing when none of those work. The scaffolder keeps IsPublic and Keys in step so they can't disagree again.

Baseline

39 of 51 clean (was 36), zero unattributed diagnostics. Six defects found and fixed by this oracle so far: generate query, generate event-handler, generate migration-script, generate business-event, generate report, generate entity.

The 9 actionable clusters left:

• forms (3 pattern cases + L2-datasource-method-basic) — DataGroup bound to field groups the table doesn't declare; one nests an AxFormActionPaneTabControl under a tab page
• L1-workflow-basic, L2-security-policy-basic, L2-security-duty-extension — empty mandatory properties
• L1-custom-service-basic — names a service class the command doesn't generate
• L2-enum-extension — extends NoYes, not extensible on this platform; a case-authoring bug

dotnet test 1016 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

Forms fixed — and the parser was still hiding a whole class of findings.

What was actually wrong

The R5 trap, confirmed. The audit listed "the <DataGroup> ⇒ sibling <DataSource> trap" as absorbable predecessor knowledge. It was real, and it was in our templates: DataGroup names a field group on a table, so without the sibling DataSource saying which table, the reader reports "Field group 'Overview' does not exist" even when the table defines it. Both were also emitted before <Controls> instead of after, where the contract declares them and every shipped form puts them.

AxFormActionPaneTabControl directly under a tab page — "not allowed for parent control of type 'AxFormTabPageControl'". Shipped ToolbarList tab pages nest TabPage > ActionPane > ActionPaneTab > ButtonGroup; the template skipped the ActionPane.

Fixture. The mini-AOT tables now carry the field groups generate table produces. Overview/General aren't part of the AOT default scaffold — they exist because every form pattern this CLI generates binds to them, a contract generate form's preflight check already warns about. The fixture was hand-written without them, so those goldens could never have loaded.

The parser was still under-reporting

FormPatternValidation Error, FormPatternValidation Fatal Error, Metadata Warning, ExternalReference Warning and Generation Information weren't in the severity list — 24 diagnostics were vanishing. They're counted now.

Baseline went 43 → 35 of 51 clean on that change alone. To repeat the rule from the last comment: the count drops whenever the parser learns a prefix it was discarding, and rises when a scaffolder is fixed. The signal to watch is the unattributed count (zero), not the clean count.

What that exposed: AOT form-pattern compliance

This is now the single biggest slice of the queue, and it's a project rather than a fix. Seven form cases fail FormPatternValidation — a different validator from this repo's own FP001–FP010, run by the AOS against its pattern registry:

• five design <Pattern> values name patterns the registry doesn't have — DetailsMaster 1.1, ListPage 1.1, Lookup 1.2, DetailsTransaction 1.1, Workspace 1.0, plus SidePanel 1.0 on a container. (Control-level patterns like FieldsFieldGroups 1.1 and ToolbarList 1.2 do resolve — so the vocabulary is partly right and partly invented.)
• missing required children — TOC Tabs for Table of Contents, Details Tabs for Simple List and Details
• required property values — ColumnsMode=Fill, WidthMode/HeightMode=SizeToAvailable, ArrangeMethod=HorizontalRight, ViewEditMode=Edit

Reconciling FormPatternCatalog with the registry the AOS validates against — and teaching FP001–FP010 the same rules, so the offline gate catches them instead of the VM — is its own piece of work. I've stopped here rather than guessing at pattern names.

dotnet test 1018 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

Form patterns: one fixed, five turned from a VM-only surprise into an offline test failure with the real answer attached — and I stopped short of the rest deliberately.

Fixed: Dialog

The AOS requires ColumnsMode=Fill on the FieldsFieldGroups body, WidthMode/HeightMode=SizeToAvailable on body and button group, and ArrangeMethod=HorizontalRight on the button group, per pattern Dialog - Basic. L1-form-dialog compiles clean.

The registry is no longer a mystery

The AOT pattern registry ships as 102 XML definitions embedded in Microsoft.Dynamics.AX.Metadata.Patterns.dll — name, version, alias, required parts with cardinality, and the property values each part must carry. scripts/emit-form-patterns.ps1 extracts them into src/D365FO.Core/FormPatterns/form-patterns.json, exactly the way emit-metadata-contracts.ps1 derives the DataContract catalog. FormPatternRegistry serves it with no installation present.

FormTemplatePatternRegistryTests now pins every template's design pattern against it. Five name a pattern that exists on no AOS, listed with the real one:

Template says	Registry has
DetailsMaster 1.1	DetailsMaster 1.4
DetailsTransaction 1.1	DetailsTransaction 1.4
ListPage 1.1	ListPage UX7 1.0
Lookup 1.2	LookupGridOnly 1.1 / LookupTab 1.0 — there is no plain Lookup
Workspace 1.0	WorkspaceOperational 1.1 / TabbedWorkspace 1.0

Shrinking that list is the work queue; adding to it is a regression.

Why I didn't just fix them

FormPatternCatalog — this repo's own model, which FP001–FP010 enforce — disagrees with the registry. A template rewritten to satisfy the AOS is rejected by our own FP003 before it can be written.

I verified that by doing it: I restructured DetailsMaster to the shipped 1.4 shape (SidePanel navigation list, Details page with a nested FastTabs tab, Overview page with the main grid, from QMSCAPACaseDetail). The AOS accepted the structure and FP003 blocked generation on the quick-filter sub-pattern. That restructure is reverted rather than left half-applied — a repo where the templates satisfy the AOS and the offline validator rejects them is worse than either state.

The catalog, the templates, the validator, the nine form goldens and ~90 form-pattern tests have to move together. That's a coherent next PR, and it's now mechanical rather than speculative because the required parts and properties are derived data.

Baseline: 36 of 51 clean (was 35), zero unattributed diagnostics.

dotnet test 1029 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

FormPatternCatalog migration started — the structural half of a pattern now comes from the AOT registry, and SimpleList is migrated end to end.

The mechanism

FormPatternCatalog derives versions, design properties, the required control tree and the design-root policy from form-patterns.json for every pattern on a RegistryDerived list. The editorial half stays hand-written, because the registry doesn't know it: purpose, when-to-use, reference forms, lifecycle guidance, sub-pattern hints.

SimpleList is on the list: derived spec, template carrying the layout properties the AOS requires (design ArrangeMethod/Columns/ColumnsMode, the action pane's and grid's sizing and Style, all in contract order), goldens re-captured, and its form plus the two form-method cases that build on it compile clean. Without those properties the repairer rewrote every generated form — "conforming" was quietly not meaning that.

What doing it taught

Two corrections, both now encoded in RegistrySpecFactory:

• A part that declares no children delegates its contents to the sub-pattern, it doesn't forbid everything. Reading the absence strictly rejected a quick filter inside a CustomFilterGroup — the one thing that belongs there.
• Versions come in two lineages. Plain numbers, and an older series whose version string is literally UX7 1.0 — which sorts above 1.4. Ranking the lineages is what keeps DetailsMaster's newest at 1.4 rather than UX7 1.2.

It also corrected my previous comment: ListPage is a real pattern name whose only version is the string UX7 1.0. That template needs a version change, not a rename. The KnownWrong notes now record what the registry has rather than what I first assumed.

And two places where the AOS is simply stricter than we were, so the fixtures moved with the spec: the custom-filter group is a required part of SimpleList, and a second root Group is legitimate (the registry's optional Footer slot).

What's left

Eight patterns, one at a time — derive the spec, restructure the template to the required parts, re-capture goldens, fix the fixtures that encoded the old model. Each migration changes what FP003/FP004 accept, which is exactly why the spec and the template have to move together rather than in separate passes.

Baseline: 36 of 51 clean, zero unattributed diagnostics.

dotnet test 1031 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

(Note: while running the suite repeatedly I saw three different tests flake once each — ScaffoldFileWriterJournalTests, McpServerHostTests, McpDispatcherTests, all temp-file/label related and unrelated to form patterns. Two consecutive clean runs after. Worth a look separately.)

[dynamics365ninja]

Three patterns migrated now — SimpleList, TableOfContents, ListPage. Baseline 38 of 51 clean.

TableOfContents

Structure, not naming — and wrong in five ways the AOS named precisely: the design has exactly one part (the vertical tab) and allows nothing beside it, so the ActionPane had to go; every page needs its title group with a MainInstruction static text; the fields belong in a sibling content group; that group is required even when empty; and the page must not declare FieldsFieldGroups — the TOC pattern already governs it, and that sub-pattern forbids the very title the TOC pattern requires.

That also produced the more important correction of the two commits: a part's declared children are the ones that must be present, not the only ones that may be. TableOfContents never writes Children="*" anywhere, yet the AOS accepts field controls on a TOC page while rejecting an ActionPane at the design root. So the closed set is the design root; inside a part the spec is permissive.

ListPage

A version problem, not a naming one — the registry has exactly one version and it is the string UX7 1.0. Plus the design's layout properties and the grid's AllowEdit=No.

The extraction was dropping parts

Two element kinds the script ignored:

• <OneOf> — a choice slot. LookupGridOnly's entire list control (Grid | Tree | ListView) lives inside one, so the derived spec had no grid at all. NodeSpec already models "one of these types", so a choice maps onto one slot with the alternatives as ControlTypes and no property requirements — demanding the Grid's AllowEdit of a Tree would be nonsense.
• <SubPattern Name="X" /> — now feeds RequiresSubPattern/AllowedSubPatterns.

Wiring sub-patterns through surfaced the next layer of the same drift: four of our sub-pattern names are wrong.

Ours	Registry
ToolbarAndList	ToolbarList
ToolbarAndFields	ToolbarFields
ToolbarAndListDouble	ToolbarListDouble
HorizontalFieldsButtonGroup	HorizontalFieldsButtonsGroup

The shipped-form census backs the registry — real forms declare ToolbarList 1.2. Two more (Custom, TabPageTabularFields) the catalog never covered. RegistrySpecFactory drops names the catalog can't resolve rather than letting FP007 pretend to check them, and the list is pinned in a test so it stays visible.

Left

Five patterns: DetailsMaster (→ 1.4), DetailsTransaction (→ 1.4), Lookup (→ LookupGridOnly 1.1), Workspace (→ WorkspaceOperational 1.1), SimpleListDetails. The first four are on the KnownWrong list with the real pattern named; each is a template restructure against derived data, verified per-pattern with eval verify-build --case.

dotnet test 1032 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

DetailsMaster and DetailsTransaction migrated — 40 of 51 clean, five patterns done, three to go.

Both declared version 1.1, which exists on no installation, and both had a shape the real pattern rejects. Rebuilt from the registry's 1.4 definitions:

Design
  ApplicationBar (Style=Standard)
  NavigationList (SidePanel group: quick filter + read-only list grid)
  PanelTab (ShowTabs=No)
    DetailsPanel  — title group with the required HeaderTitle, then
                    DetailsMaster:      a FastTabs tab
                    DetailsTransaction: HeaderLinePanels > Lines/Header panels,
                                        the Lines panel carrying its own FastTab
                                        with a toolbar and the lines grid
    GridPanel     — custom-and-quick-filter group, main grid, default action

Four things the compiler had to tell us

• the title group requires a HeaderTitle control (Skip=Yes, ShowLabel=No, WidthMode=SizeToAvailable) — bound to the first requested field, the best guess a scaffold can make;
• MainGrid's DefaultAction cannot be empty — it points at MainGridDefaultAction;
• a form may not repeat a control name, and both patterns now have two quick filters ("Duplicate value 'QuickFilterControl' detected"). The outer name must be unique; the extension's name is what identifies the control type, so that one stays QuickFilterControl — our own FP003 caught the first attempt at renaming both, which is the offline gate doing its job;
• DetailsTransaction's Lines and Header panels each require their own tab (LineViewTab, HeaderDetailsTab), which is where the header fields now live.

Three harness fixes that came with them

• Provisioned goldens are copied with the read-only attribute cleared. The compiler rewrites a form's metadata header while validating it, and otherwise reports "Failed to write metadata back to the file … often due to the file being read-only" — a property of the copy, scored against the case.
• Generation Error joins the severity list. It was the prefix on exactly that message, so the harness wasn't counting it. (Same class of gap as the earlier two — worth noting the pattern: every new prefix found so far was found by reading a log, not by the harness noticing.)
• NodeSpec.NameHint is never null now — the repairer names controls it adds from the hint, and registry parts without a Part attribute made it throw.

Left

SimpleListDetails, Lookup (→ LookupGridOnly 1.1), Workspace (→ WorkspaceOperational 1.1). The last two are the renames; SimpleListDetails already names a real pattern and needs structure only.

dotnet test 1032 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.

[dynamics365ninja]

AOT form-pattern compliance is closed. All nine patterns migrated, every form eval case compiles clean, KnownWrong is empty. Baseline 43 of 51.

The last three

Lookup → LookupGridOnly 1.1. There is no pattern named Lookup at all. Its filter group takes CustomFilters, not CustomAndQuickFilters, and its grid is never editable.

Workspace → WorkspaceOperational 1.1. Workspace exists only as an inactive 2.0. The live pattern's sections are FastTab pages — summary tiles, tabbed list, related links — not a Panorama.

The finding that mattered: an operational workspace's lists are not inline. A tabbed-list page must hold a FormPartControl pointing at a separate form whose own design pattern is FormPartSectionList. Putting fields there yields a form the AOS rejects; pointing a FormPart at a menu item that doesn't exist is the dangling reference this repo refuses to emit. So generate form --pattern Workspace now declines --field/--section and says what to do instead, rather than dropping them silently or writing something unloadable. The eval case moved with it.

SimpleListDetails. SidePanel was declared as a sub-pattern; it's a Style, and the AOS went looking for a pattern it has no record of.

The extraction was still incomplete

Sub-patterns live in their own resource namespace and weren't being read at all — the registry knew nothing about their versions. It does now: 162 definitions, up from 102, which is how the section sub-patterns' 1.0 versions turned out to be inactive with 1.1 current.

That exposed the last of the name drift. Six sub-patterns the catalog had invented:

Ours	Registry
Workspace_Tiles	SectionTiles
Workspace_Links	SectionRelatedLinks
ToolbarAndList	ToolbarList
ToolbarAndFields	ToolbarFields
ToolbarAndListDouble	ToolbarListDouble
HorizontalFieldsButtonGroup	HorizontalFieldsButtonsGroup

The catalog's own note said these were "to be confirmed by mining". The registry confirmed the alias had been the real name all along — so XmlName and alias are swapped.

Where the queue stands

43 of 51 clean, and not one form case among the remaining 8: a workflow and two security objects with an empty mandatory property, a custom service naming a class the command doesn't generate, L2-enum-extension extending the non-extensible NoYes (a case-authoring bug), and the three known-reference-gap cases that are recorded but never classified.

dotnet test 1030 passed · eval run --all 51/51 · coverage / knowledge-audit / skills-drift clean.