Finish #164 (R5 form-engine extras), and what #160 still needs

docs/CAPABILITIES.md

@@ -24,9 +24,34 @@ By default the index stores object/method *metadata* only — method bodies (the
 | `index history` | Show per-model extraction run history |
 | `index export` | Dump the index to a portable archive |
 | `index import` | Restore from an archive |
+| `index cross-check` | Report where this tool's catalogs are narrower than the installation |
 | `index optimize` | Run `VACUUM` + `ANALYZE` to compact and re-plan |
 | `doctor` | End-to-end health check: paths, schema version, object counts |
 
+### `index cross-check`
+
+Every catalog this tool answers from — the form-pattern registry, the object-type registry, the
+DataContract catalog — is generated from one platform version and then committed. That makes it
+right when it was made and silent about drift afterwards. This asks the installation instead.
+
+```sh
+d365fo index cross-check                       # gaps only
+d365fo index cross-check --show-uncovered      # plus families the tool does not cover
+```
+
+Two findings, deliberately kept apart:
+
+- **Gaps** are where the tool will be *wrong* — something the installation uses that a catalog
+  claims to cover and does not. A form pattern in the wild that the registry has never heard of
+  means `generate form`, `form-pattern validate` and `form-pattern repair` cannot judge those
+  forms, and will say so in the confident voice of a tool that has a catalog. Exit code 2.
+- **Uncovered** families are where the tool is merely *narrow* — an AOT folder it was never built
+  to handle. On a real installation that is dozens of entries (40 of 83 on the box this was
+  written against), so it is off by default and never fails the command.
+
+The fix for a gap is to regenerate the named catalog on the installation that produced it
+(`scripts/emit-form-patterns.ps1`, `scripts/emit-metadata-contracts.ps1`), not to hand-add an entry.
+
 ---
 
 ## Search & Discovery
@@ -203,6 +228,7 @@ All scaffolders write atomically (`.tmp` + move, `.bak` on overwrite). Pass `--i
 | `generate security-policy` | `AxSecurityPolicy` (XDS row-level security), including the nested `--constrained` table tree |
 | `generate systest` | `SysTestCase` skeleton — `[SysTestMethod]` Arrange/Act/Assert stub, optional `[SysTestCaseDataDependency]` and `--atl` `AtlDataRootNode` wiring (ATL-ready MVP, no test-logic generation) |
 | `generate migration-script` | Data-fix `Runnable` class with `ttsbegin`/`ttscommit` batching |
+| `generate form-clone` | Copy of an existing `AxForm` under a new name, datasources optionally re-bound |
 | `generate simple-list` | Alias for `generate form --pattern SimpleList` |
 | `modify method` | Replace an existing method's body on a live class/table/edt/form via D365FO.Bridge (`IMetadataProvider`, structured `XDocument` replace — no CDATA string surgery, no on-disk fallback). Reference/BP validation always blocks on error-severity findings. |
 | `modify property` | Set a property (`Label`, `ConfigurationKey`, `TableGroup`, …) on a live object. |
@@ -225,6 +251,32 @@ same object already uses, so a model does not accumulate `CustTable.Fleet` next
 Every `modify` write (including `modify method`) records its exact pre-image in the
 modification journal — revert with `d365fo undo`.
 
+#### Cloning a reference form
+
+A Microsoft form that already has the pattern, the control tree and the wiring right is a better
+starting point than any template, and cloning one is what a developer does by hand anyway.
+
+```sh
+d365fo generate form-clone ConVehicleGroup --from CustGroup     --rebind CustGroup=ConVehicleGroupTable --out ConVehicleGroup.xml
+```
+
+`--from` takes a form name (resolved through the index) or a path to the AxForm XML. `--rebind`
+moves a datasource onto another table, renames the datasource when it was named after the old
+table, and follows that rename into every control that referenced it — including the datasource
+entry under `<SourceCode>`, where override methods live.
+
+The edits are string-level and narrow. An `AxForm` is a V6 contract whose Design subtree is
+written in the empty namespace with `i:type` on every control, so loading it into an `XDocument`
+and writing it back rewrites namespace declarations nobody asked to change — which is also why
+`FormPatternTemplates` renders forms as strings. Verified against a shipped 16 KB `CustGroup`
+form: the clone differs on exactly the intended lines and is byte-identical everywhere else.
+
+What it deliberately does *not* do is a blind replace of the old form name. Form names are short
+and appear inside unrelated identifiers (`CustGroup` inside `Grid_CustGroupId`), so only the root
+`<Name>`, the class declaration and `formStr()` self-references move. Everything it cannot reach —
+menu items, privileges, extensions, callers elsewhere in the AOT — comes back as a warning, as
+does the fact that a rebind does not check the new table actually has the bound fields.
+
 #### The grounding gate
 
 Every `generate` subcommand runs the same gate before it writes anything — not just the

src/D365FO.Cli/CliApp.cs

@@ -216,6 +216,7 @@ public static CommandApp Build(Spectre.Console.IAnsiConsole? console = null)
                 b.AddCommand<IndexExtractCommand>("extract").WithDescription("Walk PACKAGES_PATH and ingest AOT metadata.");
                 b.AddCommand<IndexRefreshCommand>("refresh").WithDescription("Incremental extract — skip models whose XMLs haven't changed since last extract.");
                 b.AddCommand<IndexHistoryCommand>("history").WithDescription("Show recent ExtractionRuns (per-model timings persisted across runs).");
+                b.AddCommand<IndexCrossCheckCommand>("cross-check").WithDescription("Report where this tool's catalogs are narrower than the installation.");
                 b.AddCommand<IndexOptimizeCommand>("optimize").WithDescription("VACUUM + ANALYZE the index (reclaim space, refresh query-planner stats).");
                 b.AddCommand<IndexExportCommand>("export").WithDescription("Export index as a GZip-compressed snapshot for sharing or CI caching.");
                 b.AddCommand<IndexImportCommand>("import").WithDescription("Import a GZip-compressed index snapshot.");
@@ -238,6 +239,7 @@ public static CommandApp Build(Spectre.Console.IAnsiConsole? console = null)
                 b.AddCommand<GenerateFormCommand>("form").WithDescription("Create an AxForm with a chosen pattern (SimpleList, DetailsMaster, DetailsTransaction, Dialog, Lookup, ListPage, Workspace, …).");
                 b.AddCommand<GenerateDataSourceMethodCommand>("datasource-method").WithDescription("Add/override a method on a form datasource (form-level SourceCode). Omit --method to list overridable methods.");
                 b.AddCommand<GenerateControlMethodCommand>("control-method").WithDescription("Add/override a method on a form control (form-level SourceCode). Omit --method to list overridable methods.");
+                b.AddCommand<GenerateFormCloneCommand>("form-clone").WithDescription("Clone an existing AxForm under a new name, optionally re-binding its datasources.");
                 b.AddCommand<GenerateSimpleListCommand>("simple-list").WithDescription("(Deprecated) Alias for `generate form --pattern SimpleList`.");
                 b.AddCommand<GenerateEntityCommand>("entity").WithDescription("Create an AxDataEntityView over a table.");
                 b.AddCommand<GenerateExtensionCommand>("extension").WithDescription("Create a Table/Form/Edt/Enum extension.");

src/D365FO.Cli/Commands/Generate/GenerateFormCloneCommand.cs

@@ -0,0 +1,161 @@
+using D365FO.Core;
+using D365FO.Core.FormPatterns;
+using D365FO.Core.Scaffolding;
+using Spectre.Console.Cli;
+
+using static D365FO.Core.ObjectTypes.ObjectTypeRegistry;
+
+namespace D365FO.Cli.Commands.Generate;
+
+/// <summary>
+/// Clone an existing form under a new name, optionally re-binding its datasources.
+/// </summary>
+/// <remarks>
+/// Issue #164 / R5. A Microsoft form that already has the pattern, the control tree and the
+/// wiring right is a better starting point than any template, and cloning one is what a developer
+/// does by hand anyway. The edits are string-level and narrow — see <see cref="FormCloner"/> for
+/// why a round-trip through <c>XDocument</c> would return a form that differs from the original
+/// in ways nobody asked for.
+/// </remarks>
+public sealed class GenerateFormCloneCommand : Command<GenerateFormCloneCommand.Settings>
+{
+    public sealed class Settings : GenerateSettings
+    {
+        [CommandArgument(0, "<NAME>")]
+        [System.ComponentModel.Description("Name for the clone.")]
+        public string Name { get; init; } = "";
+
+        [CommandOption("--from <FORM>")]
+        [System.ComponentModel.Description("Reference form: a form name resolved through the index, or a path to its AxForm XML.")]
+        public string? From { get; init; }
+
+        [CommandOption("--rebind <SPEC>")]
+        [System.ComponentModel.Description("Repeatable: <OldTable>=<NewTable>. Moves the datasource, its name when it matched the table, and every control that references it.")]
+        public string[] Rebind { get; init; } = Array.Empty<string>();
+    }
+
+    public override int Execute(CommandContext ctx, Settings settings)
+    {
+        var kind = OutputMode.Resolve(settings.Output);
+
+        if (string.IsNullOrWhiteSpace(settings.Name))
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.BadInput, "Clone name required."));
+        if (string.IsNullOrWhiteSpace(settings.From))
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.BadInput, "--from <FORM> required."));
+
+        var (sourceXml, readError) = ReadSourceForm(settings.From!);
+        if (readError is not null)
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.SourceUnreadable, readError));
+
+        if (!TryParseRebinds(settings.Rebind, out var rebinds, out var rebindError))
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.BadInput, rebindError!));
+
+        var hasInstall = !string.IsNullOrWhiteSpace(settings.InstallTo);
+        var hasOut = !string.IsNullOrWhiteSpace(settings.Out);
+        if (!hasInstall && !hasOut)
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.BadInput, "--out or --install-to is required."));
+
+        var outPath = settings.Out;
+        if (hasInstall && !hasOut)
+        {
+            outPath = GenerateInstaller.ResolveInstallPath(kind, Folders.Form, settings.Name, settings.InstallTo!, out var fail);
+            if (fail.HasValue) return fail.Value;
+        }
+
+        FormCloneResult clone;
+        try { clone = FormCloner.Clone(sourceXml!, settings.Name, rebinds); }
+        catch (FormCloneException ex)
+        {
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail("CLONE_FAILED", ex.Message));
+        }
+
+        // The clone claims the tables it was rebound onto exist — that is the one thing here the
+        // index can prove, and the reason this goes through the gate like every other generate.
+        var gate = GenerateInstaller.Gate(
+            settings, settings.Name, doc: null,
+            requiredSymbols: rebinds.Values);
+        if (gate.Failure is not null) return RenderHelpers.Render(kind, gate.Failure);
+
+        var warnings = gate.Warnings;
+        warnings.AddRange(clone.Warnings);
+
+        try
+        {
+            var res = GenerateInstaller.Write(gate, clone.Xml, outPath!, settings.Overwrite);
+            return RenderHelpers.Render(kind, ToolResult<object>.Success(new
+            {
+                kind = "AxForm",
+                role = "Clone",
+                name = settings.Name,
+                from = settings.From,
+                rebound = clone.Rebound,
+                renamedDataSources = clone.RenamedDataSources,
+                path = res.Path,
+                bytes = res.Bytes,
+                backup = res.BackupPath,
+                model = settings.InstallTo,
+                grounding = gate.Grounding,
+            }, warnings));
+        }
+        catch (Exception ex)
+        {
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail(D365FoErrorCodes.WriteFailed, ex.Message));
+        }
+    }
+
+    /// <summary>
+    /// Read the reference form, from a path or by name through the index.
+    /// </summary>
+    /// <remarks>
+    /// A path wins when it exists, so a developer can clone a form they have in front of them
+    /// without an index. The name route reads <c>SourcePath</c> off the index and then reads the
+    /// file — the index stores metadata, not the document.
+    /// </remarks>
+    private static (string? Xml, string? Error) ReadSourceForm(string from)
+    {
+        if (File.Exists(from))
+        {
+            try { return (File.ReadAllText(from), null); }
+            catch (Exception ex) { return (null, $"Could not read '{from}': {ex.Message}"); }
+        }
+
+        try
+        {
+            var details = RepoFactory.Create().GetForm(from);
+            if (details is null)
+                return (null, $"Form '{from}' is not in the index, and no file exists at that path. " +
+                              "Run `d365fo index extract`, or pass the path to the AxForm XML.");
+
+            var path = details.Form.SourcePath;
+            if (string.IsNullOrWhiteSpace(path) || !File.Exists(path))
+                return (null, $"The index knows form '{from}' but its source file is not at '{path}'. " +
+                              "The index is a cache and is never invalidated on delete — re-run `d365fo index refresh`.");
+
+            return (File.ReadAllText(path), null);
+        }
+        catch (Exception ex)
+        {
+            return (null, $"Could not resolve form '{from}': {ex.Message}");
+        }
+    }
+
+    private static bool TryParseRebinds(
+        string[] raw, out Dictionary<string, string> rebinds, out string? error)
+    {
+        rebinds = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
+        error = null;
+
+        foreach (var spec in raw.Where(r => !string.IsNullOrWhiteSpace(r)))
+        {
+            var parts = spec.Split('=', 2, StringSplitOptions.TrimEntries);
+            if (parts.Length != 2 || parts[0].Length == 0 || parts[1].Length == 0)
+            {
+                error = $"Invalid --rebind '{spec}'. Expected <OldTable>=<NewTable>.";
+                return false;
+            }
+            rebinds[parts[0]] = parts[1];
+        }
+
+        return true;
+    }
+}

src/D365FO.Cli/Commands/Index/IndexCrossCheckCommand.cs

@@ -0,0 +1,81 @@
+using D365FO.Core;
+using D365FO.Core.Analysis;
+using Spectre.Console;
+using Spectre.Console.Cli;
+
+namespace D365FO.Cli.Commands.Index;
+
+/// <summary>
+/// Reports where this tool's catalogs are narrower than the installation in front of you.
+/// </summary>
+/// <remarks>
+/// Issue #164 / R5. Every catalog the tool answers from — the form-pattern registry, the object-type
+/// registry, the DataContract catalog — is generated from one platform version and then committed.
+/// That makes it right when it was made and silent about drift afterwards. This command asks the
+/// installation instead, and reports what the catalogs do not cover.
+/// Exit codes: 0 = no gaps (or skipped), 1 = command failure, 2 = gaps found.
+/// </remarks>
+public sealed class IndexCrossCheckCommand : Command<IndexCrossCheckCommand.Settings>
+{
+    public sealed class Settings : D365OutputSettings
+    {
+        [CommandOption("--packages <PATH>")]
+        [System.ComponentModel.Description("Packages root to sweep for AOT folders. Defaults to D365FO_PACKAGES_PATH; the form-pattern half runs without it.")]
+        public string? PackagesPath { get; init; }
+
+        [CommandOption("--show-uncovered")]
+        [System.ComponentModel.Description("Also list AOT families this tool does not cover. Off by default — on a real installation it is dozens of entries and none of them is a defect.")]
+        public bool ShowUncovered { get; init; }
+    }
+
+    public override int Execute(CommandContext ctx, Settings settings)
+    {
+        var kind = OutputMode.Resolve(settings.Output);
+        var cfg = D365FoSettings.FromEnvironment();
+
+        CrossCheckReport report;
+        try
+        {
+            var repo = RepoFactory.Create();
+            report = CatalogCrossCheck.Run(repo, settings.PackagesPath ?? cfg.PackagesPath);
+        }
+        catch (Exception ex)
+        {
+            return RenderHelpers.Render(kind, ToolResult<object>.Fail("NO_INDEX",
+                $"Cross-check needs the SQLite index: {ex.Message}",
+                "Run `d365fo index build` then `d365fo index extract` first."));
+        }
+
+        var result = ToolResult<object>.Success(new
+        {
+            clean = report.Clean,
+            objectsConsidered = report.ObjectsConsidered,
+            gaps = report.Gaps.Select(g => new { catalog = g.Catalog, item = g.Item, observed = g.Observed, detail = g.Detail }),
+            uncoveredCount = report.Uncovered.Count,
+            uncovered = settings.ShowUncovered
+                ? report.Uncovered.Select(u => new { folder = u.Folder, models = u.Models })
+                : null,
+            unusedCount = report.Unused.Count,
+            verdict = report.Clean
+                ? "Every pattern and family the installation uses is covered by a catalog that claims to cover it."
+                : $"{report.Gaps.Count} catalog gap(s) — the tool will answer wrongly about these. Regenerate the catalog named in each.",
+        });
+
+        var rc = RenderHelpers.Render(kind, result, _ =>
+        {
+            foreach (var g in report.Gaps)
+            {
+                AnsiConsole.MarkupLine($"[red]{RenderHelpers.Escape(g.Catalog)}[/] {RenderHelpers.Escape(g.Item)} [grey]({g.Observed})[/]");
+                AnsiConsole.MarkupLine($"  [grey]{RenderHelpers.Escape(g.Detail)}[/]");
+            }
+            if (settings.ShowUncovered)
+                foreach (var u in report.Uncovered)
+                    AnsiConsole.MarkupLine($"[yellow]uncovered[/] {RenderHelpers.Escape(u.Folder)} [grey]({u.Models} model(s))[/]");
+            AnsiConsole.MarkupLine(report.Clean
+                ? $"[green]no catalog gaps[/] ({report.Uncovered.Count} uncovered famil(ies), {report.Unused.Count} unused entr(ies))"
+                : $"[red]{report.Gaps.Count} catalog gap(s)[/], {report.Uncovered.Count} uncovered famil(ies)");
+        });
+
+        return rc != 0 ? rc : report.Clean ? 0 : 2;
+    }
+}