diff --git a/README.md b/README.md index c4e708e7..7d5f7628 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,4 @@ # twinBASIC User Documentation -General user documentation for twinBASIC - -> [!warning] -> -> Work in progress - This repository is the home for general user documentation for everything to do with twinBASIC. **If you would like to quickly contribute**, feel free to: diff --git a/WIP.ExamplesBuild.md b/WIP.ExamplesBuild.md new file mode 100644 index 00000000..44589de6 --- /dev/null +++ b/WIP.ExamplesBuild.md @@ -0,0 +1,336 @@ +# Compiling the Documentation's Code Samples — Design Notes + +See [WIP.md](WIP.md) for the maintenance guide. This file designs the tool that answers +*does this sample actually build*, which nothing currently asks. + +**Status: design. Nothing here is implemented.** Measurements are marked as measured; +everything else is a decision or an open question. + +## The problem + +Round 6 pointed `tbbuild` at the reference for the first time and found two samples that do +not run: `WinNativeCommonCtls/ListView`'s flagship example passes an icon key the same +package's prose says raises 35613, and `Core/Event`'s first sample is a `Sub` with no name. +Both shipped. Every gate was green over them, because a `tb` fence is something +`check_code_regions.mjs` protects the *contents* of and never evaluates. + +The five `Dim X As New Worksheet` samples in `Core/` (`Dim`, `New`, `Private`, `Public`, +`Static`) are the same class arriving from the import side: `Worksheet` is an Excel type, +and in a bare twinBASIC project that line does not compile at all. + +## Why this is not part of a build + +Stated first, because it is the constraint everything else bends around. + +- **Cost.** An IDE cold start is 8–11 s per project and flat in project size (WIP.md, + measured). A normal `build.bat` is ~4 s. One probe project would triple it. +- **`npm install` must remain sufficient** to build the docs. A twinBASIC install is not on + that path, and `dot.mjs`'s setup-failure behaviour exists to preserve exactly that. +- **CI cannot run it.** The harness needs Windows, a private desktop, and a CDP-reachable + WebView2. None of that exists on the CI box. + +So this is a **separate on-demand tool** — `examples.bat` over `scripts/check_examples.mjs` +— never invoked from `build.bat`, `check.bat`, `test.bat`, or either CI workflow. A sample +regression is caught when someone runs it, which is the same deal `sweep_a11y.mjs` (~20 min, +full site) already makes. + +## Opt-in, because the corpus says so + +A census of the 1,097 `tb` fences, classified by top-level shape: + +| shape | count | share | wrapper needed | +|---|---:|---:|---| +| whole `Class` / `Module` | 103 | 9.4% | none — drop in as its own file | +| whole procedure | 349 | 31.8% | a generated `Module` | +| declarations only | 22 | 2.0% | a generated module's declaration section | +| loose statements | 621 | 56.6% | a generated `Private Sub` | +| empty | 2 | 0.2% | — | + +**This disagrees with the census in WIP.md**, which reported 36 / 357 / 457 / 250. The +`procedure` figures agree closely (349 against 357), which suggests the difference is in how +the other three were split rather than in the extraction. The consequence is load-bearing: +WIP.md's taxonomy implies the second-largest bucket wants a *module declaration section*, +and the re-run says the largest bucket by far — 57% — is loose statements wanting a **`Sub` +body**. A generator built to the old numbers picks the wrong wrapper for most of the corpus. + +Neither census can tell a wrappable statement sequence from a true fragment (an `If` with an +elision, a snippet containing `...`). WIP.md put those at 250; they are distributed through +the `statements` row above. + +That is the argument for **opt-in**. A gate demanding every fence compile needs ~250 opt-outs +on day one, and a list of 250 exceptions is a list nobody maintains. Mark the fences that +claim to be complete and leave the rest alone — which makes the marker the thing to get +right, not the harness. + +## The markup + +**In the fence info string.** `builder/render.mjs:393` is why: + +```js +const lang = tok.info ? tok.info.trim().split(/\s+/)[0] : ""; +``` + +The fence renderer takes the first whitespace-separated token as the language and discards +the rest, so anything after `tb` is already invisible. Verified against the real pipeline +(`createMarkdownIt` + `initHighlighter`, not a bare markdown-it — the distinction WIP.md's +[Source dashes](WIP.md#source-dashes) section was burned by): + +| property | result | +|---|---| +| marked fence renders byte-identical HTML to a plain one | **true** | +| `maskCodeRegions` still hides the body | **true** | +| mask round-trips the marked fence | **true** | +| `applyPreRenderRewrites` leaves it byte-identical | **true** | + +So the markup costs nothing at render time, cannot reach the HTML, and cannot perturb +`check_code_regions.mjs`, which compares fence *contents* and never sees the info string. + +Shape — space-separated `key=value` pairs and bare flags after the language token: + + ```tb project=office-late slot=procedure id=getobject-1 run + +**No backticks in it.** CommonMark forbids them in a backtick fence's info string, and +`maskCodeRegions` skips such a fence outright (`render.mjs:1697`). + +| key | meaning | default | +|---|---|---| +| `project` | which template project to build into | `console` | +| `slot` | where the code goes | inferred | +| `id` | stable name for reporting and pinning | derived from file + ordinal | +| `run` | execute and capture Debug output, not merely compile | compile only | +| `expect-error` | the sample is *meant* not to compile; assert this error | — | + +**`slot` is inferred by default** by the classifier above, stated only when inference is +wrong. A misinference is self-reporting — it produces a compile error rather than a silent +pass — but the reporter must name the inferred slot in any error, or the author is left +debugging code that is correct. + +`expect-error` exists because the docs legitimately show code that does not compile, in order +to say why. Without it those pages could never carry a marker. + +## Template projects + +Version-controlled exported trees under `test/example-projects//` — a `Settings` file +plus `Sources/`, exactly what `twinBASIC_win32.exe export` produces and what `tbrun` already +consumes. + +| template | extra references | for | +|---|---|---| +| `console` | as shipped (stdole + VB package) | the large majority | +| `office-late` | none | `CreateObject` / `GetObject` Office examples | +| `office-early` | Excel + Word type libraries | early-bound `Excel.Application` | +| `forms` | VB Forms package + a blank form | control and event examples | +| `win32` | stdole only | `Declare` / API examples | + +`project.references` in `Settings` is a plain JSON array, so a template is made by appending +an entry. No IDE needed: + +```json +{ + "id": "{00020813-0000-0000-C000-000000000046}", + "name": "Microsoft Excel 16.0 Object Library", + "path32": "C:\\Program Files\\Microsoft Office\\Root\\Office16\\EXCEL.EXE", + "path64": "C:\\Program Files\\Microsoft Office\\Root\\Office16\\EXCEL.EXE", + "symbolId": "Excel", "versionMajor": 1, "versionMinor": 9, "lcid": 0 +} +``` + +**Measured, on 64-bit Office 16 with no `win32` typelib registered at all.** The default +build is 32-bit (`Len(CLngPtr(0))` = 4), and both binding styles reach Excel from it: + +| probe | result | +|---|---| +| `CreateObject("Excel.Application")`, no reference | `TypeName` = `Application`, `Version` = `16.0`, clean `Quit` | +| `New Excel.Application` with the reference above | same, plus `Excel.Worksheet` resolves and a real `Range("A1")` round-trip | +| `Dim X As New Excel.Worksheet` | compiles, then **raises `0x80004002` E_NOINTERFACE** on first use | + +So **`office-early` does not need pinning to x64**, which was the initial assumption and was +wrong. + +Note carefully what this does *not* show. `Excel.Application` is an out-of-process +`LocalServer32`, so Windows marshals across the bitness boundary natively. It says nothing +about twinBASIC's own 32/64 bridge — a 64-bit host process plus IPC — which is what matters +for **in-process**, 64-bit-only DLLs, the case VB6 genuinely cannot do. A template needing +one of those has to be measured separately. + +**`New Excel.Worksheet` does not work, and the declaration compiling proves nothing.** +`Worksheet` is a non-creatable *interface*, not a coclass. Measured in the IDE debugger with +the Excel reference present: the `Dim a As New Excel.Worksheet` line raises nothing, and then +both `a.Name` (which forces `As New`'s deferred instantiation) and an explicit +`Set b = New Excel.Worksheet` raise **-2147467262 / 0x80004002, “No such interface +supported”**. + +This matters editorially, because the tempting minimal fix for the five `Core/` pages is to +qualify the existing line as `Excel.Worksheet` and add a reference — which would preserve +VBA-Docs' original nonsense in a form that merely type-checks. The honest ports are +`Excel.Application`, which *is* a creatable coclass, or reaching a worksheet through the +object model the way real code does: + +```tb +Dim app As New Excel.Application +Dim wb As Excel.Workbook: Set wb = app.Workbooks.Add() +Dim ws As Excel.Worksheet: Set ws = wb.Worksheets(1) +``` + +That path is measured working end to end — `TypeName(ws)` = `Worksheet`, with a real +`Range("A1")` round-trip — from the default 32-bit build against 64-bit Office. + +## Batching, which is the whole cost question + +One project per fence is unaffordable: 1,097 fences at ~10 s is over three hours serially. +What makes it tractable is the measurement that **IDE cost is flat in project size** (WIP.md: +a one-file project and a 32-probe project both land at ~10 s, because what is paid for is IDE +startup, not compilation). + +So pack many fences into one project. At ~100 per project that is ~11 projects: roughly two +minutes serially, well under a minute at concurrency 8. + +Collision rules, all forced by putting unrelated samples in one compilation unit: + +- **One generated `Module tbx_` per fence.** The hash covers source path plus fence + ordinal, so it is stable across runs and traceable without a lookup table. +- **Everything generated is `Private`.** Two samples both declaring `MyString` must not see + each other. +- **`Sub Main` comes from the template, never from a fence.** A `module`-slot fence bringing + its own `Main` needs renaming on the way in. +- **A generated module must not share a name with the project.** `project.name = "ProbeWS"` + beside `Module ProbeWS` makes `[RunAfterBuild]`'s `ProbeWS.ProbeWS.Probe` ambiguous, and + the IDE refuses it with *"'ProbeWS' is ambiguous. Could be: [Module] ProbeWS.ProbeWS / + [Library] ProbeWS"*. Cost three probes that looked like hangs before the cause was seen, + because the error arrives at *execution* time and not at build time — `tbbuild` reports + zero errors and the run simply produces nothing. The `tbx_` scheme avoids it by + construction, but the template's `project.name` is the other half and has to be checked. +- **`[RunAfterBuild]` is exclusive.** Whether twinBASIC accepts more than one is untested; + assume not, so `run` fences get their own project, or one generated dispatcher calls each + in turn. + +Two things a batch runner must do that a single-fence runner need not: + +- **Keep a source map.** Errors return against generated file and line; the report has to + name the `.md` file, the fence, and the line within it. Emitted while generating, not + reconstructed afterwards. +- **Bisect on a compiler crash.** twinBASIC runs the compiler in-process with user code, so a + bad probe can take it down (WIP.md; `tbbuild` exits 4 for it) — and in a batch that loses + all hundred fences with it. On crash, split and recurse: O(log n) extra builds, paid only + on failure. + +## Traps already paid for + +Each cost a run during the probing that produced this file, or is recorded in `tbrun.mjs` +from earlier work. + +- **`project.buildPath` must be an explicit file.** The default `${SourcePath}\Build\...` + template opens a native Save dialog, which on the private desktop is invisible and + unreachable, so the build silently never happens — and the WebView2 renderer stays + responsive throughout, so every health check says the IDE is fine. +- **`MsgBox` hangs a run-mode probe, invisibly.** The HelloWorld template's `Main` is a + `MsgBox` and had to come out before probing. Run-mode fences must be screened for `MsgBox` + / `InputBox` and refused, not discovered at the timeout. +- **`tbrun` could not run concurrently, though WIP.md said it could — now fixed.** It staged + into a fixed `%TEMP%\tbrun\src` with a fixed `tbrun-probe.exe`, so a second invocation + `rmSync`d the first one's tree (reproduced: `EPERM` on a path the failing script had never + touched). Worse, shutdown was `taskkill /F /T /IM twinBASIC.exe` — machine-wide, so it ended + every concurrent run's IDE *and* the one the user had open. The workspace and `project.id` + are now keyed to `--port`, and `tbbuild` reports the IDE pid (`ide-pid:` in text, `idePid` + in `--json`) so the kill is by pid tree. Verified: two runs at once, 25 s wall, each + capturing its own output. +- **A probe's quiet period must outlast what it waits on.** The default 2500 ms expires while + Excel is still starting, and the run reports success having captured nothing. Anything + driving an out-of-process server needs a much longer `--quiet`. **A post-build error can + land after the window closes too**: the ambiguity error above surfaced 17.5 s after + `[BUILD] Executing`, roughly 6 s past a `--quiet 12000` capture, so the console the harness + read was silent while the console the IDE ended up holding named the fault outright. When a + probe comes back empty, re-read the live console (`--keep`, then CDP) before concluding it + hung. +- **`tbrun` used to capture only the last ~11 output lines, and said nothing about the rest. + Fixed — but read this before touching the reader.** It scraped `.innerText` off the DEBUG + CONSOLE pane, and the pane is a `createListView()`, which renders only the rows that fit. + Measured against the old reader: a probe printing 19 lines returned 11, and a probe printing + 120 lines returned 11 — the *tail* each time, with a clean-looking first line and no + truncation marker. + + This is worse than the empty-capture case above, because a truncated capture looks like a + complete one. The first probe written for this session printed a seven-line `Format` block + followed by a `vbDatabaseCompare` block, and came back holding only the second, reading + exactly like a probe that had simply not run the first half. + + **The obvious fix is the wrong one, and it was briefly written up here as fact.** `--raw` on + the 120-line probe returned exactly 22 lines, twice 11, which looks like *Show Timestamps* + costing half the budget — so turn it off and get 22. It does not: `showTimestamps` only sets + a `--timestampsDisplay` CSS variable, and the timestamp `` is in the data either way. + Measured on a live IDE holding 121 entries, flipping the option in place: `visibleCount` 10 + and 12 rendered rows in **both** states; `innerText` yields 24 lines with timestamps and 13 + without, because an `inline-block` span breaks the line in `innerText` and nowhere else. The + row budget never moved. Turning timestamps off buys nothing. + + **What the reader does now** is take `debugConsoleContent.dataNodes`, which is the complete + log — `addItem()` appends at `itemCount` and nothing in `main.js` ever removes an entry, so + only `clear()` (that is, `Debug.Cls`) empties it. The walk is the IDE's own + `tbDebugConsole_ClipboardCopyAll` minus the clipboard write, the same borrow `tbbuild` makes + for the diagnostics report, and it strips the timestamp by slicing past the first `` + rather than by matching a line against a regex. The 120-line probe now returns 120. A + missing `dataNodes` is refused outright rather than falling back to the pane, because a + silent fallback would restore exactly the failure this replaces. + + For the batch runner this removes a hard constraint: one result line per fence is now fine + at any batch size. +- **Two IDEs must not hold one source tree.** `tbbuild` takes the project directory as given + and does not stage a copy the way `tbrun` does, so two concurrent builds pointed at the same + folder — distinct `--port`s, distinct desktops, everything else correct — both wedge and + neither ever returns. Cost two runs and looked like the renderer-blocked failure above. + Concurrency needs a tree per run, not just a port per run. +- **`export` needs the output folder to exist** (one level only), and stdin redirected + (`` in each entry rather than a line of its own, so `--raw` + is a different slice of that string. Do not "fix" the old truncation by turning *Show + Timestamps* off: that option only sets a CSS variable, and the row budget does not move + --- see [WIP.ExamplesBuild.md](WIP.ExamplesBuild.md) for the measurement. - **A probe must start with `Debug.Cls`.** The IDE logs its own build to the same console and the linker writes there *after* the build, so without a clear you capture your output interleaved with `[LINKER]` lines. The script warns rather than guessing which lines are @@ -340,6 +431,21 @@ It settles on a quiet period rather than a sentinel, so no probe has to print a script knows about. Distinct `--port` values let probes run concurrently, exactly as `tbbuild`'s do. +**That last sentence was false when it was written, and is true now.** The staging directory +was a fixed `%TEMP%\tbrun\src`, so a second run deleted the first one's tree, and shutdown +was `taskkill /F /T /IM twinBASIC.exe` --- machine-wide, taking out every concurrent run's +IDE and the one you had open yourself. The workspace and `project.id` are keyed to `--port` +now, and `tbbuild` reports the IDE's pid (`ide-pid:` in text, `idePid` in `--json`) so the +kill is by pid tree. + +`tbrun` also **harvests COM servers a probe leaves behind**, because nothing else can: an +`EXCEL.EXE` from `CreateObject` has `svchost.exe` for a parent, so no tree kill reaches it, +every activation is its own process, and `Quit` does not end one while any reference is +outstanding. The sweep is a before/after snapshot diff restricted to processes that are new, +on an image allowlist, *and* windowless --- a new one that has a window is reported and left +alone, since that cannot be told from a copy the user opened. `--no-reap` turns it off, and +concurrent runs driving the same server should use it and sweep once at the end. + ## Page template Match the existing style. Worked examples to imitate: @@ -1381,6 +1487,19 @@ Batching matters too. One project per IDE is the scaling unit, so 393 whole unit each is over an hour serially; several fences per probe project, run concurrently, is what makes it minutes. That is the same arithmetic the probe-suite note above works through. +**The gate is designed in [WIP.ExamplesBuild.md](WIP.ExamplesBuild.md)** --- the opt-in fence +markup, the template projects, the batching and bisect-on-crash rules, and the measured +evidence that the markup is free at render time. It is a separate on-demand tool by design +and is never wired into `build.bat`, `check.bat`, `test.bat` or CI. + +> **Do not take the table above as settled.** Re-running the census against the slot +> taxonomy that design needs gives 103 / 349 / 22 / 621 over 1,097 fences: the `procedure` +> row agrees closely, but the largest bucket by far is **loose statements wanting a `Sub` +> body**, not declarations wanting a module. A generator built to the older split picks the +> wrong wrapper for most of the corpus. Both classifiers are heuristic and neither +> distinguishes a wrappable statement sequence from a true fragment; the discrepancy and its +> consequences are worked through in the design file. + ### A script is findable only if its bare name is a token prefix somewhere lunr's tokeniser splits on **whitespace and hyphens only** (`/[\s\-]+/`), and the site's @@ -1690,6 +1809,26 @@ and `Features/Standard-Library/New-Functions` document what `Debug.Print` emits with comma separators, where the print-zone padding *is* the behaviour being shown, and both rendered it as single spaces. +> **Those two pages were still wrong after that fix, and the reason is worth +> keeping.** A pipeline can only preserve padding that reaches it, and the +> padding was never in their *source*. Measured through `tbrun` against the +> pages' own samples: `New-Functions` claimed `1 2 3` +> where the run prints `' 1 2 3 '`, and `Pointers` +> claimed `1 2`, `3 4` and `4` where the runs print `' 1 2 '`, +> `' 3 4 '` and `' 4 '`. Every one is missing the leading space a +> positive number carries where its sign would be, and the trailing space; the +> `Pointers` pair were also showing two spaces for a thirteen-space gap. A fifth claim on +> `New-Functions`, ``7 1`` where the run prints `' 7 1 '`, went the same way. +> +> **An inline code span cannot carry a leading or trailing space naively**, which +> is the trap that keeps this defect coming back. CommonMark strips one space +> from each end of a code span whose content is not all spaces, so writing +> `` ` 1 … 3 ` `` renders as `1 … 3` --- the exact value the page is trying to +> state, silently de-padded by the parser rather than by anything in `builder/`. +> Double the outer spaces to defeat it, and verify in the built HTML rather than +> by eye. Four sites were fixed this way and the rendered `` now matches +> the measured output byte for byte. + **Two changes, and neither works alone:** - `compress.mjs` now treats inline `` as a preserved region as well as diff --git a/builder/census_attributes.mjs b/builder/census_attributes.mjs new file mode 100644 index 00000000..520dcdb9 --- /dev/null +++ b/builder/census_attributes.mjs @@ -0,0 +1,645 @@ +#!/usr/bin/env node +// Census every attribute used by the twinBASIC packages an IDE install ships. +// +// node builder/census_attributes.mjs [options] +// +// --ide twinBASIC install root (default: $TB_IDE, else the +// newest %USERPROFILE%/Desktop/twinBASIC_IDE_BETA_*) +// --src census an already-exported tree and do not export +// --cache where exports are kept (default: %TEMP%/tb-census) +// --refresh re-export even if the cache already has this build +// --samples also census projects/ and addins/, not just packages/ +// --attr restrict the report to one attribute +// --json emit JSON instead of markdown +// --out write the report to a file instead of stdout +// --quiet suppress progress on stderr +// +// Exit codes: 0 report produced, 2 the harness failed. +// +// ---------------------------------------------------------------- why this +// +// `Reference/Attributes.md` states an `Applicable to:` line per attribute, and +// the only evidence for most of them is what the shipped packages do. That +// evidence was gathered by hand, one attribute at a time, with a grep whose +// blind spots were rediscovered on each pass. This is that sweep, done once, +// over every attribute at once. +// +// **A census is evidence, not applicability.** It says where an attribute IS +// used, never where it MAY be used -- `scripts/gen_attribute_probes.mjs` plus +// `scripts/tbbuild.mjs` answer that, by asking the compiler. The two are +// complementary and the distinction is not pedantic: `[Hidden]` is used on a +// whole CoClass and on Class and Interface members, and the compiler refuses +// it (TB5155) on the Interface lines inside a CoClass body -- a census alone +// would never have found the boundary. `gen_attribute_probes.mjs` records the +// converse trap under [RedirectToStaticImplementation]: a census grouped by +// declaration keyword said "on a Property Get, a Function and a Sub", the +// entry went out saying "procedure in a Class", and the probe returned TB5155 +// because every one of those 82 uses is inside an Interface. So this groups by +// ENCLOSING CONSTRUCT as well as by keyword, and reports the pair. +// +// ------------------------------------------------- what the scanner must not do +// +// Six ways a naive sweep of this corpus gets a wrong answer, each measured +// against the BETA 983 packages rather than imagined: +// +// 1. A line matcher misses 292 of 7,604 attribute lines (3.8%), because +// `[Description("..." & vbCrLf & _` closes several lines later. Those are +// dropped silently, so the count looks plausible. Hence a character +// scanner that balances brackets across lines. +// 2. An attribute list is comma-separated -- `[DispId(126), Hidden]` -- and +// DAO.twin writes most of its Hidden uses that way. Matching `[Hidden]` +// finds a fraction of them. +// 3. Argument text has to be stripped before splitting on the comma, or +// `[Description("Returns an array of child controls, given the container")]` +// contributes an attribute named `given`. +// 4. An escaped identifier is spelled like an attribute: `[_HiddenModule].Foo`, +// `[_MAX] = 0`. What separates them is the tail -- an attribute is followed +// by a declaration, an escaped identifier by `.`, `=` or `(`. +// 5. DAO.twin writes declarations as `/* voffset &H00A8*/ Property Get X()`, +// so an inline block comment has to be removed, not used to skip the line. +// Skipping cost 14 Interface-member sites, which then read as `End Interface`. +// 6. Attributes are also written inline -- `[Default] Interface X` -- so the +// declaration is not always on the next line. +// +// One result to read before calling a row impossible: **a twinBASIC `Type` can +// contain `DeclareWide` members.** `CustomControls.twin`'s `Type SerializeInfo` +// holds a dozen, so `Type / DeclareWide` is a real construct and not a stack +// fault. It was assumed to be one here, and the assumption was wrong. +// +// Only TYPE blocks are tracked for the enclosing construct. Procedures are +// deliberately not pushed: an Interface prototype (`Sub Ping()`) has no body and +// no `End Sub`, so tracking procedures unbalances the stack on every interface +// in the corpus. Anything the scanner cannot resolve goes to an `unresolved` +// bucket and is reported -- a census that quietly buckets its own confusion is +// how the wrong answer gets published with a number beside it. +import { execFileSync } from "node:child_process"; +import { existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const REPO = path.dirname(path.dirname(fileURLToPath(import.meta.url))); +const ATTR_DOC = path.join(REPO, "docs", "Reference", "Attributes.md"); + +const argv = process.argv.slice(2); +const flag = (n) => argv.includes("--" + n); +const opt = (n, d) => { const i = argv.indexOf("--" + n); return i < 0 ? d : argv[i + 1]; }; +const die = (code, msg) => { console.error(msg); process.exit(code); }; +const log = (...a) => { if (!flag("quiet")) console.error(...a); }; + +if (flag("help")) { + console.log(readFileSync(fileURLToPath(import.meta.url), "utf8") + .split("\n").filter((l) => l.startsWith("//")).slice(1, 18).map((l) => l.slice(3)).join("\n")); + process.exit(0); +} + +// ------------------------------------------------------------- the install +// An install path contains a username, so it is never hardcoded -- the same +// rule tbbuild.mjs follows, and for the same reason. +function findInstall() { + const given = opt("ide", process.env.TB_IDE); + if (given) { + // Accept either the install root or the IDE exe inside it. + const root = /\.exe$/i.test(given) ? path.dirname(given) : given; + if (existsSync(path.join(root, "packages"))) return root; + if (existsSync(path.join(path.dirname(root), "packages"))) return path.dirname(root); + die(2, `no packages/ under ${root} -- pass the install root with --ide`); + } + const home = process.env.USERPROFILE || os.homedir(); + const desktop = path.join(home, "Desktop"); + if (!existsSync(desktop)) die(2, "no Desktop to search; pass --ide or set TB_IDE"); + const betas = readdirSync(desktop) + .map((n) => /^twinBASIC_IDE_BETA_(\d+)$/.exec(n)) + .filter(Boolean) + .map((m) => ({ n: Number(m[1]), dir: path.join(desktop, m[0]) })) + .filter((b) => existsSync(path.join(b.dir, "packages"))) + .sort((a, b) => b.n - a.n); + if (!betas.length) die(2, "no twinBASIC_IDE_BETA_* with a packages/ folder on the Desktop; pass --ide"); + return betas[0].dir; +} + +const buildNumberOf = (root) => (/_BETA_(\d+)$/.exec(root)?.[1]) ?? "unknown"; + +// ------------------------------------------------------------- the export +// The .twin sources live inside .twinproj archives; `export` unpacks one. +// Two traps, both from WIP.md and both still live: the output folder must +// already exist (only one level is created), and stdin has to be detached or +// the executable consumes the caller's and later iterations never run. +function exportAll(root, cacheDir, includeSamples) { + const exe = path.join(root, "bin", "twinBASIC_win32.exe"); + if (!existsSync(exe)) die(2, `no compiler at ${exe}`); + + const roots = [path.join(root, "packages")]; + if (includeSamples) { + for (const d of ["projects", "addins"]) { + const p = path.join(root, d); + if (existsSync(p)) roots.push(p); + } + } + + const projects = []; + for (const r of roots) { + for (const entry of readdirSync(r, { withFileTypes: true })) { + if (!entry.isDirectory()) continue; + const dir = path.join(r, entry.name); + for (const f of readdirSync(dir)) { + if (f.toLowerCase().endsWith(".twinproj")) { + projects.push({ proj: path.join(dir, f), name: entry.name, group: path.basename(r) }); + } + } + } + } + if (!projects.length) die(2, `no .twinproj found under ${roots.join(", ")}`); + + mkdirSync(cacheDir, { recursive: true }); + let exported = 0; + for (const p of projects) { + const out = path.join(cacheDir, p.group, p.name); + if (existsSync(out) && !flag("refresh")) continue; + mkdirSync(out, { recursive: true }); + try { + execFileSync(exe, ["export", p.proj, out + path.sep, "--overwrite"], + { stdio: ["ignore", "ignore", "ignore"] }); + exported++; + } catch { + log(` ! export failed: ${p.name}`); + } + } + log(` exported ${exported} project(s), ${projects.length} total in cache`); + return projects.map((p) => ({ ...p, dir: path.join(cacheDir, p.group, p.name) })); +} + +// ------------------------------------------------------------- the scanner +const TYPE_KEYWORDS = ["Class", "Module", "Interface", "CoClass", "Enum", "Type", "Union"]; +const MODS = "(?:Public|Private|Friend|Global|Protected|Static|ReadOnly|WriteOnly|Default|" + + // NotDispatchable is here because the corpus uses it and nothing else would + // say so: a modifier this list does not know stops the block being pushed at + // all, and the mismatched `End Class` then pops somebody else's block. Swept + // for empirically -- Private, Public, Protected and NotDispatchable are the + // only words that precede a block keyword in BETA 983. + "Shared|Overrides|Virtual|Const|WithEvents|Partial|MustOverride|NotInheritable|" + + "NotDispatchable)"; +// The name after the keyword is captured so a FIELD named after a block keyword +// can be rejected. Four UDTs in the packages declare `Type As Long`, and read as +// an opener that never closes it swallowed the rest of the file: one of them put +// 368 Declares inside a phantom `Type`, which is not a construct that exists. +// The name may itself be an escaped identifier: VBA declares `Module +// [_HiddenModule]`, the only one in the corpus, and a bare-identifier pattern +// skipped the open -- so its `End Module` 1,277 lines later popped a block it +// did not own. +const OPEN_RE = new RegExp( + `^\\s*(?:${MODS}\\s+)*(${TYPE_KEYWORDS.join("|")})\\b\\s+([A-Za-z_]\\w*|\\[[^\\]]*\\])`, "i"); +const CLOSE_RE = new RegExp(`^\\s*End\\s+(${TYPE_KEYWORDS.join("|")})\\b`, "i"); +const DECL_RE = new RegExp( + `^\\s*(?:${MODS}\\s+)*(Class|Module|Interface|CoClass|Enum|Type|Union|Sub|Function|` + + `Property|Event|DeclareWide|Declare|Implements)\\b`, "i"); +const VAR_RE = new RegExp(`^\\s*(?:${MODS}|Dim)\\s+[\\w\\[]`, "i"); +const BLOCK_COMMENT_RE = /\/\*[^*]*\*+(?:[^/*][^*]*\*+)*\//g; + +const decomment = (s) => s.replace(BLOCK_COMMENT_RE, " "); + +// Blank string contents in place, preserving length and quotes, so offsets stay +// valid and no comma or bracket inside a literal is ever read as syntax. +function blankStrings(s) { + let out = "", inStr = false; + for (let i = 0; i < s.length; i++) { + const c = s[i]; + if (c === '"') { inStr = !inStr; out += c; continue; } + if (!inStr && c === "'") { out += " ".repeat(s.length - i); break; } + out += inStr && c !== "\n" ? " " : c; + } + return out; +} + +// Read the run of attribute groups starting at lines[i], which may span lines. +// Returns null when the line does not open one. +function readAttrRun(lines, i) { + const startsWithBracket = (s) => decomment(s).trimStart().startsWith("["); + if (!startsWithBracket(lines[i] ?? "")) return null; + + const groups = []; + let line = i, col = decomment(lines[line]).length - decomment(lines[line]).trimStart().length; + let text = decomment(lines[line]); + + for (;;) { + while (col < text.length && /\s/.test(text[col])) col++; + if (text[col] !== "[") break; + + let depth = 0, inStr = false, closed = false; + const group = []; + scan: for (;;) { + while (col < text.length) { + const c = text[col]; + if (inStr) { if (c === '"') inStr = false; } + else if (c === '"') inStr = true; + else if (c === "[") depth++; + else if (c === "]") { depth--; if (depth === 0) { col++; closed = true; break scan; } } + group.push(c); + col++; + } + // Unclosed on this line: continue onto the next. This is the 3.8% case. + line++; + if (line >= lines.length) break scan; + text = decomment(lines[line]); + col = 0; + group.push(" "); + } + if (!closed) return null; + groups.push(group.join("").replace(/^\[/, "")); + // Another group may follow immediately, on the next line, or past a blank + // or comment line. Skipping comments matters: VBA/Strings.twin writes + // [PreserveSig(False), ...] + // ' Function to return the position of ... + // [Description("..." & vbCrLf & _ + // and stopping at the comment made the second group read as the + // declaration, which is how 142 sites landed in the unresolved bucket. + let probeLine = line, probeText = text, probeCol = col; + for (;;) { + // Scan the COMMENT-STRIPPED text. WebView2.twin opens with + // [WindowsControl("...")] ' [WindowsControl("...png")] + // and a raw scan stops on that comment, so the [ClassId] group below it + // was read as the declaration instead of joining the run. + const probeCode = blankStrings(probeText); + while (probeCol < probeCode.length && /\s/.test(probeCode[probeCol])) probeCol++; + if (probeCol < probeCode.length) break; + let k = probeLine + 1; + while (k < lines.length) { + const t = decomment(lines[k]); + if (!blankStrings(t).trim()) { k++; continue; } // blank, or a ' comment + break; + } + if (k >= lines.length) break; + const nxt = decomment(lines[k]); + if (!nxt.trimStart().startsWith("[")) break; + probeLine = k; probeText = nxt; probeCol = 0; + } + if (probeText[probeCol] !== "[") break; + line = probeLine; text = probeText; col = probeCol; + } + + return { groups, endLine: line, rest: text.slice(col) }; +} + +// Split a group's text into attribute names. Arguments are removed first, or a +// comma inside them splits the list in the wrong place. +function attrNames(group) { + const flat = blankStrings(group).replace(/\([^()]*\)/g, "()"); + const names = []; + for (const piece of flat.split(",")) { + const m = /^\s*([A-Za-z_]\w*)\s*(\(\))?\s*$/.exec(piece); + if (m) names.push({ name: m[1], hasArgs: Boolean(m[2]) }); + else if (piece.trim()) names.push({ name: null, raw: piece.trim() }); + } + return names; +} + +function classify(decl, container) { + const d = decomment(decl); + const m = DECL_RE.exec(d); + if (m) { + const k = m[1]; + return k[0].toUpperCase() + k.slice(1).replace(/^eclarewide$/i, "eclareWide"); + } + if (/^\s*End\s+\w/i.test(d)) return null; // unresolved + if (container === "Enum" && /^\s*\[?\w/.test(d)) return "EnumMember"; + if (container === "Type" || container === "Union") { + if (/^\s*\w+\s+As\s+/i.test(d)) return "TypeMember"; + } + // `Const` is in MODS, so DECL_RE consumes it as a modifier and never reports + // it as a kind. The distinction is load-bearing: `Attributes.md` states + // "constants in a module" and "variables in a Class" as different targets, + // and [DllExport] is documented on a Const and refused on a variable. + if (/^\s*(?:\w+\s+)*Const\b/i.test(d)) return "Const"; + if (VAR_RE.test(d) || /^\s*\w+\s+As\s+/i.test(d)) return "Variable"; + return null; +} + +function scanFile(file, pkg) { + const raw = readFileSync(file, "utf8").replace(/^/, ""); + const lines = raw.split(/\r?\n/); + const sites = [], problems = []; + const stack = []; + + for (let i = 0; i < lines.length; i++) { + const run = readAttrRun(lines, i); + + if (run) { + const tail = blankStrings(decomment(run.rest)).trim(); + // An escaped identifier, not an attribute: [_HiddenModule].Foo, [_MAX] = 0. + const isExpression = /^[.=(]/.test(tail) || (run.groups.length === 1 && /^[-+*/&<>]/.test(tail)); + if (!isExpression) { + // What follows the closing ] is the declaration only if it is CODE. A + // trailing line comment is common -- `[DLLStackCheck(False), ...] ' + // NOTE: PreserveSig(FALSE) here` -- and taking it as the declaration + // put the comment text in the report and lost the real target. + // blankStrings erases a ' comment, so a blank result means "no code". + let decl = blankStrings(decomment(run.rest)).trim() ? decomment(run.rest) : null; + let declLine = run.endLine; + if (!decl) { + let j = run.endLine + 1; + while (j < lines.length) { + const t = decomment(lines[j]); + if (!blankStrings(t).trim()) { j++; continue; } + // A conditional-compilation or #Region directive can sit between an + // attribute and what it decorates -- DTPicker.twin puts + // `#If FEATURE_OLEDRAGDROP Then` there. + if (/^\s*#/.test(t)) { j++; continue; } + break; + } + decl = decomment(lines[j] ?? ""); + declLine = j; + } + const container = stack.at(-1)?.kind ?? "(file)"; + const kind = classify(decl, container); + const names = run.groups.flatMap(attrNames); + for (const n of names) { + // An Enum member may BE an escaped identifier -- Report.twin declares + // `[ ]`, `[A4 Portrait]`, `[Letter Landscape]` as member names. Those + // are the member, not an attribute on it, and a name with a space (or + // none at all) is what tells them apart. + // + // The residual ambiguity is stated rather than hidden: inside an Enum + // a bare `[Hidden]` is a valid attribute AND a valid escaped + // identifier, and nothing local decides which. This treats it as the + // attribute, which is right for every case in BETA 983. + if (!n.name && container === "Enum") continue; + if (!n.name) { problems.push({ file, pkg, line: i + 1, why: "unparsed attribute text", text: n.raw }); continue; } + sites.push({ + attr: n.name, hasArgs: n.hasArgs, pkg, file, line: i + 1, + container, kind: kind ?? "UNRESOLVED", + decl: decl.trim().slice(0, 100), + inProject: stack.map((s) => s.kind).join(">"), + }); + if (!kind) problems.push({ file, pkg, line: i + 1, why: "declaration not classified", text: decl.trim().slice(0, 100) }); + } + } + if (run.endLine > i) { i = run.endLine; continue; } + } + + // Block tracking runs on the code AFTER any attributes on this line, so an + // inline "[Default] Interface X" still opens its block. + const body = run && run.endLine === i ? run.rest : lines[i]; + const code = blankStrings(decomment(body)); + if (CLOSE_RE.test(code)) { + const k = CLOSE_RE.exec(code)[1]; + if (!stack.length) problems.push({ file, pkg, line: i + 1, why: `End ${k} with nothing open` }); + else stack.pop(); + continue; + } + const o = OPEN_RE.exec(code); + if (o && !/^as$/i.test(o[2])) { + const kind = o[1][0].toUpperCase() + o[1].slice(1).toLowerCase(); + // An `Interface X` line inside a CoClass names one of the CoClass's + // interfaces; it has no body and no `End Interface`. Pushed as a block it + // ate the `End CoClass` that followed, leaving every CoClass in the + // corpus reported as unclosed -- 31 files. + const insideCoClass = stack.at(-1)?.kind === "Coclass"; + if (!(kind === "Interface" && insideCoClass)) { + stack.push({ kind, line: i + 1 }); + } + } + } + + if (stack.length) { + problems.push({ file, pkg, line: stack[0].line, why: `unclosed ${stack.map((s) => s.kind).join(">")} at end of file` }); + } + return { sites, problems }; +} + +// ------------------------------------------------------ documented attributes +function documentedAttributes() { + if (!existsSync(ATTR_DOC)) return null; + const out = new Map(); + const lines = readFileSync(ATTR_DOC, "utf8").replace(/\r\n?/g, "\n").split("\n"); + let cur = null; + lines.forEach((line, i) => { + const m = /^Syntax:\s*\*\*\[(\w+)/.exec(line); + if (m) { cur = { name: m[1], line: i + 1, app: null }; out.set(m[1], cur); return; } + const a = /^Applicable to:\s*(.*)$/.exec(line); + if (a && cur && cur.app === null) { + cur.app = a[1].replace(/\[\*\*([^\]]*)\*\*\]\([^)]*\)/g, "$1") + .replace(/\[([^\]]*)\]\([^)]*\)/g, "$1").replaceAll("**", "").replaceAll("\\", "").trim(); + } + }); + return out; +} + +// ------------------------------------------------------------------- report +const pct = (n, d) => (d ? ((n / d) * 100).toFixed(1) : "0.0"); + +function buildReport(sites, problems, files, projects, meta) { + const byAttr = new Map(); + for (const s of sites) { + if (!byAttr.has(s.attr)) byAttr.set(s.attr, []); + byAttr.get(s.attr).push(s); + } + const doc = documentedAttributes(); + + const rows = [...byAttr.entries()].map(([attr, ss]) => { + const targets = new Map(); + for (const s of ss) { + const k = `${s.container} / ${s.kind}`; + targets.set(k, (targets.get(k) ?? 0) + 1); + } + const pkgs = [...new Set(ss.map((s) => s.pkg))].sort(); + return { + attr, uses: ss.length, packages: pkgs, + withArgs: ss.filter((s) => s.hasArgs).length, + targets: [...targets.entries()].sort((a, b) => b[1] - a[1]), + documented: doc ? doc.has(attr) : null, + applicableTo: doc?.get(attr)?.app ?? null, + unresolved: ss.filter((s) => s.kind === "UNRESOLVED").length, + }; + }).sort((a, b) => b.uses - a.uses || a.attr.localeCompare(b.attr)); + + const usedNames = new Set(byAttr.keys()); + const undocumented = doc ? rows.filter((r) => !r.documented).map((r) => r.attr) : []; + const unused = doc ? [...doc.keys()].filter((n) => !usedNames.has(n)).sort() : []; + + return { meta, rows, undocumented, unused, problems, files, projects, sites: sites.length }; +} + +function renderMarkdown(rep) { + const L = []; + L.push(`# twinBASIC attribute census`); + L.push(""); + L.push(`Generated ${rep.meta.when} against **${rep.meta.install}** (BETA ${rep.meta.build}).`); + L.push(""); + L.push(`| | |`); + L.push(`|---|---|`); + L.push(`| projects scanned | ${rep.projects} |`); + L.push(`| \`.twin\` files | ${rep.files} |`); + L.push(`| attribute sites | ${rep.sites} |`); + L.push(`| distinct attributes | ${rep.rows.length} |`); + if (rep.meta.documentedCount != null) { + L.push(`| documented in \`Attributes.md\` | ${rep.meta.documentedCount} |`); + } + L.push(`| unresolved declarations | ${rep.problems.filter((p) => p.why === "declaration not classified").length} |`); + L.push(""); + L.push(`> A census says where an attribute **is** used, never where it **may** be used.`); + L.push(`> Use \`scripts/gen_attribute_probes.mjs\` with \`scripts/tbbuild.mjs\` for applicability.`); + L.push(""); + + L.push(`## Attributes by use`); + L.push(""); + L.push(`| Attribute | Uses | Pkgs | Documented | Where it is used (enclosing construct / declaration) |`); + L.push(`|---|---:|---:|:---:|---|`); + for (const r of rep.rows) { + const where = r.targets.map(([k, n]) => `${k} ×${n}`).join("
"); + const docMark = r.documented === null ? "--" : r.documented ? "yes" : "**no**"; + L.push(`| \`${r.attr}\` | ${r.uses} | ${r.packages.length} | ${docMark} | ${where} |`); + } + L.push(""); + + if (rep.undocumented.length) { + L.push(`## Used but not in \`Attributes.md\` (${rep.undocumented.length})`); + L.push(""); + L.push(`Each is an attribute the shipped packages use and the reference does not mention.`); + L.push(""); + for (const a of rep.undocumented) { + const r = rep.rows.find((x) => x.attr === a); + L.push(`- \`${a}\` --- ${r.uses} use(s) in ${r.packages.join(", ")}`); + } + L.push(""); + } + + if (rep.unused.length) { + L.push(`## Documented but unused in the packages (${rep.unused.length})`); + L.push(""); + L.push(`Not a defect: an attribute can be real, documented and simply not used by any`); + L.push(`shipped package. It does mean the census offers no evidence for its \`Applicable to:\``); + L.push(`line, so a probe is the only check available.`); + L.push(""); + L.push(rep.unused.map((a) => `\`${a}\``).join(", ")); + L.push(""); + } + + const byWhy = new Map(); + for (const p of rep.problems) { + if (!byWhy.has(p.why)) byWhy.set(p.why, []); + byWhy.get(p.why).push(p); + } + L.push(`## What the scanner could not resolve (${rep.problems.length})`); + L.push(""); + if (!rep.problems.length) { + L.push(`Nothing. Every attribute site resolved to a declaration, and every type block closed.`); + } else { + L.push(`Reported rather than bucketed silently: a census that hides its own confusion`); + L.push(`publishes a wrong number with no way to notice.`); + L.push(""); + for (const [why, ps] of [...byWhy.entries()].sort((a, b) => b[1].length - a[1].length)) { + L.push(`### ${why} (${ps.length})`); + L.push(""); + for (const p of ps.slice(0, 25)) { + L.push(`- \`${p.pkg}\` ${path.basename(p.file)}:${p.line}${p.text ? ` --- \`${p.text.replace(/`/g, "'")}\`` : ""}`); + } + if (ps.length > 25) L.push(`- ... and ${ps.length - 25} more`); + L.push(""); + } + } + return L.join("\n") + "\n"; +} + +function renderAttrDetail(rep, attr) { + const r = rep.rows.find((x) => x.attr.toLowerCase() === attr.toLowerCase()); + if (!r) return `No use of \`${attr}\` in the scanned packages.\n`; + const L = []; + L.push(`# \`[${r.attr}]\` --- census`); + L.push(""); + L.push(`${r.uses} use(s) across ${r.packages.length} package(s): ${r.packages.join(", ")}.`); + if (r.applicableTo) L.push(`\n\`Attributes.md\` says: **Applicable to:** ${r.applicableTo}`); + L.push(""); + L.push(`| Enclosing construct | Declaration it decorates | Uses |`); + L.push(`|---|---|---:|`); + for (const [k, n] of r.targets) { + const [c, kind] = k.split(" / "); + L.push(`| ${c} | ${kind} | ${n} |`); + } + L.push(""); + L.push(`> Evidence only. The compiler decides applicability -- see \`gen_attribute_probes.mjs\`.`); + return L.join("\n") + "\n"; +} + +// --------------------------------------------------------------------- main +function collectTwinFiles(dir, out = []) { + let entries; + try { entries = readdirSync(dir, { withFileTypes: true }); } catch { return out; } + for (const e of entries) { + const p = path.join(dir, e.name); + if (e.isDirectory()) collectTwinFiles(p, out); + else if (e.name.toLowerCase().endsWith(".twin")) out.push(p); + } + return out; +} + +function main() { + let projects, install = null, build = "n/a"; + const srcDir = opt("src"); + + if (srcDir) { + if (!existsSync(srcDir) || !statSync(srcDir).isDirectory()) die(2, `not a directory: ${srcDir}`); + install = srcDir; + projects = readdirSync(srcDir, { withFileTypes: true }) + .filter((e) => e.isDirectory()) + .flatMap((e) => { + const g = path.join(srcDir, e.name); + const inner = readdirSync(g, { withFileTypes: true }).filter((x) => x.isDirectory()); + return inner.length + ? inner.map((x) => ({ name: x.name, dir: path.join(g, x.name) })) + : [{ name: e.name, dir: g }]; + }); + if (!projects.length) projects = [{ name: path.basename(srcDir), dir: srcDir }]; + } else { + install = findInstall(); + build = buildNumberOf(install); + log(`install : ${install}`); + const cache = opt("cache", path.join(os.tmpdir(), "tb-census", `beta-${build}`)); + log(`cache : ${cache}`); + projects = exportAll(install, cache, flag("samples")); + } + + const allSites = [], allProblems = []; + let fileCount = 0; + for (const p of projects) { + const pkg = p.name.replace(/^\.?\{[^}]+\}_/, ""); + for (const f of collectTwinFiles(p.dir)) { + fileCount++; + try { + const { sites, problems } = scanFile(f, pkg); + allSites.push(...sites); + allProblems.push(...problems); + } catch (e) { + allProblems.push({ file: f, pkg, line: 0, why: "scanner threw", text: String(e.message) }); + } + } + } + log(`scanned : ${fileCount} .twin files, ${allSites.length} attribute sites`); + + const doc = documentedAttributes(); + const rep = buildReport(allSites, allProblems, fileCount, projects.length, { + when: new Date().toISOString().slice(0, 10), + install, build, + documentedCount: doc ? doc.size : null, + }); + + const attr = opt("attr"); + const text = flag("json") + ? JSON.stringify(attr ? rep.rows.find((r) => r.attr.toLowerCase() === attr.toLowerCase()) ?? null : rep, null, 2) + "\n" + : attr ? renderAttrDetail(rep, attr) : renderMarkdown(rep); + + // Every raw site, for answering "which file produced this row?" -- the + // question every surprising number in the report turns into. + const dump = opt("dump-sites"); + if (dump) { + writeFileSync(dump, JSON.stringify(attr + ? allSites.filter((s) => s.attr.toLowerCase() === attr.toLowerCase()) + : allSites, null, 1), "utf8"); + log(`sites : ${dump}`); + } + + const out = opt("out"); + if (out) { writeFileSync(out, text, "utf8"); log(`report : ${out}`); } + else process.stdout.write(text); +} + +main(); diff --git a/builder/page-baseline.json b/builder/page-baseline.json index 0aab8dc8..7050e4ad 100644 --- a/builder/page-baseline.json +++ b/builder/page-baseline.json @@ -1,5 +1,5 @@ { "src": "docs", - "pages": 910, - "staticFiles": 247 + "pages": 912, + "staticFiles": 248 } diff --git a/docs/Documentation/Authoring.md b/docs/Documentation/Authoring.md index 84690acf..be8b0464 100644 --- a/docs/Documentation/Authoring.md +++ b/docs/Documentation/Authoring.md @@ -362,6 +362,7 @@ could not be derived from build state at all. - **Bold** (`**...**`) for keywords and literal tokens the reader would type verbatim; *italic* (`*...*`) for placeholders and argument names. - twinBASIC code goes in a ` ```tb ` fenced block --- Shiki highlights it with the vendored twinBASIC grammar, and `twinbasic`, `vb` and `vba` select the same grammar. The other highlighted fence languages are `js`, `yaml`, `json`, `c`, `html`, `xml`, `sql` and `batch`. Anything else renders as unhighlighted plain text: the build does not fail, but it prints `highlight: unknown fence language ""` naming the language and the list to add it to, so check the build output rather than the page. - Parameter lists use the definition-list pattern (a term line, then a `: definition` line beneath it), not a markdown table. +- **An inline code span cannot start or end with a single space.** CommonMark removes one space from each end of a span whose content is not all spaces, so `` ` 1 2 ` `` renders as `1 2`. That matters whenever the padding *is* the value being shown --- what `Debug.Print` emits into its 14-column print zones, a fixed-width return such as [Partition](../tB/Modules/Interaction/Partition)'s, anything a reader might count characters in. Write two spaces at each end to get one, and confirm against the built HTML rather than the preview: five claims on two pages were quietly de-padded this way, each of them a documented output value with its leading and trailing spaces missing. - For dashes, write `--` in the source (it renders as an en-dash) or `---` (an em-dash). Never paste a literal `–` or `—`. Nothing in the build rejects one: the typographer converts the ASCII forms and passes a literal character straight through, so a stray dash ships silently and only the source becomes inconsistent. `scripts/convert_em_dash_separators.mjs` is the normaliser, and it is run by hand. ### A code sample that holds a fence marker diff --git a/docs/Documentation/Tools.md b/docs/Documentation/Tools.md index 132326d6..998c5d41 100644 --- a/docs/Documentation/Tools.md +++ b/docs/Documentation/Tools.md @@ -8,14 +8,14 @@ permalink: /Documentation/Development/Tools # Tools and Scripts {: .no_toc } -One-line-per-tool reference for every executable in the documentation repository: the five Windows batch wrappers at the repository root, the Node and Python scripts under `scripts/` (cross-platform except for [`tbbuild.mjs`](#tbbuild), which drives the twinBASIC IDE), the `tbdocs` orchestrator and its CLI flags, and the PDF render driver. If you are looking for the day-to-day workflow rather than a cheat sheet, the [Building and Deployment](Building) page is the gentler read; if you are modifying the build pipeline itself, the [tbdocs Internals](Builder) page goes one level deeper. +One-line-per-tool reference for every executable in the documentation repository: the five Windows batch wrappers at the repository root, the Node and Python scripts under `scripts/` (cross-platform except for [`tbbuild.mjs`](#tbbuild), which drives the twinBASIC IDE), the `tbdocs` orchestrator and its CLI flags, [`census_attributes.mjs`](#census-attributes) under `builder/`, and the PDF render driver. If you are looking for the day-to-day workflow rather than a cheat sheet, the [Building and Deployment](Building) page is the gentler read; if you are modifying the build pipeline itself, the [tbdocs Internals](Builder) page goes one level deeper. * TOC goes here {:toc} ## Batch wrappers at the repository root {: #batch-wrappers } -All five sit at the repository root, beside `package.json` --- not under `docs/`. Each uses `@pushd "%~dp0"` to run from that root regardless of where it is invoked from, and each entry below gives the POSIX equivalent of what it runs. Those equivalents have no `pushd` in front of them, so **run them from the repository root** --- `tbdocs`'s `--src docs`, [`check_publish_policy.mjs`](#check-publish-policy)'s default source root, and every path handed to [`render-book.mjs`](#bookrender-bookmjs) are all resolved against the working directory. The only other Windows-specific tool is [`scripts/tbbuild.mjs`](#tbbuild), which drives the twinBASIC IDE and is no part of the site build. Nothing else in the repository is: `tbdocs` and every gate in both wrappers is a Node script, and CI runs all of them on `ubuntu-latest` except [`check_tree_fresh.mjs`](#check-tree-fresh), which guards against a failure mode CI cannot have. +All five sit at the repository root, beside `package.json` --- not under `docs/`. Each uses `@pushd "%~dp0"` to run from that root regardless of where it is invoked from, and each entry below gives the POSIX equivalent of what it runs. Those equivalents have no `pushd` in front of them, so **run them from the repository root** --- `tbdocs`'s `--src docs`, [`check_publish_policy.mjs`](#check-publish-policy)'s default source root, and every path handed to [`render-book.mjs`](#bookrender-bookmjs) are all resolved against the working directory. Three other tools are Windows-specific, and none is part of the site build: [`scripts/tbbuild.mjs`](#tbbuild) and [`scripts/tbrun.mjs`](#tbrun), which drive the twinBASIC IDE, and [`census_attributes.mjs`](#census-attributes), which runs the twinBASIC compiler's `export` verb --- though that one is cross-platform when given an already-exported tree with `--src`. Nothing else in the repository is: `tbdocs` and every gate in both wrappers is a Node script, and CI runs all of them on `ubuntu-latest` except [`check_tree_fresh.mjs`](#check-tree-fresh), which guards against a failure mode CI cannot have. ### build.bat @@ -514,7 +514,8 @@ Two files under `scripts/lib/` belong to it and are never run directly. `tb-cdp. {: #tbrun } node scripts/tbrun.mjs [--port N] [--timeout S] [--quiet MS] - [--json] [--raw] [--keep] [--show|--hide] + [--json] [--raw] [--keep] [--no-reap] + [--reap-images a,b] [--show|--hide] Builds a probe project and captures what it writes to the IDE's [Debug Console](../../tB/IDE/Project/DebugConsole). Where [`tbbuild.mjs`](#tbbuild) answers @@ -545,19 +546,39 @@ build log, and the linker writes there *after* the build, so a probe that does n first comes back interleaved with `[LINKER]` lines. The script warns when a probe omits it, and warns again when there is no `[RunAfterBuild]` at all. +**The capture is complete however much a probe prints**, so there is no reason to keep one +short. `tbrun` reads the console's backing array rather than the pane, which is a virtualised +list view holding only the rows that fit --- reading that instead returns the last ten or so +lines of a long probe and looks no different from a full capture. `Debug.Cls` is what empties +the array, which is the other reason to begin with it. + | Flag | Effect | |---|---| -| `--port ` | DevTools port for the IDE. Default 9346. Distinct ports let probes run concurrently. | +| `--port ` | DevTools port for the IDE. Default 9346. Distinct ports let probes run concurrently --- the staging directory and the project id are keyed to it, so two runs never share a workspace. | | `--timeout ` | Give up waiting for console output. Default 120. | -| `--quiet ` | How long the console must stop changing before the output counts as complete. Default 2500. There is no sentinel string to match, so any probe works without telling the script anything. | +| `--quiet ` | How long the console must stop changing before the output counts as complete. Default 2500. There is no sentinel string to match, so any probe works without telling the script anything. Raise it well above the default for a probe that drives an out-of-process server, which can take longer than that to start. | | `--raw` | Keep the console's timestamp column, which is otherwise stripped. | -| `--json` | One object with the built exe's path and the captured lines. | -| `--keep` | Leave the IDE running. | +| `--json` | One object with the built exe's path, the captured lines, the IDE pid and anything reaped. | +| `--keep` | Leave the IDE running. Implies `--no-reap`. | +| `--no-reap` | Do not harvest automation servers the probe left behind. | +| `--reap-images ` | Replace the harvested image list. Default is the Office suite. | | `--show` / `--hide` | Passed through to `tbbuild.mjs`. | Exit codes: **0** captured output, **1** the project has compile errors (the diagnostics are printed), **2** the harness failed, **3** nothing reached the console before the timeout. +**A probe that activates a COM server can leak one per run.** `CreateObject("Excel.Application")` +is activated by DCOM, so the `EXCEL.EXE` that appears is a child of `svchost.exe` rather than +of anything the harness started --- no tree kill reaches it. Each activation is its own +process, so they accumulate, and calling `Quit` is not enough: the process exits only once +every COM reference has been released. `tbrun` therefore takes a process snapshot before it +starts the IDE and harvests what appeared afterwards, subject to three conditions --- the +process must be new, its image must be on the reap list, and it must have no window open. +Anything new and on the list but *windowed* is reported and left alone, because that is +indistinguishable from a copy the user opened. Two concurrent runs both driving Excel cannot +tell their servers apart, so whichever finishes first harvests both: pass `--no-reap` there +and sweep once at the end. + > [!IMPORTANT] > The one trap worth knowing even if you never read the script: a project whose > `project.buildPath` is still the default `${SourcePath}\Build\...` template opens a native @@ -589,6 +610,35 @@ It also writes a key naming the `Attributes.md` line each probe came from, besid **That command's exit code is `0` whether it worked or not**, so a script that packs a tree and then builds it will happily compile the previous `.twinproj`. Test the last line of its output for `... DONE` instead; [Import/Export Tool](../../Features/Packages/Import-Export-Tool#the-exit-code-is-always-zero) has the caveat in full and a batch-file form of the test. That page also covers why this verb runs opposite to the standalone scripts' `import`. Re-run the generator after editing `Attributes.md`. Exits 0, or 2 with usage when given no output directory. +### census_attributes.mjs +{: #census-attributes } + + node builder/census_attributes.mjs [--ide ] [--src ] [--cache ] + [--refresh] [--samples] [--attr ] + [--json] [--out ] [--dump-sites ] [--quiet] + +Reports, for every attribute the twinBASIC packages use, **which enclosing construct and which kind of declaration it decorates**. It exports each package of an IDE install with the compiler's own `export` verb, scans the `.twin` sources, and writes a Markdown or JSON report. No arguments are needed: it finds the newest `twinBASIC_IDE_BETA_*` the same way [`tbbuild.mjs`](#tbbuild) does, caches the export under the build number, and reuses it on later runs. It is not part of the site build and nothing calls it during one. + +Against BETA 983 that is 619 files, 9,673 attribute sites and 55 distinct attributes. + +**A census is evidence, not applicability.** It says where an attribute *is* used, never where it *may* be used, and the two differ in both directions. The packages contain no use of `[Hidden]` on a whole **Class**, yet the compiler accepts one; they contain many on **Class** and **Interface** members, and the compiler refuses the same attribute on the **Interface** lines inside a **CoClass**. Neither fact is reachable from the other tool, so pair this with [`gen_attribute_probes.mjs`](#gen-attribute-probes) and [`tbbuild.mjs`](#tbbuild), which ask the compiler directly. + +Grouping is by enclosing construct *and* declaration keyword, because the keyword alone misleads. An earlier hand-written census of `[RedirectToStaticImplementation]` grouped its 82 uses by keyword, reported "a Property Get, a Function and a Sub", and produced the claim *procedure in a Class* --- which the compiler rejected with TB5155, because every one of those uses is inside an **Interface**. + +| Flag | Effect | +|---|---| +| `--ide ` | The install root to census. Defaults to `$TB_IDE`, else the newest `twinBASIC_IDE_BETA_*` on the Desktop. | +| `--src ` | Census an already-exported tree and skip the export entirely. | +| `--cache ` | Where exports are kept. Defaults to a per-build folder under the system temp directory. | +| `--refresh` | Re-export even when the cache already holds this build. | +| `--samples` | Also census `projects/` and `addins/`, not only `packages/`. | +| `--attr ` | Report one attribute in detail instead of the whole table. | +| `--dump-sites ` | Write every raw site as JSON --- which file and line produced each row. | +| `--json` | Emit JSON instead of Markdown. | +| `--out ` | Write to a file instead of standard output. | + +The report ends with what the scanner could not resolve, and **that section is expected to be empty**. A census that quietly buckets its own confusion publishes a wrong number with nothing to notice it by, so an unresolved site is reported as a scanner bug rather than absorbed. Reaching zero took handling several things this corpus does that a simpler sweep gets wrong: attributes spanning lines (`[Description("..." & vbCrLf & _` accounts for 3.8% of all attribute lines), comma-separated lists, arguments containing commas, escaped identifiers that look exactly like attributes (`[_HiddenModule].Foo`, and Enum members genuinely named `[A4 Portrait]`), comments in four different positions, and block-tracking traps such as a UDT field called `Type As Long` or a module named `[_HiddenModule]`. Exits 0 once a report is produced, or 2 if no install or source tree can be found. + ### scripts/impexp.mjs and scripts/impexp.py {: #impexp } diff --git a/docs/Features/Language/Pointers.md b/docs/Features/Language/Pointers.md index ad9e3181..fe782ad4 100644 --- a/docs/Features/Language/Pointers.md +++ b/docs/Features/Language/Pointers.md @@ -83,7 +83,7 @@ Sub test1(ByVal ptr As LongPtr) End Sub ``` -This will print `1 2`. +This will print ` 1 2 ` --- a comma moves to the next 14-column [print zone](../../tB/Modules/Debug#print), and a positive number carries a leading space where its sign would be. ```tb Sub call2() @@ -101,7 +101,7 @@ Sub test2(b As bar) End Sub ``` -This will print `3 4`. +This will print ` 3 4 `. ```tb Sub call3() @@ -117,7 +117,7 @@ Sub test3(b As bar) End Sub ``` -This will print `4`. Free standing use and nesting is also allowed; the above will print `4`. While the examples here are local code only, this is particularly useful for APIs, where you're forced to work with pointers extensively. +This will print ` 4 `. Free standing use and nesting is also allowed; the above will print ` 4 `. While the examples here are local code only, this is particularly useful for APIs, where you're forced to work with pointers extensively. ## Len/LenB(Of \) Support diff --git a/docs/Features/Packages/Images/LibrarySymbols.png b/docs/Features/Packages/Images/LibrarySymbols.png new file mode 100644 index 00000000..c3d76bfb Binary files /dev/null and b/docs/Features/Packages/Images/LibrarySymbols.png differ diff --git a/docs/Features/Packages/Library symbols.md b/docs/Features/Packages/Library symbols.md new file mode 100644 index 00000000..34010665 --- /dev/null +++ b/docs/Features/Packages/Library symbols.md @@ -0,0 +1,46 @@ +--- +title: Library Symbols +parent: Package Management +grand_parent: Features +nav_order: 8 +permalink: /Features/Packages/Library-Symbols +--- + +# Library symbols + +Every library a project references contributes its components under a *library symbol* --- the name that qualifies them in code. The VBA compatibility package contributes under `VBA`, so `VBA.Strings.Left` names the function; OLE Automation contributes under `stdole`, so `stdole.StdFont` names the class. A symbol is only needed where a name would otherwise be ambiguous, which is why most code never writes one. + +The symbol is a property of the *reference*, not of the library, so a project can change it. + +## Changing a symbol + +Open *Project Settings*, find **Library References**, and select the **Enabled Libraries** tab. The **Library Symbol** column shows the symbol each library currently contributes under, and a pencil icon beside each one opens it for editing. + +![The Project Settings dialog, Library References, Enabled Libraries tab. A Name column lists the VBA, VBRUN, OLE Automation and VB libraries, each with a tick box; a Library Symbol column shows VBA, VBRUN, stdole and VB, each followed by a small pencil icon; a Version column follows. The VB row's symbol reads VB struck through with *MyVB on the line beneath it.](Images/LibrarySymbols.png) + +A symbol that has been changed is shown as the original struck through, with the replacement beneath it --- the `VB` / `*MyVB` pair in the picture above. That is display only: in code the library is `MyVB`, and `VB` no longer names anything. + +## Exposing a library's private symbols + +A package's top-level components can be declared **Private**, which keeps them internal to the package. Prefixing the library symbol with an asterisk makes them visible to the referencing project as well. + +The asterisk is an instruction rather than part of the name, and is stripped from the symbol. A library set to `*VB` is still written `VB` in code; one set to `*MyVB` is written `MyVB`. + +| Library symbol | Public components | Private components | +|----------------|-------------------|--------------------| +| `VB` | `VB.Form` | not reachable | +| `*VB` | `VB.Form` | `VB.IVBPrint` | +| `*MyVB` | `MyVB.Form` | `MyVB.IVBPrint` | + +Without the asterisk a private component does not resolve, and the compiler reports *TB5079 Unrecognized datatype symbol*. The name always has to be qualified: exposing the private symbols does not put them in scope unqualified. + +> [!NOTE] +> A component is **Private** because the package author did not intend it to be part of the package's interface, so it is internal and may change between releases. That is usually a reason to be deliberate about it rather than a reason not to: a private component may be exactly the right thing to use, and a package author who changes one will say so. + +[**IVBPrint**](../../tB/Packages/VB/IVBPrint) in the VB package is a worked example: it is the interface the [**Print**](../../tB/Core/Print) statement dispatches through, and implementing it in a class of your own is what makes that class a valid **Print** target. + +## See Also + +- [Importing a Package from a TWINPACK File](Importing-TWINPACK) -- installing a package from a local file +- [Import/Export Tool](Import-Export-Tool) -- reading a project's settings outside the IDE +- [**IVBPrint**](../../tB/Packages/VB/IVBPrint) interface -- a private component a project may legitimately want diff --git a/docs/Features/Packages/index.md b/docs/Features/Packages/index.md index 4c218ddc..e5d6c6a5 100644 --- a/docs/Features/Packages/index.md +++ b/docs/Features/Packages/index.md @@ -29,5 +29,6 @@ Please be aware that TWINPACK files currently contain the full source code of yo - [Updating a Package](Updating) -- removing an outdated package and installing a newer version from TWINSERV. - [TWINPACK File Format](File-Format) -- binary format specification for `.twinproj` and `.twinpack` files. - [Import/Export Tool](Import-Export-Tool) -- unpacking and repacking `.twinproj` and `.twinpack` files from the command line, with the compiler executable or with a standalone script. +- [Library Symbols](Library-Symbols) -- the name a referenced library's components are qualified with, how to change it, and how to expose a package's private components. [^1]: A service of TWINBASIC LTD offered to the user community. diff --git a/docs/Features/Standard-Library/New-Functions.md b/docs/Features/Standard-Library/New-Functions.md index 64d96cc0..585a3999 100644 --- a/docs/Features/Standard-Library/New-Functions.md +++ b/docs/Features/Standard-Library/New-Functions.md @@ -78,7 +78,7 @@ Array(a, b, c) = d Debug.Print a, b, c ``` -This would print `1 2 3` --- a comma starts each value in the next 14-column [print zone](../../tB/Modules/Debug#print). You could also assign multiple variables at once like this and get the same result: +This would print ` 1 2 3 ` --- a comma starts each value in the next 14-column [print zone](../../tB/Modules/Debug#print), and a positive number carries a leading space where its sign would be, so the values begin at columns 1, 15, and 29 rather than 0, 14, and 28. You could also assign multiple variables at once like this and get the same result: ```tb Dim a As Long, b As Long, c As Long @@ -95,4 +95,4 @@ Dim c() As Long = Array(a, b) Debug.Print c(1), UBound(c) ``` -Which prints `7 1`. +Which prints ` 7 1 `. diff --git a/docs/Reference/Attributes.md b/docs/Reference/Attributes.md index 12d899cf..c2f02456 100644 --- a/docs/Reference/Attributes.md +++ b/docs/Reference/Attributes.md @@ -574,9 +574,28 @@ Applicable to: [**Class**](Class) Syntax: **[Hidden** [ **(** **True** \| **False** **)** ] **]** -Applicable to: [**Class**](Class), [**CoClass**](CoClass), [**Interface**](Interface) +Applicable to: [**Class**](Class), [**CoClass**](CoClass), [**Interface**](Interface), [**Module**](Module), a [procedure](../Gloss#procedure) in a Class or Module, a procedure in an Interface, a variable in a Class, and an [**Enum**](Enum) member + +Hides the declaration from certain IntelliSense and other lists. It applies to a whole type --- a **Class**, **CoClass**, **Interface** or **Module** --- and equally to a single member of one, so a member can be kept out of those lists without hiding the type that declares it. Within a **Class** that covers procedures, variables, constants and events; within an **Interface**, the member prototypes; within a **Module**, procedures, variables, constants and [**Declare**](Declare) statements. + +> [!NOTE] +> A **CoClass** can only be hidden whole. Its body holds nothing but **Interface** lines, and the attribute is refused there with TB5155 --- unlike [**Default**](#default) and [**Source**](#source), which are interface-line attributes. It is likewise refused on an **Enum** or [**Type**](Type) declaration, on a **Type** member, and on a procedure parameter, though an individual **Enum** *member* does accept it. + + -Hides the interface or class from certain Intellisense and other lists. ## IdeButton (String) {: #idebutton } @@ -732,7 +751,7 @@ Applicable to: variables and [procedures](../Gloss#procedure) in a [**Class**](C Keeps a member out of the surfaces that list a class's members, while leaving it callable. The twinBASIC packages apply it to members that exist for the framework's own use, such as `InternalSectionId` and `hWndHeader`. -This is distinct from [Hidden](#hidden), which applies to a whole type, and from [Restricted](#restricted). +This is distinct from [Hidden](#hidden), which reaches the same member as well as the whole type, and from [Restricted](#restricted). -### See Also - -- [ChDrive](ChDrive), [MkDir](MkDir), [RmDir](RmDir) statements -- [CurDir](CurDir), [Dir](Dir) functions - ### Example This example uses the **ChDir** statement to change the current directory or folder. @@ -55,4 +50,9 @@ ChDir "MYDIR" ' Assume "C:" is the current drive. The following statement changes ' the default directory on drive "D:". "C:" remains the current drive. ChDir "D:\WINDOWS\SYSTEM" -``` \ No newline at end of file +``` + +### See Also + +- [ChDrive](ChDrive), [MkDir](MkDir), [RmDir](RmDir) statements +- [CurDir](CurDir), [Dir](Dir) functions diff --git a/docs/Reference/Default/VBA/FileSystem/EOF.md b/docs/Reference/Default/VBA/FileSystem/EOF.md index 6c41eb9e..fb4a2358 100644 --- a/docs/Reference/Default/VBA/FileSystem/EOF.md +++ b/docs/Reference/Default/VBA/FileSystem/EOF.md @@ -33,7 +33,7 @@ Dim InputData Open "MYFILE" For Input As #1 ' Open file for input. Do While Not EOF(1) ' Check for end of file. Line Input #1, InputData ' Read line of data. - Debug.Print InputData ' Print to the Immediate window. + Debug.Print InputData ' Print to the Debug Console. Loop Close #1 ' Close file. ``` diff --git a/docs/Reference/Default/VBA/FileSystem/Input.md b/docs/Reference/Default/VBA/FileSystem/Input.md index 748c0852..e101aacb 100644 --- a/docs/Reference/Default/VBA/FileSystem/Input.md +++ b/docs/Reference/Default/VBA/FileSystem/Input.md @@ -32,14 +32,14 @@ For files opened for **Binary** access, an attempt to read through the file usin ### Example -This example uses the **Input** function to read one character at a time from a file and print it to the immediate window. *TESTFILE* is assumed to be a text file with a few lines of sample data. +This example uses the **Input** function to read one character at a time from a file and print it to the [Debug Console](../../IDE/Project/DebugConsole). *TESTFILE* is assumed to be a text file with a few lines of sample data. ```tb Dim MyChar As Variant Open "TESTFILE" For Input As #1 ' Open file. Do While Not EOF(1) ' Loop until end of file. MyChar = Input(1, #1) ' Get one character. - Debug.Print MyChar ' Print to the immediate window. + Debug.Print MyChar ' Print to the Debug Console. Loop Close #1 ' Close file. ``` diff --git a/docs/Reference/Default/VBA/Interaction/AppActivate.md b/docs/Reference/Default/VBA/Interaction/AppActivate.md index 3f759dc3..ae1d58a3 100644 --- a/docs/Reference/Default/VBA/Interaction/AppActivate.md +++ b/docs/Reference/Default/VBA/Interaction/AppActivate.md @@ -31,23 +31,27 @@ In determining which application to activate, *title* is compared to the title s ### Example -This example illustrates various uses of the **AppActivate** statement to activate an application window. The **Shell** statements assume the applications are in the paths specified. +This example illustrates various uses of the **AppActivate** statement to activate an application window. [**Shell**](Shell) needs a path it can pass straight to `CreateProcess`: it does not search the registry's *App Paths* key, so a bare `"WINWORD.EXE"` fails even though Office registers it there. The paths below assume a Click-to-Run installation of Office 16 --- an MSI installation has no `root` folder, and the version segment tracks the Office release. ```tb -Dim MyAppID, ReturnValue -AppActivate "Microsoft Word" ' Activate Microsoft - ' Word. +Dim MyAppID As Double, ReturnValue As Double + +' Activate by window title. No path is involved. +AppActivate "Microsoft Word" + +' ProgramW6432 is the 64-bit Program Files folder even in a 32-bit build, +' which is where a 64-bit Office lives. Environ$("ProgramFiles") would +' return the "(x86)" tree instead. +Dim Office As String +Office = Environ$("ProgramW6432") & "\Microsoft Office\root\Office16\" ' AppActivate can also use the return value of the Shell function. -MyAppID = Shell("C:\WORD\WINWORD.EXE", 1) ' Run Microsoft Word. -AppActivate MyAppID ' Activate Microsoft - ' Word. - -' You can also use the return value of the Shell function. -ReturnValue = Shell("c:\EXCEL\EXCEL.EXE",1) ' Run Microsoft Excel. -AppActivate ReturnValue ' Activate Microsoft - ' Excel. +MyAppID = Shell(Office & "WINWORD.EXE", vbNormalFocus) +AppActivate MyAppID + +ReturnValue = Shell(Office & "EXCEL.EXE", vbNormalFocus) +AppActivate ReturnValue ``` ### See Also diff --git a/docs/Reference/Default/VBA/Interaction/Partition.md b/docs/Reference/Default/VBA/Interaction/Partition.md index ec8544bf..967673a8 100644 --- a/docs/Reference/Default/VBA/Interaction/Partition.md +++ b/docs/Reference/Default/VBA/Interaction/Partition.md @@ -45,11 +45,20 @@ Any argument may be a decimal value, but is rounded to the nearest even integer ### Example -This example uses **Partition** in an SQL `SELECT` to count the orders whose freight cost falls into each of several ranges. With *start* = 0, *stop* = 500, *interval* = 50, the first range is `" 0: 49"`, and so on up to 500. - -```sql -SELECT DISTINCTROW Partition([Freight], 0, 500, 50) AS Range, - Count(Orders.Freight) AS [Count] -FROM Orders -GROUP BY Partition([Freight], 0, 500, 50); +This example passes a series of freight costs to **Partition** with *start* = 0, *stop* = 500, and *interval* = 50, and prints the range each one falls into. The first range is `" 0: 49"`, and so on up to 500. Note that 12.5 is rounded before the range is chosen, and that a value below *start* or above *stop* gets one of the open-ended forms. + +```tb +Dim Freight As Variant +For Each Freight In Array(-1, 12.5, 49, 50, 275, 499, 500, 501) + Debug.Print Freight & " -> [" & Partition(Freight, 0, 500, 50) & "]" +Next Freight + +' -1 -> [ : -1] +' 12.5 -> [ 0: 49] +' 49 -> [ 0: 49] +' 50 -> [ 50: 99] +' 275 -> [250:299] +' 499 -> [450:499] +' 500 -> [500:500] +' 501 -> [501: ] ``` diff --git a/docs/Reference/Default/VBA/Interaction/SaveSetting.md b/docs/Reference/Default/VBA/Interaction/SaveSetting.md index c07ddd62..2ceb1c19 100644 --- a/docs/Reference/Default/VBA/Interaction/SaveSetting.md +++ b/docs/Reference/Default/VBA/Interaction/SaveSetting.md @@ -16,7 +16,7 @@ Syntax: **SaveSetting** *appname*, *section*, *key*, *setting* *appname* -: String expression containing the name of the application or project to which the setting applies. On the Macintosh, this is the filename of the initialization file in the Preferences folder in the System folder. +: String expression containing the name of the application or project to which the setting applies. *section* diff --git a/docs/Reference/Default/VBA/Strings/Format.md b/docs/Reference/Default/VBA/Strings/Format.md index 71160e17..fa8b61c0 100644 --- a/docs/Reference/Default/VBA/Strings/Format.md +++ b/docs/Reference/Default/VBA/Strings/Format.md @@ -163,7 +163,7 @@ The following table identifies the predefined date and time format names. |-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **General Date** | Display a date and/or time, for example, 4/3/93 05:34 PM. If there is no fractional part, display only a date, for example, 4/3/93. If there is no integer part, display time only, for example, 05:34 PM. Date display is determined by the system settings. | | **Long Date** | Display a date according to the system long date format. | -| **Medium Date** | Display a date using the medium date format appropriate for the language version of the host application. | +| **Medium Date** | Display a date in the abbreviated `dd-mmm-yy` form, for example, 03-Feb-93. | | **Short Date** | Display a date using the system short date format. | | **Long Time** | Display a time using the system long time format; includes hours, minutes, seconds. | | **Medium Time** | Display time in 12-hour format using hours and minutes and the AM/PM designator. | diff --git a/docs/Reference/Statements.md b/docs/Reference/Statements.md index aeefcbcb..b7c3b658 100644 --- a/docs/Reference/Statements.md +++ b/docs/Reference/Statements.md @@ -100,7 +100,7 @@ These statements are built into the language itself. They are understood by the * [ParamArray](../tB/Core/ParamArray) -- declares the final parameter of a procedure as an arbitrary-arity list of arguments -* [Print #](../tB/Core/Print) -- writes display-formatted data to a sequential file +* [Print](../tB/Core/Print) -- writes display-formatted data to a file, the Debug Console, or a drawing surface * [Private](../tB/Core/Private) -- declares module-level variables accessible only within the declaring module diff --git a/scripts/check_tree_fresh.mjs b/scripts/check_tree_fresh.mjs index 63ec806b..1207496a 100644 --- a/scripts/check_tree_fresh.mjs +++ b/scripts/check_tree_fresh.mjs @@ -46,7 +46,13 @@ const IGNORED_DIRS = new Set([ // page-baseline.json is written by the drift guard whenever the page count // rises (builder/page-baseline.mjs), so `build.bat && check.bat` would have // failed on the next run after any page addition. -const IGNORED_FILES = new Set(["page-baseline.json"]); +// census_attributes.mjs lives under builder/ but decides none of the built +// bytes -- it censuses twinBASIC package sources and never runs during a build. +// Without this, editing it marks every output tree stale, which blocks check.bat +// and makes book.bat refuse to render, for a tool the build never calls. Same +// reasoning as page-baseline.json: the sources this script watches are "the +// inputs that decide the built bytes", and neither file is one. +const IGNORED_FILES = new Set(["page-baseline.json", "census_attributes.mjs"]); // The inputs that decide the built bytes. The source tree is the obvious // one; the builder and the theme sources matter just as much, and are diff --git a/scripts/gen_attribute_probes.mjs b/scripts/gen_attribute_probes.mjs index 76f14f8f..bd3aac77 100644 --- a/scripts/gen_attribute_probes.mjs +++ b/scripts/gen_attribute_probes.mjs @@ -938,6 +938,10 @@ const RULES = [ [/variable/i, ["VAR_MODULE"]], [/declare|api\s+declaration/i, ["DECLARE"]], [/^type\b/i, ["TYPE"]], + // An Enum MEMBER is a different target from the Enum, and must win over the + // bare rule below. It is not ^-anchored because the phrase carries an article + // where it appears -- "and an Enum member" -- which a ^ rule cannot reach. + [/enum\s+member/i, ["ENUM_MEMBER"]], [/^enum\b/i, ["ENUM"]], // "Const", but also "constants in a module." -- \b after "const" fails on // the plural, which silently dropped a target until it was noticed. @@ -1046,6 +1050,13 @@ function render(target, tag, attr, needsHintEnum, idx) { } return `Public Module ${tag}\n ${attr}\n Public Enum ProbeEnum${pad(idx, 3)}\n` + ` ProbeValue${pad(idx, 3)} = 1\n End Enum\nEnd Module\n`; + // An Enum MEMBER, not the Enum itself. The two cannot share a probe: + // [Hidden] is accepted on a member and refused on the Enum with TB5155, + // so one skeleton would answer for both and get one of them wrong. + // Core/Open documents the [Hidden, Restricted] pair on exactly this target. + case "ENUM_MEMBER": + return `Public Module ${tag}\n Public Enum ProbeEnum${pad(idx, 3)}\n` + + ` ${attr}\n ProbeValue${pad(idx, 3)} = 1\n End Enum\nEnd Module\n`; case "CONST": return `Public Module ${tag}\n${hint} ${attr}\n Public Const ProbeConst As Long = 1\n` + "End Module\n"; @@ -1068,7 +1079,8 @@ const HUMAN = { PROC_CLASS: "on a Sub in a Class", PROC_INTERFACE: "on a prototype in an Interface", DECLARE: "on a Declare", TYPE: "on a Type (UDT)", ENUM: "on an Enum", - CONST: "on a Const", VAR_CLASS: "on a variable in a Class", + CONST: "on a Const", ENUM_MEMBER: "on an Enum member", + VAR_CLASS: "on a variable in a Class", VAR_MODULE: "on a variable in a Module", PARAM: "on a procedure parameter", LIBRARY_INTERFACE: "on an Interface in a Library", FUNC_MODULE_BOOL: "on a Boolean Function in a Module", diff --git a/scripts/tbbuild.mjs b/scripts/tbbuild.mjs index 6d1023a9..05c35fea 100644 --- a/scripts/tbbuild.mjs +++ b/scripts/tbbuild.mjs @@ -8,7 +8,11 @@ // --port DevTools port to start the IDE on (default 9333) // --timeout give up waiting for the compile (default 180) // --json emit one JSON object instead of text -// --keep leave the IDE running afterwards +// --keep leave the IDE running afterwards. The IDE's pid is +// then printed as `ide-pid: N` (and is always in --json +// as `idePid`), because whoever inherits a kept IDE has +// to be able to end that one rather than every IDE on +// the machine. // --show / --hide put the IDE on your desktop where you can watch it, // or on a private one where it cannot take focus. // Default: hidden, unless TBBUILD_SHOW is set -- @@ -39,7 +43,7 @@ // them. See WIP.md, "Compiling a twinBASIC project without the IDE in front // of you". import { spawn, execFileSync } from "node:child_process"; -import { existsSync, readdirSync, readFileSync } from "node:fs"; +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { attach } from "./lib/tb-cdp.mjs"; @@ -89,6 +93,29 @@ if (!proj || flag("help")) { "[--show|--hide]"); process.exit(2); } +// Refuse anything that is not a .twinproj, rather than discovering it two +// minutes later. A source directory is the tempting mistake -- it is what +// `tbrun` takes -- and handing one to the IDE does not fail: the IDE starts, +// the renderer answers CDP normally, and nothing ever reports the project as +// open, so this exits 3 ("the compile never settled") after the full timeout +// and reads like a wedged IDE. Pack the tree first, or use tbrun, which packs +// it for you. +if (proj && !/\.twinproj$/i.test(proj)) { + console.error(`not a .twinproj: ${proj}\n` + + (existsSync(proj) && statSync(proj).isDirectory() + ? " That is a source tree. tbbuild takes a packed project; scripts/tbrun.mjs\n" + + " takes a source tree, and packs it for you." + : " tbbuild takes a packed project file.")); + process.exit(2); +} +// A path that merely ENDS in .twinproj gets the same treatment, because the +// IDE's behaviour is identical: it launches, the renderer answers CDP, and +// the project is never reported open. Checking the extension alone still left +// a typo'd or deleted path costing the full timeout. +if (proj && !existsSync(proj)) { + console.error(`no such project: ${proj}`); + process.exit(2); +} if (!IDE) { console.error("no twinBASIC IDE found: pass --ide , set TB_IDE, " + "or unpack a twinBASIC_IDE_BETA_ folder on your Desktop"); @@ -207,14 +234,58 @@ const probe = () => c.evaluate(`JSON.stringify({ i: document.getElementById("infoCount")?.textContent ?? "", p: typeof projectFilePath !== "undefined" ? projectFilePath : null, rows: (() => { - let out = [], t = 1; + const out = []; if (typeof problemsPanel === "undefined" || !problemsPanel) return out; - while (true) { - t = problemsPanel.tree.view.data.getNextVisibleNode2(t, false, true); - if (!t) break; - const o = problemsPanel.tree.view.data.generateNodeInfo(t); - const n = generateCopyPasteTextForProblem(o, true); - if (n) out.push(n); + const d = problemsPanel.tree.view.data; + if (typeof PROBLEMSPANEL_CUSTOMPROPERTY_TYPE === "undefined") { + // Older/newer IDE without the constants: fall back to the IDE's own + // helper, with its "errors only" flag OFF so warnings still appear. + let t = 1; + while (true) { + t = d.getNextVisibleNode2(t, false, true); + if (!t) break; + const n = generateCopyPasteTextForProblem(d.generateNodeInfo(t), false); + if (n) out.push(n); + } + return out; + } + // The panel hides hints and info by default (hideGroup3/hideGroup4 are + // true), so they are not in the walk at all. Clear all four, walk, and put + // them back -- this whole function is one synchronous evaluate, so the IDE + // never renders the intermediate state. + const saved = [d.hideGroup1, d.hideGroup2, d.hideGroup3, d.hideGroup4]; + d.hideGroup1 = d.hideGroup2 = d.hideGroup3 = d.hideGroup4 = false; + try { + // LSP DiagnosticSeverity: 1 Error, 2 Warning, 3 Information, 4 Hint -- + // which is what the compiler's language socket speaks. 3 and 4 are the + // opposite way round from the status bar's hint-then-info column order, + // and guessing from that order got them backwards. No backticks in any + // comment here: this block is inside a template literal, and one + // truncates the whole evaluate string. Measured by forcing + // TB0013 into project.warnings.hints (status bar "2 hint(s)", severity 4) + // and then into .info (status bar "2 info", severity 3). + const LABEL = { 1: "{ERROR}", 2: "{WARNING}", 3: "{INFO}", 4: "{HINT}" }; + let t = 1; + while (true) { + t = d.getNextVisibleNode2(t, false, true); + if (!t) break; + const o = d.generateNodeInfo(t); + if (!o) continue; + // Type 0 is the per-file header row, which carries no diagnostic. + if (o.getCustomData(PROBLEMSPANEL_CUSTOMPROPERTY_TYPE) + !== PROBLEMSPANEL_CUSTOMPROPERTY_TYPE1_DIAGNOSTIC) continue; + const url = o.getParentNodeInfo + ? (o.getParentNodeInfo() || {}).getCustomData( + PROBLEMSPANEL_CUSTOMPROPERTY_TYPE0_FILEHEADER_URL) : ""; + const sev = o.getCustomData(PROBLEMSPANEL_CUSTOMPROPERTY_TYPE1_DIAGNOSTIC_SEVERITY); + const ln = o.getCustomData(PROBLEMSPANEL_CUSTOMPROPERTY_TYPE1_DIAGNOSTIC_LINENUM); + const ch = o.getCustomData(PROBLEMSPANEL_CUSTOMPROPERTY_TYPE1_DIAGNOSTIC_CHARNUM); + out.push((LABEL[sev] || ("{SEVERITY" + sev + "}")) + " " + (url || "") + + " [" + (ln + 1) + "," + (ch + 1) + "]: " + o.getCaption()); + } + } finally { + d.hideGroup1 = saved[0]; d.hideGroup2 = saved[1]; + d.hideGroup3 = saved[2]; d.hideGroup4 = saved[3]; } return out; })() })`); @@ -249,14 +320,28 @@ const counts = ["e", "w", "h", "i"].map((k) => Number(final[k] ?? 0)); // A row count that disagrees with the status bar means the compile was still // moving when the sample was taken. Refuse rather than report either number. +// +// This invariant was unsatisfiable for two years on any project with a warning. +// The walk passed the IDE's copy helper its "errors only" flag -- the helper is +// `if (t && severity !== 1) return;` -- and the panel hides hints and info by +// default, so `rows` could only ever hold errors while `counts` held all four. +// A project with 0 errors and 2 warnings read as "0 rows against 0/2/0/0" and +// exited 3, which looks exactly like a compile that never settled. The walk +// above now reads severity from the panel's own node data instead, so the two +// sides count the same things and a real race is again the only way to trip it. if (counts.reduce((a, b) => a + b, 0) !== rows.length) { die(3, `unsettled: ${rows.length} rows against ${counts.join("/")} in the status bar`); } +// The IDE's pid is reported so a caller can clean up precisely. It matters most +// under --keep, where this process leaves the IDE running and something else has +// to end it: killing by image name instead takes out every concurrent run's IDE, +// and the user's own open IDE with it. if (asJson) { console.log(JSON.stringify({ project: proj, errors: counts[0], warnings: counts[1], hints: counts[2], infos: counts[3], + idePid: child?.pid ?? null, kept: keep, diagnostics: rows, dialogs, }, null, 2)); } else { @@ -264,6 +349,8 @@ if (asJson) { console.log(`--- ${counts[0]} error(s), ${counts[1]} warning(s), ` + `${counts[2]} hint(s), ${counts[3]} info`); if (dialogs.length) console.log("dialogs:", JSON.stringify(dialogs)); + // Only under --keep, where the pid is still alive and therefore actionable. + if (keep && child?.pid) console.log(`ide-pid: ${child.pid}`); } c.close(); diff --git a/scripts/tbrun.mjs b/scripts/tbrun.mjs index 61d5bb8f..dd3e8c61 100644 --- a/scripts/tbrun.mjs +++ b/scripts/tbrun.mjs @@ -8,7 +8,10 @@ // (default 2500) // --json emit one JSON object instead of text // --raw do not strip the console's timestamp column -// --keep leave the IDE running afterwards +// --keep leave the IDE running afterwards (implies --no-reap) +// --no-reap do not harvest automation servers the probe left behind +// --reap-images comma-separated image names to harvest +// (default: the Office suite -- see REAP_IMAGES) // --show / --hide passthrough to tbbuild // // Exit: 0 captured output, 1 the project has compile errors, 2 the harness @@ -27,7 +30,7 @@ // the IDE once the exe is built, so anything it writes with Debug.Print lands // in the IDE's DEBUG CONSOLE, where this script reads it back over CDP. // -// ------------------------------------------------- five things it gets right +// ----------------------------------------------- seven things it gets right // // Each of these cost an hour when the probe was first done by hand. // @@ -42,9 +45,15 @@ // 2. A JAVASCRIPT .click() ON THE BUILD BUTTON DOES NOTHING. `#buildIcon` is // a plain DIV wired through the IDE's own pointer handling; it needs real // CDP Input.dispatchMouseEvent presses at its centre. -// 3. THE CONSOLE INTERLEAVES A TIMESTAMP LINE per output line, because the -// pane's "Show Timestamps" option is on by default. Those lines are the -// pane's, not the program's, and are stripped unless --raw. +// 3. READ THE CONSOLE'S BACKING ARRAY, NOT THE PANE. The DEBUG CONSOLE is a +// virtualised list view: only the rows that fit are in the DOM, so an +// `.innerText` scrape of it returns the tail of a long probe and looks +// exactly like a complete capture. Measured against the old reader: a +// probe printing 120 lines came back with 11. `debugConsoleContent +// .dataNodes` is the whole log, and the walk here is the IDE's own +// "Copy All" minus the clipboard write. The timestamp column comes off in +// the same step, since it is a nested in each entry -- so --raw is +// now a different slice of the same string rather than a line filter. // 4. START THE PROBE WITH Debug.Cls. The DEBUG CONSOLE is also where the IDE // writes its own build log, and the linker writes there after the build -- // so without a clear, a probe's output comes back interleaved with @@ -52,6 +61,21 @@ // 5. QUIET-PERIOD, NOT A MARKER. Waiting for a sentinel string means every // probe has to print one and the script has to know it. Waiting for the // console to stop changing works for any probe. +// 6. EVERY RUN OWNS ITS OWN WORKSPACE AND KILLS ONLY ITS OWN IDE. Both were +// shared, and both broke concurrency in ways that looked like something +// else. The staging directory was a fixed %TEMP%/tbrun/src, so a second +// run rmSync'd the first one's tree out from under it -- observed as an +// EPERM from a script that had touched no such path. Worse, shutdown was +// `taskkill /F /T /IM twinBASIC.exe`, which is machine-wide: it ended +// every concurrent run's IDE, and the IDE you had open yourself. The work +// directory and the project.id are now keyed to --port, and the IDE is +// killed by the pid tbbuild reports. +// 7. A COM SERVER THE PROBE STARTED IS NOT A CHILD OF ANYTHING WE OWN. +// CreateObject("Excel.Application") is activated by DCOM, so the EXCEL.EXE +// that appears has svchost.exe for a parent -- measured. No tree kill can +// reach it, and a probe that throws before app.Quit leaves it running +// forever. Harvesting it therefore has to be a before/after diff, which is +// a blunt enough instrument to need the guard rails in reapOrphans(). import { spawn, execFileSync } from "node:child_process"; import { existsSync, readFileSync, writeFileSync, mkdirSync, statSync, readdirSync, @@ -70,12 +94,16 @@ const opt = (n, d) => { const die = (code, msg) => { console.error(msg); process.exit(code); }; +// Every flag that TAKES A VALUE has to be named here, or its value is mistaken +// for the source directory. +const VALUE_FLAGS = ["port", "timeout", "quiet", "ide", "reap-images"]; const positional = argv.filter((a, i) => - !a.startsWith("--") && !(i > 0 && ["port", "timeout", "quiet", "ide"].includes(argv[i - 1]?.replace(/^--/, "")))); + !a.startsWith("--") && !(i > 0 && VALUE_FLAGS.includes(argv[i - 1]?.replace(/^--/, "")))); if (!positional.length || flag("help")) { die(2, "usage: node scripts/tbrun.mjs [--port N] [--timeout S] " + - "[--quiet MS] [--json] [--raw] [--keep] [--show|--hide]"); + "[--quiet MS] [--json] [--raw] [--keep] [--no-reap] [--reap-images a,b] " + + "[--show|--hide]"); } const srcDir = path.resolve(positional[0]); @@ -93,6 +121,15 @@ const port = Number(opt("port", 9346)); const timeoutMs = Number(opt("timeout", 120)) * 1000; const quietMs = Number(opt("quiet", 2500)); +// Images a probe can leave behind through COM activation. Office is the set that +// prompted this; --reap-images replaces the list for anything else. Only out-of- +// process (LocalServer32) servers can outlive the probe at all -- an in-process +// one dies with it -- so this list is short by nature rather than by omission. +const REAP_IMAGES = [ + "excel", "winword", "powerpnt", "msaccess", "outlook", + "onenote", "mspub", "visio", "winproj", +]; + // ---------------------------------------------------------------- the IDE function findIde() { @@ -116,7 +153,13 @@ if (!existsSync(compilerExe)) die(2, `no compiler beside the IDE at ${compilerEx // ------------------------------------------------- pin the build output (1) -const work = path.join(tmpdir(), "tbrun"); +// (6) The workspace is keyed to --port, which is already the thing that has to +// differ between concurrent runs -- the IDE's DevTools port, its WebView2 user +// data folder and its private desktop are all keyed to it in tbbuild. Sharing +// one %TEMP%/tbrun/src meant the second run deleted the first one's tree. +const runKey = String(port); +const work = path.join(tmpdir(), "tbrun", runKey); +rmSync(work, { recursive: true, force: true }); mkdirSync(work, { recursive: true }); // Staged into a temp copy rather than edited in place. Pinning buildPath is what @@ -124,7 +167,6 @@ mkdirSync(work, { recursive: true }); // caller's project, and a probe harness that rewrites the tree you pointed it at // is one you stop trusting with a real project. const stage = path.join(work, "src"); -rmSync(stage, { recursive: true, force: true }); cpSync(srcDir, stage, { recursive: true }); const stagedSettings = path.join(stage, "Settings"); const exePath = path.join(work, "tbrun-probe.exe"); @@ -133,8 +175,11 @@ const projPath = path.join(work, "tbrun-probe.twinproj"); const settings = JSON.parse(readFileSync(stagedSettings, "utf8")); const wasTemplate = /\$\{/.test(settings["project.buildPath"] ?? ""); settings["project.buildPath"] = exePath; -// Two probes sharing a project.id confuse the IDE's recents list. -settings["project.id"] = "{7B247000-0000-4000-9000-7B2470000001}"; +// Two probes sharing a project.id confuse the IDE's recents list -- so this is +// keyed to the port too, not a constant. The last group is 12 hex digits, of +// which the port fills the low six. +settings["project.id"] = + `{7B247000-0000-4000-9000-7B2470${port.toString(16).padStart(6, "0")}}`; writeFileSync(stagedSettings, JSON.stringify(settings, null, "\t"), "utf8"); const sourceText = (() => { @@ -168,6 +213,12 @@ if (!/\.\.\. DONE\s*$/.test(packed.trim())) { // ------------------------------------------------------ compile, via tbbuild +// (7) Taken before the IDE starts, so anything in it is somebody else's and is +// never a candidate for harvesting. Cheap enough to be unconditional (~0.4 s +// against a ~10 s run) and skipping it under --no-reap would only make the two +// paths differ in a way nobody would remember. +const processesBefore = snapshotProcesses(); + const here = path.dirname(fileURLToPath(import.meta.url)); const passthrough = ["--keep", "--port", String(port)]; if (flag("show")) passthrough.push("--show"); @@ -180,20 +231,71 @@ build.stdout.on("data", (d) => { buildOut += d; }); build.stderr.on("data", (d) => { buildOut += d; }); const buildCode = await new Promise((res) => build.on("exit", res)); +// tbbuild ran with --keep, so the IDE it started is ours to end. It prints the +// pid for exactly this reason: killing by image name would end every concurrent +// run's IDE and the one the user has open. +const idePid = Number(/^ide-pid:\s*(\d+)\s*$/m.exec(buildOut)?.[1]) || null; +if (!idePid) { + console.error("warning: tbbuild did not report an ide-pid -- falling back to " + + "killing by image name, which will also end any other IDE running now."); +} + if (buildCode !== 0) { process.stdout.write(buildOut); - killIde(); + shutdown(); process.exit(buildCode === 1 ? 1 : 2); } // --------------------------------------------- build the exe, read the console -const CONSOLE_JS = `(() => { - const tw = [...document.querySelectorAll(".toolWindowContainer")] - .find(e => /DEBUG CONSOLE/i.test(e.textContent || "")); - return tw ? (tw.innerText || "") : null; +// (3) Read the console's BACKING ARRAY, never the pane. `debugConsoleContent` +// is a createListView(), which renders only the rows that fit -- so the old +// `.innerText` scrape returned the last ~11 lines of any longer probe and gave +// no sign that it had. `dataNodes` is the complete log: addItem() appends at +// `itemCount` and nothing in main.js ever removes an entry, so the array holds +// every line written since the last clear(). +// +// The walk below is the IDE's own `tbDebugConsole_ClipboardCopyAll`, minus the +// clipboard write -- the same borrow tbbuild makes for the diagnostics report. +// Each entry is `TIMETEXT`, so +// slicing past the first `` drops the timestamp, which is what the +// IDE's own two Copy All variants differ by. Note that the timestamp is always +// present in the data: the pane's "Show Timestamps" option only sets a +// `--timestampsDisplay` CSS variable, so it cannot change what we read here. +// +// Two deliberate departures from the IDE's version: +// +// * DECODE WITH textContent ON OUR OWN DETACHED NODE, not the IDE's +// HTMLToTEXT. That helper reads `.innerText` off a shared `hiddenDiv`, and +// it preserves runs of spaces only because `initMisc()` creates that div +// with no parent -- an element that is not rendered has innerText === +// textContent. Attach it in some future build and every padded value a +// probe prints starts collapsing silently. Probes measure things like +// Debug.Print zone widths and Partition's space-padded labels, so that is +// the one thing this reader must not get wrong. +// * TOLERATE A MISSING TIMESTAMP SPAN. indexOf returns -1 when there is +// none, and the IDE's `substr(i + 7)` would then quietly eat six +// characters of real output. No current addItem() path omits it; the guard +// costs a comparison and removes a silent-corruption mode. +const consoleJs = (withTimestamps) => `(() => { + if (typeof debugConsoleContent === "undefined" || !debugConsoleContent || + !debugConsoleContent.dataNodes) return null; + const decode = (html) => { + const d = document.createElement("div"); // never attached; see above + d.innerHTML = html; + return d.textContent; + }; + return debugConsoleContent.dataNodes.map(n => { + const i = n.indexOf(""); + if (i < 0) return decode(n); + return decode(${withTimestamps} + ? n.substr(0, i + 7) + " " + n.substr(i + 7) + : n.substr(i + 7)); + }).join("\\n"); })()`; +const CONSOLE_JS = consoleJs(flag("raw")); + let captured = null, failure = null; try { const cdp = await attach(port); @@ -218,7 +320,11 @@ try { while (Date.now() - started < timeoutMs) { await new Promise((r) => setTimeout(r, 400)); const now = await cdp.evaluate(CONSOLE_JS); - if (now === null) throw new Error("the DEBUG CONSOLE pane is not open in this IDE"); + if (now === null) { + throw new Error("no debugConsoleContent.dataNodes in this IDE -- the DEBUG CONSOLE " + + "was never created, or this build moved it. Refusing rather than " + + "falling back to scraping the pane, which silently truncates."); + } if (now !== last) { last = now; lastChange = Date.now(); if (strip(now).length) seen = true; } else if (seen && Date.now() - lastChange > quietMs) break; } @@ -228,7 +334,7 @@ try { failure = e.message; } -if (!flag("keep")) killIde(); +const reaped = shutdown(); if (failure) die(2, `tbrun: ${failure}`); if (!captured.length) { @@ -239,33 +345,105 @@ if (!captured.length) { } if (flag("json")) { - console.log(JSON.stringify({ exe: exePath, lines: captured }, null, 2)); + console.log(JSON.stringify({ exe: exePath, lines: captured, idePid, reaped }, null, 2)); } else { for (const l of captured) console.log(l); } // ------------------------------------------------------------------ helpers -// (3) drop the pane's own chrome: the header, the input prompt, and the -// timestamp line the console emits beside every output line. +// Trim blank lines off both ends. That is all this has to do now: reading +// dataNodes rather than the pane means the header, the ">" input prompt and +// the timestamp column never arrive in the first place, so the three filters +// that used to live here are gone along with the guesswork in them. function strip(text) { if (!text) return []; - const lines = text.split("\n"); - const out = []; - for (const raw of lines) { - const l = raw.replace(/\r$/, ""); - if (/^DEBUG CONSOLE$/.test(l.trim())) continue; - if (l.trim() === ">") continue; - if (!flag("raw") && /^\s*\d{2}:\d{2}:\d{2}\.\d+\s*$/.test(l)) continue; - out.push(l); - } + const out = text.split("\n").map((l) => l.replace(/\r$/, "")); while (out.length && !out[0].trim()) out.shift(); while (out.length && !out[out.length - 1].trim()) out.pop(); return out; } -function killIde() { - for (const image of ["twinBASIC.exe", "twinBASIC_win32.exe", "twinBASIC_win32_noDEP.exe"]) { - try { execFileSync("taskkill", ["/F", "/T", "/IM", image], { stdio: "ignore" }); } catch {} +// (6) Kill OUR IDE by pid, never by image name. /T takes the probe exe and +// anything it spawned with CreateProcess; what it cannot take is a COM server, +// which is what reapOrphans is for. +function killTree(pid) { + try { execFileSync("taskkill", ["/PID", String(pid), "/T", "/F"], { stdio: "ignore" }); } + catch { /* already gone */ } +} + +function shutdown() { + if (flag("keep")) return null; // the IDE is the caller's problem now + if (idePid) killTree(idePid); + else { + // Only reachable when tbbuild did not report a pid, which is warned about + // above. Machine-wide, and the lesser evil against leaking an IDE. + for (const image of ["twinBASIC.exe", "twinBASIC_win32.exe", "twinBASIC_win32_noDEP.exe"]) { + try { execFileSync("taskkill", ["/F", "/T", "/IM", image], { stdio: "ignore" }); } catch {} + } + } + return flag("no-reap") ? null : reapOrphans(); +} + +// Identity is pid + start time, because a pid alone is reused and a run that +// reaped a recycled pid would be killing a stranger. +function snapshotProcesses() { + const ps = "Get-Process | Select-Object Id, ProcessName, " + + "@{n='Start';e={try{$_.StartTime.ToFileTimeUtc()}catch{0}}}, " + + "@{n='Win';e={$_.MainWindowTitle}} | ConvertTo-Json -Compress"; + try { + const out = execFileSync("powershell", ["-NoProfile", "-NonInteractive", "-Command", ps], + { encoding: "utf8", stdio: ["ignore", "pipe", "ignore"], + maxBuffer: 32 * 1024 * 1024 }); + const parsed = JSON.parse(out); + return new Map((Array.isArray(parsed) ? parsed : [parsed]) + .map((p) => [`${p.Id}:${p.Start}`, p])); + } catch { + return null; // reaping degrades to off, never to guessing + } +} + +// A COM server started by the probe has svchost.exe for a parent (measured), so +// there is no ancestry to walk and this has to be a before/after diff. Three +// guard rails, because a diff over a live machine is a blunt instrument: +// +// - it must be NEW -- present now, absent from the pre-launch snapshot; +// - its image must be on REAP_IMAGES, so an unrelated process that happened +// to start during the run is never a candidate; +// - it must have NO main window, which is what separates a server the probe +// activated from the copy of Excel the user opened to look at a spreadsheet. +// +// Anything new and on the list but WINDOWED is reported and left alone. That is +// the case where the evidence is ambiguous, and killing it could discard +// somebody's unsaved work. +// +// Known limit: two concurrent runs both driving Excel cannot tell their servers +// apart, so whichever finishes first harvests both. Pass --no-reap for that and +// sweep once at the end of the batch. +function reapOrphans() { + if (!processesBefore) return null; + const after = snapshotProcesses(); + if (!after) return null; + + const images = new Set((opt("reap-images", "") || "") + .split(",").map((s) => s.trim().toLowerCase().replace(/\.exe$/, "")).filter(Boolean)); + const wanted = images.size ? images : new Set(REAP_IMAGES); + + const killed = [], skipped = []; + for (const [key, p] of after) { + if (processesBefore.has(key)) continue; + if (!wanted.has(String(p.ProcessName).toLowerCase())) continue; + if (p.Win && String(p.Win).trim()) { skipped.push(p); continue; } + killTree(p.Id); + killed.push({ pid: p.Id, image: p.ProcessName }); + } + + for (const p of skipped) { + console.error(`note: ${p.ProcessName} (pid ${p.Id}) started during this run but has a ` + + `window open, so it was left alone -- close it yourself if it is a leak.`); + } + if (killed.length) { + console.error("reaped: " + killed.map((k) => `${k.image} (pid ${k.pid})`).join(", ")); } + return killed; }