feat: axTree() — box model of a live application - #12
Merged
Merged
Conversation
Returns element boxes, hierarchy, roles and labels for a running app from the accessibility API, optionally with colours sampled from a capture and typography from the AX attributed string. Geometry is measured, not inferred from OCR bounding boxes. Ships as a fourth prebuilt helper through the existing native pipeline, so nothing compiles on the user's machine. Cost is bounded on purpose. Every attribute read is a synchronous IPC round trip, and the target app's implementation dominates rather than tree size — the same 4000 elements measured 1.6s in Safari and 11s in Finder. Reads are batched, offscreen subtrees culled, the walk capped, and `budget` reports what happened including `capped: true`, so a truncated tree is never presented as complete. Two things only showed up once it ran: - The obvious JSON cost more than the screenshot it replaces — ~12.8k tokens for 250 Safari nodes against ~6.9k for the image. Encoding box as [x,y,w,h] and omitting default-valued fields cut 44%; pruning unlabelled containers cut another 48%. A full tree is still not a token saving over a screenshot, and the README says so instead of implying one — the case for this is exact geometry and semantics, plus being able to take a slice. - Swift omits nil rather than encoding null, so the root has no `parent` key; the TypeScript type promised `number | null` and a consumer checking `=== null` would have been wrong. A test caught it. Also fixes captureScreen() accepting an invalid region: given a negative or fully offscreen rect, screencapture clamps and exits 0 on an unlocked Mac while failing elsewhere, so the caller's mistake surfaced as corrupt output. The rect is now checked against real display bounds. This also makes a pre-existing test deterministic — it asserted on screencapture's own behaviour and failed on any unlocked machine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
docs/BOX-MODEL.mdphases 1–3: the accessibility tree of a running app as a box model, with optional colours and typography.Ships as a fourth prebuilt helper (
ax-helper) through the existing native pipeline — nothing compiles on the user's machine.Cost is bounded on purpose
Every attribute read is a synchronous IPC round trip, and the target app's implementation dominates rather than tree size — the same 4000 elements measured 1.6 s in Safari and 11 s in Finder. So: attribute reads are batched, offscreen subtrees culled,
maxElements/maxDepthcap the walk, andbudgetreports what happened includingcapped: true. A truncated tree is never presented as complete.Two things that only appeared once it ran
The obvious JSON cost more than the screenshot it replaces — ~12.8k tokens for 250 Safari nodes against ~6.9k for the image. Encoding
boxas[x,y,w,h]and omittingenabled: true/focused: falsecut 44%; pruning unlabelled containers (detail: 'content', the default) cut another 48%, 600 → 289 nodes on Finder. Net ~25 tokens/node instead of ~51.Even so, a full tree is still not a token saving over a screenshot (~7.3k vs ~6.9k on Finder). The README says that plainly rather than implying a saving that does not exist — the case for this is exact geometry, roles and enabled state, plus being able to take a slice.
Swift omits
nilrather than encodingnull, so the root node has noparentkey. The TypeScript type promisednumber | null; a consumer checking=== nullwould have been wrong. A test caught it; the type is nowparent?: number.Drive-by fix
captureScreen()accepted an invalid region: given a negative or fully offscreen rect,screencaptureclamps and exits 0 on an unlocked Mac while failing elsewhere, so a caller's mistake surfaced as a corrupt 3.6 KB image. The rect is now validated against real display bounds. This also makes a pre-existing test deterministic — it asserted onscreencapture's own behaviour and failed on any unlocked machine (verified: it fails onmaintoo).Verification
91/91 tests, including 8 new ones over
axTree. Typecheck, lint and build clean. Exercised live against Finder, Safari, TextEdit and an Electron app.🤖 Generated with Claude Code