[Improvement] Per-model coordinate spaces via coord_scale

[aaronsmulktis]

Gives OpenApps a single place where a model's predicted coordinates are converted into viewport pixels, and fixes two bugs found while building it.

Why

Vision models disagree about what an (x, y) in their output means:

convention	coord_scale	models
raw viewport pixels	null	UI-TARS 1.5, GPT-4o-style computer use
normalized 0-1000	1000	Qwen-VL, GLM-VL
normalized [0, N)	N	PaliGemma/Gemma lineage bins to 1024

Getting this wrong is silent. The click lands somewhere plausible, the page does not change, and the episode scores 0 with no error anywhere in the logs. Before this change the only handling lived inside the Qwen parser as a hardcoded _COORD_SPACE = 1000, so no other family could express a convention at all.

What changed

action_parsers/coords.py::rescale_xy is now the one conversion point; every parser reaches it through ActionParser.rescale. Each family carries a default (uitars: raw pixels, qwen3vl: 0-1000), overridable per model in the agent yaml with coord_scale: N. Qwen's hardcoded constant becomes that field — same behaviour, now expressible rather than baked in.

Two real bugs, both of which made a configured coord_scale silently not apply:

1. browsergym-syntax mouse actions were never rescaled. uitars_parser only remapped UI-TARS-native forms (click(point=…) and friends). A model prompted in browsergym syntax emits mouse_click(x=, y=), which matched none of them and passed through untouched. This is the common case — config/agent/default.yaml has always prompted for mouse_click(x=612, y=455). Now rewritten in place, preserving trailing kwargs so button='right' survives.

2. The "no conversion" path still converted. UITarsActionParser.parse always passed a rescale hook, even with coord_scale=None, so raw pixels round-tripped through int(round(float(x))) — rounding floats and rewriting already-valid calls. It now passes no hook at all. Caught by its own regression test: mouse_click(x=500.5) was coming back as x=500.

On the motivation, and what the evidence actually showed

This work was prompted by Gemma 4 31B clicking off-target in screenshot-only mode, on the theory that coordinates needed rescaling. Cluster runs falsified that theory, and reviewers should not read this PR as a fix for it.

Two runs, inverting the executed action back through the conversion:

run	model's stated coords	coord_scale	executed	expected
1	(88, 238)	1024	(165, 251)	(165, 251) ✓
2	(611, 645)	1000	(1173, 697)	(1173, 697) ✓

Exact to the pixel, both times, at two different scales on a 1920×1080 viewport. The pipeline faithfully put the cursor where the model asked. Gemma's failure is upstream of the conversion — its own spatial estimate is wrong — which is the case the README describes as grounding badly, not scaling wrong. Set-of-marks bids are the route for that model.

So the value here is not the original motivation. It is: Qwen's convention becomes configurable instead of hardcoded, two latent bugs are fixed, and any future family (GLM-VL, a Gemma checkpoint that does ground well) can declare its space in one line.

The most reliable setup, documented in the README, is to declare the grid in the prompt and set coord_scale to match — correct by construction if the model complies — rather than reverse-engineering a checkpoint's native convention. Qwen3.6-VL-computer-use.yaml does exactly this.

Notes for review

• rescale_xy applies one scalar against each viewport axis, which is what a square normalized grid means. It cannot express a model predicting in its own non-square resized image space; that would need a per-axis pair. Called out in the docstring.
• The (1920, 1080) viewport fallback in vLLM_prompt.py now logs a warning. It is only reachable with use_screenshot off, but under a non-1080p preset it would silently miscale every converted coordinate.
• The default UI-TARS path is unchanged and pinned by a regression test.
• The README documents a calibration recipe using the ground-truth boxes in set_of_marks_coordinates.json, so the next model's convention gets measured rather than guessed.

Testing

46 tests in tests/test_action_parsers.py and tests/test_uitars_parser.py, covering rescale_xy directly (raw passthrough, 0-1000, 0-1024, per-axis scaling, negative deltas, zero-scale), both parser families, the browsergym-form rewrite, and the unchanged default path.

[Copilot]

Pull request overview

This PR centralizes coordinate-space conversion for vision-model actions by introducing a single rescale_xy implementation and plumbing a per-parser (and per-model override) coord_scale through the agent and action parser registry. It also updates UI-TARS parsing to correctly rescale browsergym-form mouse actions and adds warnings when viewport fallback could silently mis-scale coordinates.

Changes:

• Added shared coordinate conversion (action_parsers/coords.py::rescale_xy) and ActionParser.rescale, plus export wiring.
• Introduced configurable per-model coord_scale (family defaults + YAML override) and propagated it through VLLMAgent.
• Expanded tests and documentation around coordinate conventions and UI-TARS/browsergym rescaling behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_action_parsers.py	Adds coverage for coord_scale defaults/overrides and rescale_xy, plus UI-TARS/browsergym rescaling cases.
src/open_apps/agent/vLLM_prompt.py	Logs a warning when falling back to a default viewport (important for coordinate rescaling correctness).
src/open_apps/agent/vLLM_agent.py	Adds coord_scale to agent args and passes it into get_action_parser.
src/open_apps/agent/utils.py	Extends flexible_parser/uitars_parser to accept an optional rescale hook and rescales browsergym mouse actions.
src/open_apps/agent/README.md	Documents coordinate-space conventions and a calibration workflow.
src/open_apps/agent/action_parsers/uitars.py	Makes UITars parser carry coord_scale and conditionally provide rescaling behavior.
src/open_apps/agent/action_parsers/qwen3vl.py	Replaces hardcoded coordinate space with configurable coord_scale via ActionParser.rescale.
src/open_apps/agent/action_parsers/coords.py	New shared coordinate conversion implementation.
src/open_apps/agent/action_parsers/base.py	Adds coord_scale to the base parser and provides rescale.
src/open_apps/agent/action_parsers/init.py	Adds optional coord_scale override and exports rescale_xy.
config/agent/default.yaml	Documents new coord_scale configuration knob.

Suppressed comments (1)

src/open_apps/agent/utils.py:420

• The right-click coordinate extraction uses re.findall(r"\d+"), which mis-parses float coordinates and drops negative signs, producing incorrect coordinates before any rescaling.

        x, y = rescale(coords[0], coords[1])
        result["action"] = f"mouse_click(x={x}, y={y}, button='right')"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.