feat: GenericAgent adapter (GPT-5.4, Completion 90.46%) + fix rubric parser + report 13 rubric bugs

[Sophomoresty]

Summary

We are the GenericAgent team. This PR adds our GA adapter to Harness-Bench, achieving Completion 90.46% on the 28 public tasks with GPT-5.4 (reasoning_effort=none).

Additionally, we identified and documented 13 rubric bugs in the official task set that prevent Process/Combined scoring for those tasks.

Results (28 public tasks, 10-way parallel, fully isolated)

Metric	Value	Scope
Completion (outcome)	90.46%	all 28 tasks
Process	94.31%	15 tasks with functional rubrics
Combined	89.04%	15 tasks with functional rubrics
Avg Input	23.8K tokens/task	—
Avg Output	5.1K tokens/task	—

13 tasks produce N/A for Process/Combined due to upstream rubric bugs (documented below).

Changes

1. GA Adapter (src/clawbench_v2/adapters/ga_agent.py)

Bridges Harness-Bench tasks to the in-process GenericAgent runtime:

• Native agent_runner_loop integration with GPT-5.4 backend
• Multimodal image injection for vision tasks
• Time-budget guard and deliverable re-entry for QA
• Benchmark isolation: memory tools (update_working_checkpoint, start_long_term_update) stripped from tool schema — agent cannot access cross-task SOPs or experience files
• Task-agnostic system/user prompts: no hardcoded task IDs, filenames, or oracle thresholds

2. Process Rubric Parser Fix (src/clawbench_v2/process_grading.py)

Hardens the rubric response parser to handle benign grader-output variation:

• Extracts scoring object from multi-JSON output (graders often emit plan/status preamble before the verdict)
• Accepts flat dimension scores without {"scores": {...}} wrapper
• Computes total as arithmetic mean when the field is absent

This does NOT mask the 13 upstream rubric bugs — those still produce N/A by design.

3. Upstream Rubric Bug Report (RUBRIC_BUGS.md)

We discovered 3 classes of defects in tasks/*/llm_rubric.py:

#	Bug class	Tasks affected	Failure mode
1	Format KeyError at module load	17-like-record, 18-album-metadata-retrieval	JSON braces not escaped in .format() — module cannot import
2	Unescaped {reference} in USER_TEMPLATE	19–25, 27–29 (10 tasks)	.format(task_name=..., payload=...) raises KeyError
3	No JSON output instruction	26-db-doc-consistency	Grader returns Markdown prose instead of parseable JSON

Reproduction:

import importlib.util
from pathlib import Path
for name in ["17-like-record", "19-landmark-recognition", "27-provider-failover-audit"]:
    p = Path("tasks")/name/"llm_rubric.py"
    spec = importlib.util.spec_from_file_location("r", p)
    m = importlib.util.module_from_spec(spec)
    try:
        spec.loader.exec_module(m)
        m.USER_TEMPLATE.format(task_name=name, payload="{}")
    except Exception as e:
        print(name, type(e).__name__, repr(str(e)))

4. Supporting Infrastructure

• Adapter registration in registry.py and adapters/__init__.py
• Runner: multi-round session support, usage proxy integration
• report_final.py: standalone script computing Completion/Process/Combined independently with proper N/A handling

Isolation Guarantee

We ensure full reproducibility:

• No memory tools: update_working_checkpoint and start_long_term_update explicitly stripped from benchmark tool schema
• Independent workspaces: each task gets a UUID-suffixed sandbox directory
• No session reuse across tasks: session_id is unique per task (contains task_id + random hex)
• No hardcoded oracle knowledge: system prompt contains only generic execution methodology
• No SOP/experience leakage: verified by grepping all transcripts — zero references to external memory files

No task source modifications

All tasks/*/llm_rubric.py and tasks/*/oracle_grade.py files are unmodified in this PR. The rubric bugs are documented for upstream fix, not worked around.

About GenericAgent

GenericAgent is a lightweight, tool-augmented agent framework. For this evaluation we use its native agent_runner_loop with a minimal 7-tool schema (file_read, file_write, file_patch, code_run, web_scan, web_execute_js, ask_user).

[RenBing-Sumeru]

Thank you for your attention and feedback. Our latest paper has made corresponding revisions to these issues and has now been added to the 106 questions. And I will push the corresponding updates up tomorrow. We sincerely welcome you to provide us with more valuable suggestions.
We sincerely invite you to test the latest 106 questions. And we will incorporate your latest features into the system in the future. Thank you for your valuable feedback.