Skip to content

plan.md

Latest commit

 

History

History
1131 lines (907 loc) · 64.9 KB

plan.md

File metadata and controls

1131 lines (907 loc) · 64.9 KB
Raw

Trace2Skill Universal Framework — Master Plan

Paper tham chiếu: Trace2Skill — Distill Trajectory-Local Lessons into Transferable Agent Skills (arXiv 2603.25158) Goal: Framework harness-agnostic để evolve skill cho bất kỳ agent / CLI / dự án nào thông qua trajectory analysis Timeline: 16 tuần (full), có thể nén xuống 8 tuần (aggressive) hoặc 4 tuần (PoC) Repo model: Monorepo License: MIT — open source từ ngày đầu LLM strategy: API-only — Anthropic / OpenAI / Gemini / OpenAI-compatible (OpenRouter, DeepSeek, Groq, Together, Qwen API, xAI, ...). Không cần GPU. vLLM/Ollama = optional, community-tier. Ngày tạo: 2026-04-22 (revised 2026-04-22: API-only)

Changelog

  • 2026-04-22 — Initial plan
  • 2026-04-22 (rev) — Chốt API-only: bỏ yêu cầu GPU ở Phase 4, thêm GeminiProvider + OpenAICompatibleProvider làm first-class, vLLM/Ollama → optional. Default analyst = Sonnet 4.6, default judge = Haiku 4.5, paper replication = Qwen qua OpenRouter.
  • 2026-04-23 — Sessions 1-6 shipped: Phase 0 / 1 / 2 done; Phase 1.5 items #5 (cost tracking) + #6 (agentic Error Analyst) done; Phase 3 partial (AnthropicLLMProvider, AnthropicSkillFormat, SimpleReActHarness interim). 154 tests passing (150 unit + 4 live). Framework usable end-to-end with real Claude API. See §0.5 Progress Snapshot for full status + remaining checklist.
  • 2026-04-23 (session 12) — 2 parts. (a) Competitive scan: SkillClaw (AMAP-ML, arXiv 2604.08377) reviewed. Cloned at _external/SkillClaw/ (gitignored). Complementary, not a scoop. Borrow-list parked in memory skillclaw_competitor.md + §0.6. (b) Phase 4 paper-replication REAL RUN executed multiple times (~$0.30 total, Gemini 2.5 Pro). Discovered + fixed 4 critical infra gaps the scaffolding had silently left: (1) SpreadsheetEvaluator couldn't handle range (J2:J4) or quoted-sheet ('COLLECTION'!A1) answer_positions — fixed with range tuple compare + quote stripping; (2) AnalystDispatcher was wired to simple ErrorAnalyst not the agentic one — flipped to AgenticErrorAnalyst (the USP); (3) _ground_truth_text couldn't read xlsx (tried UTF-8 decode on binary) — added _dump_xlsx_as_text helper so read_ground_truth tool actually works for spreadsheet domain; (4) pydantic's TypeAdapter.json_schema() emits $defs + $ref which Gemini OpenAI-compat rejects when embedded inside tool params — added _inline_refs resolver. 291 unit tests passing (+6: 5 evaluator range/quoted-sheet, 2 agentic xlsx GT). Proof-of-life confirmed: pipeline runs end-to-end agentic with real xlsx GT visible; quality gate (drop) fires correctly; merger produces 0-1 ops at n_evolve=5 (prevalence threshold ≥2 too tight for subset this small — paper-level delta needs n_evolve ≥ 15-20, deferred). Committed bb66c7d.
  • 2026-04-24 (session 13) — Phase 5: LangChain harness + evidence adapters. Two pieces shipped. (a) LangChainHarnessAdapter at trace2skill/harnesses/langchain.py (~200 LOC) — wraps LangChain's AgentExecutor, per-task executor build with skill-in-system-prompt + create_tool_calling_agent, converts intermediate_steps → ReActStep. Lazy langchain import. 13 unit tests via mock executor. (b) LangChainEvidenceAdapter at trace2skill/evidence_adapters/langchain.py (~280 LOC) — backed by LangSmith API. collect(run_id) fetches the root run + child runs, walks all descendants of run_type == "tool" (including nested sub-chains), orders by start_time, extracts feedback via client.list_feedback. behavioral has duration + token counts + project; execution has tool call/error counts with capped error snippets; feedback API errors swallowed gracefully (returns None) so evidence collection never fails on a flaky side-channel. 21 unit tests with _FakeRun/_FakeFeedback duck types + patch("langsmith.Client") for lazy-ctor verification. Added [langchain] pyproject extra (covers both adapters; langsmith comes transitively with langchain). Both modules pass ruff clean + pyright --strict 0 errors. 319 unit tests passing (+21 from 298). Committed 2c23518 + 41a4e69. Remaining Phase 5: Cline / OpenCode harness adapters, Gemini native SDK (thinking-mode), examples/04_langchain_rag_agent/ demo, LangSmith thread-id convention for next_user_turns.
  • 2026-04-24 (session 14 slice 1) — Phase 7 kickoff: project polish. README rewrite (was stale "Phase 0 scaffolding" — totally misleading while framework is at Phase 5/6 with 319 tests, 3 harness adapters, 5 provider families, 2 evidence sources). New README has real quickstart (example 02), minimal Python API example, architecture diagram, adapter matrix, honest status ("pre-alpha, actively developed"), citation. CONTRIBUTING.md added (closes Phase 0 deferred item): dev setup, 3-check gate (pytest + ruff + pyright strict), conventions, "add a new adapter" walkthrough with file pointers. .github/ISSUE_TEMPLATE/ bug + feature templates, plus pull_request_template.md with plan.md phase + checklist. Committed 1e6d99b.
  • 2026-04-24 (session 14 slice 2) — Phase 7 docs site. mkdocs-material configured at mkdocs.yml with indigo palette + instant navigation + code-copy + pymdownx extensions. 10 doc pages under docs/: index (landing + ASCII pipeline diagram), getting-started (install + Windows-path caveat + Example 02 walkthrough), architecture (4 axes + 3 stages + data model + async choices), harnesses/{index,claude-code,langchain,custom} (custom.md has a ~30-line skeleton new adapter can start from), providers, evidence, rubrics (no-regex rule + bilingual pattern), roadmap (summary pointing at plan.md), contributing (summary pointing at CONTRIBUTING.md). Added [docs] pyproject extra + .github/workflows/docs.yml that builds on push/PR and deploys to GitHub Pages on main. mkdocs build --strict passes. /site/ gitignored. Committed daba526.
  • 2026-04-24 (session 14 slice 3) — Phase 7 PyPI publish prep. Version bumped 0.0.1 → 0.1.0 (reflects the real functional surface: 3 harness adapters + 2 evidence adapters + 5 provider families + 319 tests, not "pre-alpha scaffolding"). Dev Status classifier upgraded Pre-Alpha → Alpha. Classifiers expanded: Intended Audience :: Science/Research, Operating System :: OS Independent, Python :: 3 / 3.13, Topic :: Software Development :: Libraries :: Python Modules, Typing :: Typed. Keywords broadened (claude-code, langchain, react, prompt-engineering, agentic). [project.urls] expanded from 2 → 6 (Homepage, Documentation→gh-pages, Repository, Issues, Changelog, Paper). trace2skill/py.typed marker added so downstream users get type info without reportMissingTypeStubs noise. [tool.hatch.build] explicit include/exclude lists to keep tests/, benchmarks/, _external/, examples/, docs/, site/ OUT of the wheel. CHANGELOG.md written in Keep-a-Changelog format. python -m build produces clean trace2skill-0.1.0-py3-none-any.whl (101KB) + sdist (75KB) with prompts + py.typed + LICENSE + entry_points.txt shipped. Wheel METADATA inspected — all classifiers + URLs + keywords propagate correctly. Not uploaded to PyPI yet (needs user's PyPI account + twine upload). 319 tests still pass.

0. TL;DR (đọc trước khi bắt đầu)

Trace2Skill là một kỹ thuật cải tiến file SKILL.md tự động cho agent, bằng cách:

  1. Cho agent chạy 200 task có đáp án → thu 200 trajectory
  2. Gọi 200 "analyst" song song, mỗi analyst đề xuất patch cho skill dựa trên 1 trajectory
  3. Merge tất cả patch theo cây (hierarchical) → chỉ giữ pattern lặp lại ≥2 lần

Không đụng vào weights của model (không phải finetuning). Không online sequential (như Hermes). Batch parallel → nhanh 20× và chất lượng cao hơn.

Framework này = implementation của Trace2Skill + abstraction 4 trục plugin để chạy được với mọi agent harness (Claude Code, LangChain, OpenCode, Cline, custom, ...).

Key insight từ paper phải nhớ:

  • Skill evolved từ model nhỏ (35B) transfer được sang model lớn (122B)
  • Skill cho domain A transfer được sang domain OOD B
  • Parallel batch > online sequential (+6.8pp chất lượng, 20× tốc độ)
  • Error analyst PHẢI là agentic ReAct loop (không phải single LLM call), có access ground truth + artifacts
  • Merge operator PHẢI chỉ giữ edit xuất hiện ≥2 lần trong pool patch (prevalence-weighted)
  • 3 guardrail deterministic: file tồn tại, line-range conflict, format validator

0.5. Progress Snapshot

Last updated: 2026-04-24 (session 14 slices 1+2+3 — Phase 7 polish + docs + PyPI prep) Sessions invested: 14 (Phase 0 → 3.5 → 6 → 5 subset → 4 scaffolding → SkillClaw + Phase 4 infra → Phase 5 LangChain x2 → Phase 7 polish + docs + PyPI prep) Test status: 319 unit + e2e + 7 live integration passing (unchanged — session 14 is docs + packaging, no code). ruff clean, pyright --strict 0 errors trên source em viết. Pre-existing 1 Windows path test failure. Version: 0.1.0 (bumped session 14 slice 3). Wheel + sdist build clean with python -m build. PyPI upload pending user's account. Framework state: plug-and-forget với real Claude Code + multi-provider + paper-replication infra complete (session 12). Agentic error analyst with xlsx-aware GT now actually active in pipeline (was silently wired to simple analyst before). Paper-replication runs end-to-end on real SpreadsheetBench data with Gemini Pro 2.5; signal-level delta paper-faithful run (n_evolve ≥ 15-20) deferred for cost/time. Competitive landscape: SkillClaw (arXiv 2604.08377) reviewed session 12. Complementary product. See §0.6 Competitive landscape + memory skillclaw_competitor.md.

Phase completion

Phase Status Notes
0. Foundation ✅ Done Skipped: CONTRIBUTING.md, adapter template docs, uv workspaces (dùng hatchling)
1. Core Pipeline MVP ✅ Done Async parallel, W=32 default (paper 32-64 rollout, 128 analyst — lên được qua kwarg), hierarchical merge B=32/L_max=6, 3/3 guardrails
1.5. Polish (deferred từ Phase 1) ✅ 6/6 ✅ #1 async Stage 1 + Stage 2, ✅ #2 trial-apply guardrail, ✅ #3 ensemble judge, ✅ #4 hierarchical merge, ✅ #5 cost tracking, ✅ #6 agentic Error Analyst
2. Signal Layer ✅ Done (MVP scope) 1/5 rubrics shipped (generic.yaml). Lint rule deferred. Ensemble judge đã có (Phase 1.5 #3)
3. First Real Harness ✅ 4/4 ✅ AnthropicLLMProvider, ✅ AnthropicSkillFormat, ✅ ClaudeCodeHarnessAdapter (subprocess + session JSONL parse + SSE proxy fallback + double-encode-JSON recovery), ✅ ClaudeCodeEvidenceAdapter
4. Paper Replication 🟢 Infra complete (session 12) Subset scaffolding (session 11) + 4 infra fixes (session 12): SpreadsheetEvaluator range + quoted-sheet; AnalystDispatcher wired to AgenticErrorAnalyst (not simple); _ground_truth_text xlsx-aware dump; _inline_refs schema resolver for Gemini tool-calling. Real runs executed with Gemini 2.5 Pro (~$0.30 total session 12). End-to-end functional including agentic analyst quality gate firing. Paper-level delta (n_evolve ≥ 15-20, ~$0.30-0.40, ~20 min wall clock) deferred. Full paper scale (6 conditions × 2 models × 200/200) deferred.
5. Multi-harness Expansion 🟡 Providers + LangChain pair done (session 10 + 13) ✅ OpenAICompatibleProvider generic (session 10). ✅ LangChainHarnessAdapter + ✅ LangChainEvidenceAdapter via LangSmith (session 13) — 34 unit tests. Remaining: Cline/OpenCode harness, Gemini native SDK (if thinking-mode needed), LangChain example demo.
6. Semi-online Closed Loop ✅ Done (MVP) SessionStore (SQLite), evolve_from_trajectories() skip-Stage-1, skill deploy + atomic backup + rollback, trace2skill evolve-online + trace2skill rollback CLI, example 03 với SessionEnd hook. Skip y=0 trajectories (agentic ErrorAnalyst cần GT — paper §2.3, heuristic error mode defer).
7. Release & Ecosystem 🟢 Slices 1+2+3 done (session 14) ✅ Slice 1 project polish. ✅ Slice 2 mkdocs docs site. ✅ Slice 3 PyPI publish prep: version 0.0.1 → 0.1.0, classifiers expanded (Alpha + Science/Research + Python 3.11/3.12/3.13 + Typing::Typed), keywords broadened, [project.urls] expanded to 6 links (Docs + Repo + Issues + Changelog + Paper), trace2skill/py.typed marker added, hatch.build include/exclude explicit, CHANGELOG.md written. python -m build produces clean trace2skill-0.1.0-py3-none-any.whl (101KB) + .tar.gz (75KB) with prompts + py.typed + LICENSE + entry_points shipped. Not pushed to PyPI — needs account + twine upload. Remaining: PyPI upload itself (manual), launch materials (HN/Twitter/Discord).

Files shipped (mapping tới plan §4 repo structure)

Core (Phase 0):

  • trace2skill/core/models.py — 15 dataclasses + ROOT_FILENAME constant
  • trace2skill/core/protocols.py — 5 Protocol classes (HarnessAdapter / LLMProvider / SkillFormat / EvidenceAdapter / Evaluator) + Message / Tool / Action
  • trace2skill/core/config.py — Pydantic YAML schema
  • trace2skill/core/loader.py — importlib-based adapter loader
  • trace2skill/__init__.py, trace2skill/core/__init__.py

Pipeline stages (Phase 1 + 1.5):

  • trace2skill/stages/stage1_rollout.py — Phase 1.5 #1 async asyncio.Semaphore(workers=32) + gather, JSONL checkpoint written after gather
  • trace2skill/stages/stage2_analyze/dispatcher.py — Phase 1.5 #1 async parallel analyst dispatch (workers=32 default, paper W=128 lên được qua kwarg); routes y==1 → SuccessAnalyst, y==0 → ErrorAnalyst
  • trace2skill/stages/stage2_analyze/success_analyst.py — single-pass
  • trace2skill/stages/stage2_analyze/error_analyst.py — non-agentic MVP (fallback)
  • trace2skill/stages/stage2_analyze/agentic_error_analyst.py — Phase 1.5 #6 — 6 tools (inspect_skill_file, read_ground_truth, try_patch, diff_vs_gt, finish_with_patch, drop), quality gate
  • trace2skill/stages/stage2_analyze/formatters.py — trajectory/skill/ground_truth → prompt text
  • trace2skill/stages/stage3_consolidate/guardrails.py — 3/3: file-exists + line-range overlap + Phase 1.5 #2 check_trial_apply (dry-run Skill.apply_patch)
  • trace2skill/stages/stage3_consolidate/merger.py — Phase 1.5 #4 hierarchical recursion (batch_size=32, max_levels=6 default; paper B_merge=32, L_max=6). Singleton chunks pass through; cap triggers single-shot final merge.
  • trace2skill/pipeline.py — Trace2SkillPipeline orchestrator + PipelineResult; evolve() kwargs: rollout_workers, analyst_workers, merge_batch_size, merge_max_levels

Mocks (test doubles):

  • trace2skill/mocks/harness.py — MockHarnessAdapter
  • trace2skill/mocks/llm.py — MockLLMProvider (complete + complete_structured + react queues)
  • trace2skill/mocks/evaluator.py — MockEvaluator
  • trace2skill/mocks/fixtures.py — make_mock_skill, make_patch_add_bullet consistency helpers

Prompts:

  • trace2skill/prompts/error_analyst.md
  • trace2skill/prompts/success_analyst.md
  • trace2skill/prompts/merge_operator.md
  • trace2skill/prompts/agentic_error_analyst.md — Phase 1.5 #6
  • trace2skill/prompts/judge.md — Phase 2
  • trace2skill/prompts/__init__.py — load() + render() với $placeholder substitution

Signal layer (Phase 2):

  • trace2skill/signal/rubric.py — Pydantic Rubric + from_yaml
  • trace2skill/signal/evidence_formatter.py
  • trace2skill/signal/judge.py — LLMJudge với Phase 1.5 #3 ensemble (samples: int = 1; samples > 1 → majority vote, tie → y=None, mean confidence, union evidence/failure_modes)
  • rubrics/generic.yaml — bilingual VN + EN signals

Providers + formats + harnesses + CLI (Phase 3 + 3.5 + 6 + 5 subset):

  • trace2skill/llm/anthropic_provider.py — complete / complete_structured / react. Phase 3.5 fixes: _parse_sse_stream cho proxy trả text/event-stream; _unstringify_json_fields retry khi model (Haiku) double-encode list field thành JSON string.
  • trace2skill/llm/openai_compatible.py — Phase 5 subset session 10 — OpenAICompatibleProvider generic (1 class, wrap openai.AsyncOpenAI + custom base_url). Cover: OpenAI, Gemini (via compat endpoint), OpenRouter (Qwen/DeepSeek/Llama/...), DeepSeek, Groq, Together, xAI, Mistral. Reuse _unstringify_json_fields cho Haiku-style double-encode quirk. Gemini arguments-as-dict quirk handled qua _parse_tool_arguments.
  • trace2skill/llm/cost.py — Phase 1.5 #5 — CostAccountant + Pricing + BudgetExceeded + DEFAULT_ANTHROPIC_PRICING. Phase 5 session 10: thêm DEFAULT_OPENAI_PRICING / DEFAULT_GEMINI_PRICING / DEFAULT_OPENROUTER_PRICING / DEFAULT_DEEPSEEK_PRICING.
  • trace2skill/skill_formats/anthropic.py — AnthropicSkillFormat load/save với junk-dir filtering (includes .trace2skill-backups/)
  • trace2skill/harnesses/simple_react.py — interim (Phase 3) in-process ReAct
  • trace2skill/harnesses/_session_parser.py — Phase 3.5 — pure parser parse_session_jsonl(path) -> ParsedSession (trajectory + cwd + raw_user_messages). Shared bởi harness + evidence adapter.
  • trace2skill/harnesses/claude_code.py — Phase 3.5 — ClaudeCodeHarnessAdapter (subprocess claude -p ..., wall-clock timeout, JSONL parse, slug fallback scan, verbose mode + stderr drain).
  • trace2skill/harnesses/langchain.py — Phase 5 session 13 — LangChainHarnessAdapter wraps LangChain AgentExecutor with create_tool_calling_agent. Lazy import. _response_to_trajectory converts intermediate_steps → ReActStep list. _coerce_tool_input normalizes dict/str/other inputs.
  • trace2skill/evidence_adapters/langchain.py — Phase 5 session 13 — LangChainEvidenceAdapter backed by LangSmith. collect(run_id) reads root + child runs via client.read_run(..., load_child_runs=True), walks tool-type descendants ordered by start_time, fetches feedback via client.list_feedback. Feedback API errors swallowed to None (free-tier flakiness). Lazy langsmith import. Module file-level # pyright disables for the langsmith-untyped-Run handling.
  • trace2skill/evidence_adapters/claude_code.py — Phase 3.5 — ClaudeCodeEvidenceAdapter (next_user_turns, git status --porcelain, tool_error counts); accepts projects_dir as str OR Path cho YAML config.
  • trace2skill/core/loader.py — Phase 3.5 thêm load_llm_provider + load_tasks_from_jsonl. Phase 5 session 10: 5 factories mới (openai, gemini, openrouter, deepseek, openai-compat generic) với env-var key injection + default pricing per provider.
  • trace2skill/core/session_store.py — Phase 6 SQLite ledger (UPSERT mark_processed, filter_unprocessed)
  • trace2skill/core/skill_deploy.py — Phase 6 atomic deploy + timestamped backup + reversible rollback
  • trace2skill/pipeline.py — Phase 6 thêm evolve_from_trajectories() cho online mode (skip Stage 1, filter y=1 only); shared _run_analysis_stages cho cả 2 flow
  • trace2skill/core/config.py — Phase 6 thêm OnlineSpec (sessions_dir, min_sessions, session_store_path, skill_name); harness thành optional vì online mode không cần
  • trace2skill/cli/main.py + trace2skill/__main__.py — Phase 3.5 evolve subcommand; Phase 6 thêm evolve-online + rollback subcommands
  • pyproject.toml — Phase 3.5 thêm [project.scripts] trace2skill = "trace2skill.cli.main:run"
  • examples/02_claude_code_basic/ — Phase 3.5 demo: README, trace2skill.yaml, skill-v0/SKILL.md, tasks.jsonl (20 tasks), evaluator.py, install-skill.sh
  • examples/03_claude_code_semi_online/ — Phase 6 demo: README, trace2skill-online.yaml, settings.json.example (SessionEnd hook)
  • examples/02_claude_code_basic/trace2skill-gemini.yaml — Phase 5 session 10 — copy of trace2skill.yaml with Gemini as analyst/judge/merger. Cost drops ~10× vs Haiku.

Paper replication (Phase 4 subset — session 11 + 12):

  • trace2skill/evaluators/__init__.py + trace2skill/evaluators/spreadsheet.py — SpreadsheetEvaluator (openpyxl cell-level diff with string-trim + numeric 1e-6 tolerance + multi-sheet refs). Session 12: range answer_positions (e.g. J2:J4, C12:H14) via tuple-of-tuples compare; quoted sheet names ('COLLECTION'!A1) via quote strip.
  • trace2skill/stages/stage2_analyze/dispatcher.py — session 11 AnalystDispatcher.analyst_modes kwarg ({"error"} = paper's +Error, {"success"} = +Success, both/None = +Combined). Wired through Trace2SkillPipeline.evolve(). Session 12: swapped ErrorAnalyst → AgenticErrorAnalyst (the USP — was previously inactive). error_turn_budget kwarg.
  • trace2skill/stages/stage2_analyze/agentic_error_analyst.py — session 12 _dump_xlsx_as_text(path, max_cells=400) helper + xlsx-aware branch in _ground_truth_text so read_ground_truth tool returns actual cell values (not UnicodeDecodeError). _inline_refs() resolver inlines pydantic $defs + $ref before embedding DiffOp schema in tool params (Gemini OpenAI-compat rejects $ref inside tool params).
  • benchmarks/paper_replication/load_spreadsheetbench.py — lazy HF download (Verified 400 tasks, 15MB), JSON parser, per-task workspace materializer.
  • benchmarks/paper_replication/run_subset.py — orchestrator: baseline eval → Stage 1 evolve → Stage 2/3 → evolved eval → results.json with delta_pp. Supports --dry-run.
  • benchmarks/paper_replication/skill-v0/SKILL.md — minimal xlsx seed skill.
  • benchmarks/paper_replication/README.md — setup + methodology deviations + scale-up guide.
  • benchmarks/paper_replication/runs/ — session 12 artifacts (gitignored via /runs/): stage1.jsonl + skill-evolved/ + results.json from the real Pro runs.

Infra:

  • pyproject.toml — hatchling + deps + ruff/pyright/pytest config, optional [anthropic] extras
  • README.md, LICENSE (MIT), .gitignore, .env.example
  • .github/workflows/ci.yml
  • tests/conftest.py — auto-load .env for integration tests

Tests (289 unit + 2 e2e + 7 live = 298, -1 Windows-path pre-existing fail):

  • Core: test_smoke (5), test_skill_apply_patch (20), test_loader (6), test_loader_llm_and_tasks (11), test_mocks (10)
  • Stages: test_stage1_rollout (10), test_stage2_analyze (11 — session 11 +3 analyst_modes; session 12 rewired 3 dispatcher tests for agentic), test_stage3_consolidate (21), test_agentic_error_analyst (19 — session 12 +2 xlsx GT dump)
  • Evaluators: test_spreadsheet_evaluator (16 — session 11: 11 for cell match/mismatch/tol/trim/multi-sheet/missing/errors; session 12 +5: range column, range rect, range mismatch, quoted sheet, quoted sheet + range)
  • Paper-replication: test_spreadsheetbench_loader (6 Phase 4 — split, FileGroundTruth, writable workspace copy, not-enough-tasks, missing cache, id_whitelist)
  • Signal: test_signal (17)
  • Providers: test_anthropic_provider (18 — Phase 3.5 +3 SSE, +1 double-encode), test_openai_compatible (16 Phase 5), test_cost (16)
  • Loader: test_loader_providers (8 Phase 5 — each new factory, explicit api_key override, default pricing injection, unknown provider)
  • Formats + harnesses: test_skill_format_anthropic (11), test_simple_react_harness (12), test_claude_code_session_parser (14), test_claude_code_harness (8), test_claude_code_evidence (6)
  • CLI: test_cli (5 batch), test_cli_online (7 Phase 6: threshold, dry-run, end-to-end deploy, rollback list/restore/no-backups)
  • Phase 6 new: test_session_store (7), test_pipeline_online (5), test_skill_deploy (8)
  • Integration: test_anthropic_live (2), test_agentic_live (1), test_full_pipeline_live (1), test_claude_code_live (1), test_gemini_live (2 Phase 5 — complete + complete_structured)
  • E2E: test_pipeline (2 mock)

Architectural deviations from plan (đã document)

  1. Skill.apply_patch(patch: Patch) not (diff: UnifiedDiff) — plan §5.2 signature thay đổi. Patch là internal structured form, đủ cho MVP. UnifiedDiff dataclass vẫn còn, để dành cho future text-based interop (git apply).
  2. SimpleReActHarness làm Phase 3 interim — không có trong plan gốc. Dùng AnthropicLLMProvider.react() + basic fs tools. Real ClaudeCodeHarnessAdapter (subprocess / claude-agent-sdk) deferred sang Phase 3.5.
  3. Cost tracking build ở Phase 1.5 #5 — plan §7 có nó trong Phase 2 bullet list. Move up vì nó hook ở lowest layer (messages.create) và được Phase 2 (judge) + Phase 3 (provider) dùng ngay.
  4. Flat pyproject.toml thay vì uv workspaces — plan §11 first-day nói uv workspaces. Dùng hatchling flat cho đơn giản MVP. Migrate lúc nào cũng được.
  5. ROOT_FILENAME = "SKILL.md" constant — plan §5.2 Skill không có concept này. Thêm để Skill.apply_patch biết op nào target root_md vs resources.
  6. LLM-related types trong trace2skill/llm/ thay vì llm-providers/ — plan §4 repo structure có llm-providers/ là sibling package. Dùng subfolder trong core package cho đơn giản MVP. Refactor khi ship PyPI.

Deferred checklist (phải làm khi pick up)

Phase 0 polish (nếu cần):

  • CONTRIBUTING.md
  • Dev docs "How to add new adapter" template
  • Migrate flat pyproject → uv workspaces (optional)

Phase 1.5 complete (6/6): all closed out in session 7 — see Files shipped above.

Phase 2 remaining:

  • rubrics/code_editing_agent.yaml
  • rubrics/qa_agent.yaml
  • rubrics/search_agent.yaml
  • rubrics/spreadsheet_agent.yaml
  • Lint rule ban regex trong trace2skill/signal/ via ast-grep

Phase 3.5 complete (session 8): all closed out — see Files shipped above. Real Claude Code CLI plug-and-play: trace2skill evolve --config trace2skill.yaml wraps subprocess rollout + pipeline.

Phase 4 subset (session 11 + 12) — infra complete:

  • Scaffolding shipped session 11; 4 critical infra fixes session 12 (evaluator range/quoted, agentic wired, xlsx GT dump, schema inliner).
  • Dataset cached at ~/.cache/trace2skill/spreadsheetbench/ (15MB, Verified 400 tasks).
  • Real runs executed session 12 with Gemini 2.5 Pro (3 attempts, ~$0.30 total): 5/3 simple analyst → 20/10 simple → 5/3 agentic+xlsx-fixed. Pipeline verified end-to-end agentic with quality gate firing.
  • Paper-level signal deferred: prevalence-weighted merge threshold (≥2 agreement) needs n_evolve ≥ 15-20 to produce >0 final ops reliably. Cost ~$0.30-0.40 Gemini, wall-clock ~20-25 min. Run when signal fidelity matters for publication/demo.
  • Windows-specific env vars needed: PYTHONPATH="$(pwd)" (workaround for Vietnamese-char .pth file resolution) + PYTHONIOENCODING=utf-8 (for → in stderr logs) + GEMINI_API_KEY (script reads from env; .env has key under GOOGLE_API_KEY — aliasing or dotenv loader TODO).

Phase 5 remaining:

  • LangChain/Cline/OpenCode harness adapters
  • Gemini native SDK (if thinking-mode needed)

Phase 6 complete (session 9): all closed out. See examples/03_claude_code_semi_online/ for SessionEnd hook setup.

Phase 7: not started

Next session — đọc gì trước

  1. §0.5 này (progress snapshot)
  2. §13 Notes — các cái KHÔNG được quên (1-8)
  3. git log --oneline — latest commits:
    • edf4fd1 Phase 1.5 + 3.5 + 6 (session 9)
    • c2fa704 Phase 5 subset OpenAICompatibleProvider (session 10)
    • (session 11 Phase 4 subset — still uncommitted per git status)
  4. Run python -m pytest -q --ignore=tests/integration → baseline 288 tests (~1s)
  5. python benchmarks/paper_replication/run_subset.py --dry-run --n-evolve 3 --n-test 2 → verify dataset cache still valid

Recommended next session — Phase 5 harness adapters

Trace2Skill hiện chỉ plug-and-play với Claude Code (+ OpenAI-compat providers). Phase 5 mở rộng ecosystem:

  1. LangChain adapter — LangChainHarnessAdapter (AgentExecutor wrap) + LangChainEvidenceAdapter (Callbacks + LangSmith API). ~150-250 LOC + test + example.
  2. Cline adapter — CLI wrapper + ClineSkillFormat (.clinerules converter).
  3. OpenCode adapter — similar pattern to Claude Code.

Mỗi adapter ~1-2 sessions. Validate: adapter mới viết < 150 LOC, core package không đổi.

LEGACY: Recommended next session (Phase 4 subset REAL RUN) — STALE, done session 12

# Session 12 executed this 3 times with Gemini 2.5 Pro. See §0.5 Phase 4 row.

LEGACY: Recommended next session (Phase 3.5) — STALE, already done

Phase 3.5 — finish real Claude Code integration để unlock Phase 6 (semi-online):

  1. ClaudeCodeHarnessAdapter — thay SimpleReActHarness cho real Claude Code agent
  2. ClaudeCodeEvidenceAdapter — parse user's actual sessions
  3. CLI — wrap Trace2SkillPipeline trong entry point trace2skill evolve

Sau Phase 3.5 + Phase 6 = "plug & forget" UX: Claude Code tự improve skill theo thời gian (cron + EvidenceAdapter đọc real sessions + evolve + deploy).

0.6. Competitive landscape

Last scanned: 2026-04-23 (session 12)

SkillClaw (AMAP-ML / Alibaba)

  • Paper: arXiv 2604.08377 — "SkillClaw: Let Skills Evolve Collectively with Agentic Evolver" (Apr 2026, #2 HuggingFace Daily Papers)
  • Repo: https://github.com/AMAP-ML/SkillClaw (MIT, ~18k LOC Python)
  • Local clone for reference: _external/SkillClaw/ (gitignored)
  • Architecture: 2 components joined by shared storage (OSS/S3/local):
    • Client Proxy (FastAPI) — intercepts /v1/chat/completions + /v1/messages from any OpenAI-compatible agent, transparent to agent code
    • Evolve Server — batch evolution, 2 engines: workflow (fixed 3-stage LLM) vs agent (OpenClaw-driven workspace)
  • Pipeline (workflow engine): Summarizer → SessionJudge (4-D scoring) → Aggregation (by skill-reference) → Execution (4 actions: create / improve / optimize_description / skip) → SkillVerifier (LLM gate)
  • 5-way failure taxonomy: SKILL_CONTENT_STALE, SKILL_MISSELECT, SKILL_GAP, TOOL_ERROR, MODEL_BASELINE
  • Primary target agents: Hermes, Codex, Claude Code, OpenClaw, QwenPaw, IronClaw, PicoClaw, ...

Competitive positioning — Trace2Skill vs SkillClaw

Trục Trace2Skill (ours) SkillClaw
Design center Research framework, harness-agnostic, paper-faithful Product/deployment, daemon, multi-user
Integration HarnessAdapter plugin (subprocess, SDK) Transparent proxy /v1/*
Scope Single-user offline batch + semi-online Multi-user collective (OSS/S3)
Signal Binary y + 3 deterministic guardrails + agentic Error Analyst with tools 4-D float score + 5-way failure + LLM-only SkillVerifier
Merge Hierarchical prevalence-weighted (≥2 occurrences, inductive) Session-level aggregation by skill-name reference
Skill lookup Full skill loaded (paper-faithful, no RAG) Embedding retrieval optional (sentence-transformers)
Storage SQLite + local FS OSS-first, S3 / local fallback
Evolution trigger CLI or threshold Cron + idle client

Trace2Skill's USPs to defend (SkillClaw does NOT have these)

  1. Agentic ReAct Error Analyst with 6 tools (inspect_skill_file, read_ground_truth, try_patch, diff_vs_gt, finish_with_patch, drop) — SkillClaw's verifier is single LLM call
  2. Prevalence-weighted hierarchical merge (edits ≥2 occurrences) — paper §2.4 inductive reasoning
  3. 3 deterministic guardrails (file-exists, line-range, trial-apply)
  4. Zero-regex signal layer with YAML rubric
  5. Faithful paper replication on SpreadsheetBench (±3pp target)
  6. 4-plugin generic architecture (Harness/LLM/SkillFormat/Evidence) — 150-line adapter bar

Features from SkillClaw worth borrowing later (parked post-Phase 4)

Detailed list + file pointers in memory skillclaw_competitor.md. Summary:

  1. 4-D Judgment scoring instead of binary y=0/1 (task_completion 0.55 + response_quality 0.30 + efficiency 0.05 + tool_usage 0.10)
  2. 5-way failure taxonomy — actionable for analyst prompts (tells analyst WHERE to edit)
  3. Session → skill-reference mapping — improve Phase 6 semi-online (target relevant sessions per skill)
  4. Proxy integration mode — alternative to HarnessAdapter, for agents without SDK. 5th adapter type.
  5. Two-engine pattern — add agent engine alongside workflow (analyst gets write access to skill dir)
  6. Dashboard — Phase 7 polish
  7. Staged validation publish — candidate → worker → publish (opt-in review before shared push)
  8. Collective/multi-user shared storage — Phase 8+ if community adoption warrants

Strategic conclusion

Not scooped. SkillClaw and Trace2Skill occupy different niches (product vs framework). The two USP clusters — agentic analyst + prevalence merge + guardrails on our side, proxy + multi-user + dashboard on theirs — are orthogonal. Continue current roadmap; revisit borrow-list after Phase 4 real run lands.

1. Mission & Scope

Mission

Xây 1 framework độc lập harness, plug được vào bất kỳ agent nào (existing hoặc custom), để:

  • Evolve skill từ trajectory thực tế (batch offline hoặc semi-online)
  • Không cần finetuning, không cần retrain
  • Không cần retrieval module ở inference time
  • Output: file SKILL.md + resources portable giữa các agent

Scope

✅ LÀM ❌ KHÔNG LÀM
Pipeline 3-stage (rollout → analyze → consolidate) Không build 1 agent mới (không thay thế Claude Code/Hermes)
Plugin system: Harness / LLM / SkillFormat / Evidence Không tie vào model cụ thể
Rubric-driven signal extraction (zero regex) Không online sequential (đã chứng minh kém hơn)
CLI + Python API Không UI (có thể phase sau)
Batch offline + semi-online closed loop Không realtime skill update sau mỗi request
Full audit trail (mọi patch trace về trajectory nguồn) Không thay thế finetuning (orthogonal)
Multi-harness adapters (Claude Code, LangChain, ...) Không lock vào 1 ecosystem

Non-goals tường minh

  1. Không reinvent agent harness — dùng lại ReAct/SDK có sẵn
  2. Không build LLM provider từ đầu — wrap thư viện có sẵn (anthropic, openai, vllm)
  3. Không signal extraction bằng regex/hardcode — LLM-as-judge với rubric YAML
  4. Không fully online — chỉ batch hoặc semi-online (scheduled batch)

2. Key concepts từ paper (cheatsheet)

Skill structure

skill_dir/
├── SKILL.md              # procedural knowledge (when/how/failure modes)
├── scripts/              # executable helpers (.py, .sh)
├── references/           # case-specific lookups (.md)
└── assets/               # static data (.json, ...)

Pipeline 3 stages

Stage 1: Trajectory Generation
  IN:  (S_0, D_evolve, agent π_θ)
  OUT: TrajectoryCorpus = T⁺ ∪ T⁻

Stage 2: Parallel Multi-Agent Patch Proposal
  IN:  (FrozenSkill S_0, T⁺, T⁻)
  OUT: PatchPool P = {p_i}
  - ErrorAnalyst (agentic ReAct, có access GT + artifacts)
  - SuccessAnalyst (single-pass)
  - N sub-agents chạy song song, mỗi agent xử 1 trajectory

Stage 3: Conflict-Free Consolidation
  IN:  (S_0, PatchPool P)
  OUT: S* = S_0 + apply(p*)
  - Hierarchical merge với B_merge = 32
  - L = ⌈log_B_merge |P|⌉ levels
  - Inductive reasoning: chỉ giữ edit lặp ≥2 lần
  - 3 guardrail deterministic

Hyperparameters (paper defaults)

Param Value Ghi chú
W (analyst workers) 128 Parallel fan-out Stage 2
B_merge 32 Batch size Stage 3
L_max 6 Cap log depth
turn_budget (rollout) 100 ReAct turns cho agent chính
turn_budget (error analyst) 20-30 Ít hơn agent chính
rollout_workers 32-64 Concurrent rollout

2 modes

  • Deepening: S_0 = human-written skill. Pipeline refine.
  • Creation: S_0 = LLM parametric draft (ineffective). Pipeline tạo skill tốt từ 0.

3. Architecture — 4 trục plugin

┌─────────────────────────────────────────────────────────────┐
│                  trace2skill CORE                           │
│         (harness-agnostic, zero domain logic)               │
│                                                             │
│   Stage 1 Rollout → Stage 2 Analyze → Stage 3 Merge         │
│                    + Signal Layer                           │
└───┬──────────────┬──────────────┬──────────────┬────────────┘
    │              │              │              │
┌───▼────┐    ┌────▼────┐    ┌────▼────┐    ┌────▼──────┐
│Harness │    │   LLM   │    │  Skill  │    │ Evidence  │
│Adapter │    │Provider │    │ Format  │    │  Adapter  │
└────────┘    └─────────┘    └─────────┘    └───────────┘

User tự mix combo:
  Claude Code + Anthropic + Anthropic skill + Claude Code evidence
  LangChain + OpenAI + Cline rules + LangSmith evidence
  Custom ReAct + Gemini + Custom + Custom
  Custom ReAct + OpenRouter (Qwen) + Anthropic skill + Custom  # paper replication
  ...

4 trục — nhiệm vụ từng trục

Trục Nhiệm vụ Ví dụ implementation
HarnessAdapter Chạy 1 task trên agent harness cụ thể → trả Trajectory ClaudeCodeAdapter, LangChainAdapter, CustomReActAdapter
LLMProvider Wrap LLM API cho analyst + merger dùng AnthropicProvider, OpenAIProvider, GeminiProvider, OpenAICompatibleProvider (generic)
SkillFormat Đọc/ghi skill theo format trên disk AnthropicSkillFormat (SKILL.md + resources/), ClaudeCodeFormat (CLAUDE.md), ClineFormat (.clinerules)
EvidenceAdapter Thu thập signal thô từ session thật (cho semi-online) ClaudeCodeEvidence (parse ~/.claude/projects/), LangSmithEvidence (query API)

Nguyên tắc vàng: Core không biết adapter nào tồn tại. User install adapter họ cần, config qua YAML.

4. Repo structure (monorepo)

trace2skill/
├── trace2skill/                    # core package (zero adapter deps)
│   ├── core/
│   │   ├── models.py               # Task, Trajectory, Patch, Skill, Evidence, Judgment
│   │   ├── protocols.py            # 4 plugin interfaces (Protocol classes)
│   │   └── config.py               # YAML schema + Pydantic validator
│   ├── stages/
│   │   ├── stage1_rollout.py
│   │   ├── stage2_analyze/
│   │   │   ├── dispatcher.py
│   │   │   ├── error_analyst.py    # agentic ReAct
│   │   │   └── success_analyst.py  # single-pass
│   │   └── stage3_consolidate/
│   │       ├── merger.py           # hierarchical merge
│   │       ├── guardrails.py       # 3 layer deterministic
│   │       └── diff_translator.py  # JSON patch → unified diff
│   ├── signal/
│   │   ├── judge.py                # Generic LLMJudge
│   │   ├── rubric.py               # Rubric loader + validator
│   │   └── evidence.py             # Evidence dataclass + aggregator
│   ├── pipeline.py                 # Top-level Trace2Skill orchestrator
│   └── prompts/                    # Tất cả prompt template, tách khỏi code
│       ├── error_analyst.md
│       ├── success_analyst.md
│       ├── merge_operator.md
│       └── judge.md
│
├── adapters/                       # mỗi adapter = package con (pip install trace2skill-<name>)
│   ├── claude-code/
│   ├── langchain/
│   ├── opencode/
│   ├── cline/
│   ├── custom-react/
│   └── openai-assistants/
│
├── llm-providers/
│   ├── anthropic/              # Claude (first-class)
│   ├── openai/                 # GPT (first-class)
│   ├── gemini/                 # Gemini (first-class)
│   ├── openai-compatible/      # OpenRouter, DeepSeek, Groq, Together, Qwen API, xAI, ... (1 wrapper)
│   ├── vllm/                   # optional, self-host
│   └── ollama/                 # optional, local dev
│
├── skill-formats/
│   ├── anthropic/
│   ├── claude-code/
│   └── cline/
│
├── evidence-adapters/
│   ├── claude-code/
│   ├── langsmith/
│   └── generic-filesystem/
│
├── cli/                            # `trace2skill evolve ...`
│   └── trace2skill_cli/
│
├── examples/
│   ├── 01_replicate_paper_spreadsheet/
│   ├── 02_claude_code_basic/
│   ├── 03_claude_code_semi_online/
│   ├── 04_langchain_rag_agent/
│   └── 05_custom_project_template/
│
├── rubrics/                        # thư viện rubric dùng chung
│   ├── code_editing_agent.yaml
│   ├── qa_agent.yaml
│   ├── search_agent.yaml
│   ├── spreadsheet_agent.yaml
│   └── generic.yaml
│
├── docs/                           # docusaurus hoặc mkdocs
├── tests/
│   ├── unit/
│   ├── integration/
│   └── e2e/
├── benchmarks/
│   └── paper_replication/
├── pyproject.toml                  # uv / poetry / hatch
├── README.md
├── CONTRIBUTING.md
├── LICENSE (MIT)
└── .github/
    └── workflows/                  # CI: lint, type, test, publish

Tooling stack đề xuất:

  • Package manager: uv (fastest, modern)
  • Type check: pyright strict mode
  • Linter: ruff (format + lint)
  • Testing: pytest + pytest-asyncio
  • Docs: mkdocs-material
  • CI: GitHub Actions

5. Core data models

5.1 Task (input cho pipeline)

@dataclass
class Task:
    task_id: str
    query: str
    inputs: dict[str, Path]            # ví dụ {"workbook": "input.xlsx"}
    ground_truth: GroundTruth          # flexible: file | value | callable
    metadata: dict                     # domain-specific

5.2 Skill

@dataclass
class Skill:
    root_md: str                       # SKILL.md content
    resources: dict[str, bytes]        # {relative_path: content}
    
    def freeze(self) -> FrozenSkill: ...   # immutable view, dùng cho Stage 2
    def apply_patch(self, diff: UnifiedDiff) -> Skill: ...
    def validate_format(self) -> list[str]: ...

@dataclass(frozen=True)
class FrozenSkill:
    """Read-only view pass cho analysts để đảm bảo không drift."""
    root_md: str
    resources: Mapping[str, bytes]

5.3 Trajectory

@dataclass
class Trajectory:
    task_id: str
    query: str
    steps: list[ReActStep]
    final_answer: str
    y: int                             # 0 fail, 1 success (từ evaluator)
    artifacts: dict[str, Path]         # output files
    model: str
    metadata: dict                     # tokens, duration, turn count

@dataclass
class ReActStep:
    turn: int
    reasoning: str
    action: ToolCall
    observation: str

5.4 Patch

@dataclass
class Patch:
    patch_id: str
    source_traj_ids: list[str]         # Stage 2: len=1; Stage 3 merged: len>1
    source_type: Literal["error", "success", "merged"]
    rationale: str                     # cho merge operator đọc
    ops: list[DiffOp]

@dataclass
class DiffOp:
    op: Literal["create", "modify", "delete"]
    path: str                          # relative to skill_dir
    hunks: list[Hunk] | None
    content: str | None                # cho op="create"

@dataclass
class Hunk:
    start_line: int                    # 1-indexed, inclusive
    end_line: int                      # inclusive
    old_content: str                   # verify before replace
    new_content: str

5.5 Evidence + Judgment (cho signal layer)

@dataclass
class Evidence:
    """Raw signals từ session thật. KHÔNG parse, KHÔNG phân loại."""
    trajectory: Trajectory
    next_user_turns: list[str]         # user nói gì sau (raw)
    behavioral: dict                   # git diff, file mtime, ...
    execution: dict                    # tool outputs, exit codes
    explicit_feedback: dict | None     # 👍/👎 nếu có
    session_metadata: dict

@dataclass
class Judgment:
    y: int | None                      # 1, 0, hoặc None (insufficient)
    confidence: float                  # 0.0 - 1.0
    reasoning: str                     # LLM's explanation
    evidence_used: list[str]           # which signals mattered
    failure_modes: list[str]           # nếu y=0

6. Core protocols (plugin interfaces)

# core/protocols.py

class HarnessAdapter(Protocol):
    """Bridge tới agent harness cụ thể."""
    def run_task(
        self,
        query: str,
        skill_dir: Path,
        workspace: Path,
        turn_budget: int,
    ) -> Trajectory: ...

class LLMProvider(Protocol):
    """Cho analyst + merger + judge dùng."""
    def complete(self, prompt: str, thinking: bool = False) -> str: ...
    def complete_structured(self, prompt: str, schema: type[T]) -> T: ...
    def react(self, ctx: list[Message], tools: list[Tool]) -> Action: ...

class SkillFormat(Protocol):
    """Đọc/ghi skill theo format trên disk."""
    def load(self, path: Path) -> Skill: ...
    def save(self, skill: Skill, path: Path) -> None: ...

class EvidenceAdapter(Protocol):
    """Thu thập raw evidence từ session của harness cụ thể."""
    def collect(self, session_id: str) -> Evidence: ...

class Evaluator(Protocol):
    """Domain-specific scoring (cho batch mode có ground truth)."""
    def evaluate(self, task: Task, trajectory: Trajectory) -> EvalResult: ...

7. Development Phases (16 tuần)

Phase 0 — Foundation (Tuần 1) 🏗️

Goal: Repo setup + core abstractions, chưa có functionality.

Tasks:

  • Tạo monorepo với uv workspaces
  • Setup CI: lint (ruff), type check (pyright), test (pytest)
  • Viết core/models.py — tất cả dataclass
  • Viết core/protocols.py — 4 Protocol classes
  • Viết core/config.py — Pydantic model cho trace2skill.yaml
  • Viết README, CONTRIBUTING, LICENSE
  • Setup GitHub Actions cho test + type check
  • Dev docs: "How to add new adapter" template

Exit criteria:

  • uv sync && pytest pass
  • pyright --strict không error
  • README rõ ràng về mission + architecture

Deliverable: Skeleton repo public trên GitHub.

Phase 1 — Core Pipeline MVP (Tuần 2-3) ⚙️

Goal: Stage 1-2-3 chạy được với MockHarnessAdapter + MockLLMProvider.

Week 2 tasks:

  • MockHarnessAdapter — sinh trajectory synthetic
  • MockLLMProvider — trả về prompt + canned response
  • Stage1RolloutEngine — parallel rollout (async)
  • Checkpoint system — dump trajectory JSONL
  • Unit tests Stage 1

Week 3 tasks:

  • ErrorAnalyst — agentic ReAct loop (với mock tools)
    • Phải có quality gate: nếu không verify được cause → drop
  • SuccessAnalyst — single-pass
  • AnalystDispatcher — ThreadPool với W=128
  • HierarchicalMerger — tree merge với B_merge=32
  • Guardrails — 3 layer deterministic:
    1. File existence check
    2. Line-range conflict detection (intra-patch)
    3. Format validator (post-apply trial)
  • DiffTranslator — JSON patch → unified diff
  • Pipeline orchestrator với full checkpoint
  • Prompt templates:
    • prompts/error_analyst.md
    • prompts/success_analyst.md
    • prompts/merge_operator.md
  • E2E test với synthetic data

Exit criteria:

  • Chạy pipeline.evolve(S_0, D_evolve, mode="deepening") end-to-end
  • Output patch apply được, skill result valid format
  • Checkpoint JSONL đọc lại được

Deliverable: Core pipeline functional với mock data.

Phase 2 — Signal Layer (Tuần 4) 🎯

Goal: Generic LLMJudge — zero regex, rubric-driven.

Tasks:

  • Evidence dataclass — canonical form
  • Rubric YAML schema + Pydantic validator
  • LLMJudge class:
    • Structured output qua complete_structured
    • Ensemble mode (multi-sample) optional
    • Cost tracking
  • Rubric mẫu:
    • rubrics/code_editing_agent.yaml
    • rubrics/qa_agent.yaml
    • rubrics/search_agent.yaml
    • rubrics/spreadsheet_agent.yaml
    • rubrics/generic.yaml
  • prompts/judge.md — generic prompt template
  • Unit tests với evidence synthetic đa ngôn ngữ
  • Lint rule: ban regex trong trace2skill/signal/ (ast-grep)

Exit criteria:

  • Judge label được test evidence tiếng Việt, English, sarcasm
  • y=None khi evidence thiếu
  • Zero regex detect được bằng lint rule

Deliverable: Signal layer functional + 5 rubric mẫu.

Phase 3 — First Real Harness: Claude Code (Tuần 5-6) 🤖

Goal: End-to-end trên harness thật, không còn mock.

Week 5:

  • AnthropicLLMProvider — wrap anthropic SDK
    • Hỗ trợ prompt caching (quan trọng cho cost)
    • Thinking mode support
    • Structured output via tool use
  • AnthropicSkillFormat — load/save SKILL.md + resources/
  • ClaudeCodeHarnessAdapter:
    • Dùng Claude Agent SDK
    • Hoặc spawn subprocess với claude CLI
    • Parse message history → Trajectory
  • Test: chạy 5-10 task đơn giản

Week 6:

  • ClaudeCodeEvidenceAdapter:
    • Parse ~/.claude/projects/<proj>/*.jsonl
    • Extract user messages, tool results, git state
    • Build Evidence dataclass
  • CLI: trace2skill evolve --config ./trace2skill.yaml
  • Config file schema hoàn chỉnh
  • Example: examples/02_claude_code_basic/
    • 20 task mẫu đơn giản
    • Evaluator Python function
    • Full config + README
  • Integration tests

Exit criteria:

  • trace2skill evolve trên 20 task thật → skill tốt hơn baseline
  • Transcript parser không crash trên session phức tạp
  • README example reproduce được

Deliverable: First real adapter + example chạy được.

Phase 4 — Paper Replication (Tuần 7-8) 📊

Goal: Verify framework đúng paper, reproducible benchmark — qua API, không cần GPU.

Week 7:

  • CustomReActHarnessAdapter — ReAct loop generic (dùng OpenAICompatibleProvider)
  • Model access qua API thay vì vLLM local:
    • Qwen-3.5-35B qua OpenRouter hoặc DashScope
    • Model "lớn" qua OpenRouter: Qwen-3.5-122B nếu available, hoặc equivalent (DeepSeek V3, Llama 3.3 405B)
    • Pin version cụ thể để reproducible
  • Spreadsheet domain:
    • Load SpreadsheetBench-Verified
    • SpreadsheetEvaluator — compare xlsx files (openpyxl diff)
    • Reproduce Anthropic official xlsx skill làm S_0 baseline
  • Script: benchmarks/paper_replication/run_deepening.py
  • Cost estimate + budget cap (set max_usd_per_run trước khi chạy)

Week 8:

  • Chạy experiment (scale theo budget, không scale theo GPU):
    • Deepening +Error / +Success / +Combined
    • Creation +Error / +Success / +Combined
  • Cross-model transfer eval (model nhỏ ↔ model lớn, cả hai qua API)
  • OOD eval trên WikiTableQuestions
  • Results table so với paper Table 1
  • Caveat report: document lệch với paper do API variance (rate limit, stochastic sampling, version pinning)
  • Blog post / writeup về replication

Exit criteria:

  • Delta từ baseline trong ±3pp so với paper (relaxed từ ±2pp vì API variance)
  • Full audit trail: trajectories, patches, merge tree, judgment log, cost log
  • Nếu hết budget trước khi chạy full → công bố subset nhỏ hơn có methodology đúng

Deliverable: Reproducibility artifact (API-based) + public benchmark results.

Note: Phase này là "nice to have". Nếu busy / hết budget → postpone, framework vẫn ship được sau Phase 3.

Phase 5 — Multi-harness Expansion (Tuần 9-12) 🔌

Goal: Mở rộng ecosystem, verify plugin design thực sự generic.

Week 9: LangChain

  • LangChainHarnessAdapter — dùng AgentExecutor
  • LangChainEvidenceAdapter — qua Callbacks + LangSmith API
  • Example: examples/04_langchain_rag_agent/

Week 10: Multi-provider (OpenAI / Gemini / OpenAI-compat)

  • OpenAIProvider — GPT family
  • GeminiProvider — Gemini family (Google Generative AI SDK)
  • OpenAICompatibleProvider — generic wrapper (1 class) cover OpenRouter, DeepSeek, Groq, Together, Qwen API, xAI, Mistral API, ...
  • OpenAIAssistantsAdapter — cho Assistants API users
  • Cross-provider integration tests: cùng 1 task, 3 provider (Anthropic / OpenAI / Gemini) → skill evolved tương đương (schema & quality)

Week 11: Cline + OpenCode

  • ClineHarnessAdapter — CLI wrapper
  • ClineSkillFormat — .clinerules converter
  • OpenCodeHarnessAdapter
  • Converter: Anthropic skill ↔ Claude Code CLAUDE.md ↔ Cline rules

Week 12: Optional providers + polish

  • OllamaProvider (optional tier, cho local dev zero-cost)
  • VLLMProvider (optional tier, cho ai self-host)
  • Adapter template docs (cho community)
  • Performance + cost benchmarks từng provider (throughput, $/task, p50/p95 latency)
  • Bug fixes + UX polish

Exit criteria:

  • 3+ harness mới chạy được từ config-only
  • Adapter mới viết < 150 dòng
  • Core package không đổi khi thêm adapter

Deliverable: Multi-harness ecosystem + community adapter template.

Phase 6 — Semi-online Closed Loop (Tuần 13-14) 🔄

Goal: Production-ready continuous improvement.

Week 13:

  • SessionLogger middleware (harness-agnostic)
  • Session store abstraction:
    • FilesystemStore (JSONL)
    • SQLiteStore
    • PostgresStore (optional)
  • EvolutionTrigger:
    • Cron-based
    • Threshold-based (N sessions)
    • Manual trigger
  • Integration với Signal Layer (Phase 2)

Week 14:

  • Skill versioning + rollback
  • Deploy workflow:
    • Atomic skill swap
    • A/B testing (2 skill versions song song)
    • Metrics: acceptance rate, failure modes
  • Monitoring dashboard (simple HTML + SQLite)
  • Example: examples/03_claude_code_semi_online/
    • Setup từ 0 trong 10 phút
    • Tự động evolve weekly

Exit criteria:

  • Deploy trên 1 dự án thật, chạy 1 tuần không crash
  • Skill version 2 > version 1 (đo bằng acceptance rate)
  • Rollback < 30 giây

Deliverable: Production-ready closed loop.

Phase 7 — Release & Ecosystem (Tuần 15-16) 🚀

Goal: Public release, documentation, community.

Week 15: Docs

  • Full docs site (mkdocs-material)
    • Getting started (5 phút)
    • Architecture overview
    • Per-harness tutorials (Claude Code, LangChain, Cline, custom)
    • Rubric writing guide
    • Adapter contribution guide
  • Tutorial series (5-7 bài, mỗi bài tự chứa)
  • Video demo (optional)

Week 16: Release

  • PyPI publish: trace2skill + adapters
  • Benchmark leaderboard (dynamic, auto-updated)
  • Contributing guide + issue templates
  • Discord/GitHub Discussion setup
  • Launch post (HN, Twitter, r/LocalLLaMA)
  • Outreach tới LangChain/Cline/OpenCode maintainers

Exit criteria:

  • pip install trace2skill works
  • Docs đầy đủ, reproducible
  • 100+ GitHub stars trong 2 tuần post-launch

Deliverable: Public launch.

8. Decision points (quyết trước khi bắt đầu)

Quyết đã có

Decision Choice Lý do
Repo model Monorepo Refactor dễ, shared CI, release đồng bộ
Timeline 16 tuần Đủ test kỹ từng phase
License MIT Max reach, compatible mọi ecosystem
OSS Từ đầu Build trust sớm, attract contributors

Chưa quyết (cần suy nghĩ khi về nhà)

Question Options Khuyến nghị
First harness target? Claude Code / Custom ReAct / LangChain Claude Code — dễ test, SDK sẵn
Paper replication priority? Phase 4 sớm / Phase 4 muộn Sớm — verify framework đúng trước khi mở rộng
LLM judge default model? Claude Haiku 4.5 / Gemini 2.5 Flash / GPT-4.1-mini Haiku 4.5 — rẻ, prompt caching, đủ cho binary judgment
Analyst default model? Claude Sonnet 4.6 / Gemini 2.5 Pro / GPT-4.1 Sonnet 4.6 — balance cost/quality + structured output tốt
Paper replication model? OpenRouter Qwen / DeepSeek V3 / Claude OpenRouter Qwen — gần paper nhất mà không cần GPU
Package manager? uv / poetry / hatch uv — nhanh nhất, modern
Python version min? 3.11 / 3.12 3.11 (bạn đang dùng)
Async framework? asyncio / trio asyncio — ecosystem lớn hơn
Storage default? SQLite / JSONL / Postgres SQLite — đơn giản, zero-config

Bạn tự hỏi mình khi về nhà

  1. Budget API cho Phase 4 bao nhiêu? Rough estimate: $50-200 cho full replication qua OpenRouter. Set max_usd_per_run trước khi chạy.
  2. Dự án cá nhân nào test đầu tiên? SmartSearch v3, hay Excel personal, hay cái khác?
  3. Monetization path (nếu có)? Open-core? Hosted service? Consulting? Hay pure OSS?
  4. Maintainer plan? Solo? Rủ thêm người? Accept contribution khi nào?
  5. Provider mặc định cho examples/ có thể khác tùy audience — tutorial Claude Code dùng Anthropic, tutorial generic dùng OpenAI-compatible (dễ access nhất).

9. Risks & Mitigations

Risk Severity Likelihood Mitigation
Paper results không replicate (API variance) High Medium Phase 4 riêng, ±3pp tolerance, document API vs local differences. Community verify bằng GPU setup riêng
Judge confidence không reliable High Medium Multi-sample ensemble + explicit feedback override + manual spot-check
Claude Agent SDK breaking changes Medium Medium Lock version trong adapter, abstract qua HarnessAdapter
Cost API (analyst + judge + rollout) tăng nhanh High High Prompt caching (Anthropic/OpenAI) + Haiku cho judge + batch + filter low-value session + budget cap cứng
API rate limit / provider outage Medium Medium Multi-provider fallback config, exponential backoff, checkpoint để resume, retry với provider khác
Adapter sprawl → maintenance Medium Medium Community-maintained Tier 3, clear ownership per adapter
Skill format conflict Low Medium Canonical internal form + lossless converter
Competition (Hermes, concurrent papers) Medium High Differentiate: harness-agnostic + audit trail + paper replication. Đánh mạnh "plug vào agent bạn đang dùng"
Low community adoption High Medium Phase 7 docs + launch kỹ, example đa dạng, tutorial dễ follow
Personal time constraint High Medium 16 tuần có slack. Phase 4 có thể skip nếu bận. Focus MVP (Phase 0-3) trước.

10. Success metrics

Technical

  • Paper replication: ±2pp trên SpreadsheetBench-Verified (ít nhất 1 setup)
  • Skill transfer: evolved skill từ model A cải thiện model B trên ≥50% test cases
  • Adapter LOC: <150 dòng cho adapter mới
  • Core test coverage: >85%
  • E2E pipeline latency: <10 phút cho 50 task (với 32 workers)
  • Zero regex detect được trong signal layer (lint rule)

Adoption (post-launch)

  • Month 1: 10+ users deploy, 50+ GitHub stars
  • Month 3: 3+ external contributors, 2+ community adapters
  • Month 6: 500+ stars, 1+ integration vào major framework
  • Month 12: 1000+ stars, talk tại 1 conference

Quality

  • Every patch traceable: trajectory → analyst → merge level → final patch
  • All prompts version-controlled với changelog
  • Audit bundle reproducible: re-run với same seeds → same result

11. First day checklist (khi về nhà bắt đầu)

# 1. Tạo repo
mkdir trace2skill && cd trace2skill
git init

# 2. Setup uv workspace
uv init --package trace2skill
# ... thêm workspaces cho adapters/, llm-providers/, etc.

# 3. Tạo structure
mkdir -p trace2skill/{core,stages/stage2_analyze,stages/stage3_consolidate,signal,prompts}
mkdir -p adapters llm-providers skill-formats evidence-adapters cli examples rubrics docs tests/{unit,integration,e2e}

# 4. Viết pyproject.toml với base deps (CORE ONLY, không LLM SDK)
# - pydantic (models + config)
# - pyyaml (rubric)
# - anyio hoặc asyncio
# Note: LLM SDK (anthropic, openai, google-genai, ...) nằm trong từng llm-providers/ package,
# KHÔNG trong core. Core chỉ biết Protocol, không import provider SDK.

# 5. Viết core/models.py dựa trên section 5 của plan
# 6. Viết core/protocols.py dựa trên section 6
# 7. Setup ruff + pyright + pytest

# 8. First commit
git add . && git commit -m "feat: initial scaffolding (Phase 0)"

# 9. Push lên GitHub (repo mới, public, MIT license)
gh repo create trace2skill --public --license MIT
git push -u origin main

Câu hỏi phải trả lời trước khi code:

  • Repo tên là gì? (trace2skill đơn giản nhất)
  • GitHub org/user nào?
  • CI: GitHub Actions hay CircleCI? → GA
  • Python min version: 3.11
  • Package manager: uv

12. Key references

Paper (phải in ra đọc kỹ lại)

  • Trace2Skill full paper: https://arxiv.org/abs/2603.25158
  • Section 2 (Method): 3-stage pipeline chi tiết — đọc kỹ
  • Section 4.1: Parallel vs Sequential comparison — lý do không làm online
  • Section 4.3: Agentic vs LLM-only analyst — lý do Error Analyst phải agentic
  • Appendix B: Prompt templates — dùng làm baseline cho prompts/

Related work phải đọc

  • Anthropic Skills: https://docs.anthropic.com (skill framework, progressive disclosure)
  • DSPy: https://github.com/stanfordnlp/dspy (analogous: optimize prompts)
  • TextGrad: textual gradient optimization
  • Reasoning Bank (ouyang2026): retrieval-based baseline so sánh ở §4.2
  • Anthropic skill-creator: closest manual counterpart

Tools to learn/refresh

  • Claude Agent SDK (cho Claude Code adapter)
  • Anthropic / OpenAI / google-genai Python SDK (cho 3 provider first-class)
  • OpenAI-compatible API spec (1 wrapper cover cả chục provider)
  • OpenRouter docs (cho paper replication + multi-model access)
  • LangChain Callbacks + LangSmith (cho LangChain adapter)
  • uv workspaces (cho monorepo)
  • Pydantic v2 (cho config + rubric validation)

Inspiration (không copy)

  • Hermes Agent — online sequential (đối lập với approach này)
  • AutoSkill, XSkill, EvoSkill — concurrent online evolution work
  • SkillRL — co-evolution skill+policy (đối lập với frozen-model approach)

13. Notes / reminders

Những điểm KHÔNG được quên

  1. Error Analyst = agentic ReAct, không phải LLM call đơn. Phải có tools:
    • inspect_file(path)
    • read_ground_truth()
    • run_fix_candidate(patch)
    • diff_vs_gt()
  2. Analyst phải nhận FrozenSkill, không phải skill đã edit dần. Nếu nhầm → sequential drift → mất lợi thế parallel.
  3. Merger prompt PHẢI instruct "keep edits appearing ≥2 times". Đây là cơ chế inductive reasoning chính.
  4. 3 guardrail PHẢI deterministic (programmatic), không phải LLM check. Paper §2.4.
  5. Drop trajectory mà Error Analyst không verify được cause. Quality gate quan trọng hơn quantity.
  6. Main SKILL.md cho universal rules, references/ cho case-specific quirks. Hierarchical disclosure — paper §4.4.
  7. Signal layer: ZERO regex, zero if/else sentiment. Chỉ LLM judge + rubric YAML.
  8. Evidence thô → judge → judgment. Không preprocess trong adapter. Adapter chỉ extract, không phân loại.

Những cái bẫy thường gặp

  1. Overfit prompt: viết prompt cho 1 domain → fail trên domain khác. Test cross-domain từ đầu.
  2. Cache LLM response sai: phải cache theo full prompt hash, không chỉ task_id.
  3. Race condition trong parallel analyst: dùng immutable FrozenSkill, không share mutable state.
  4. Diff apply không atomic: phải trial apply → validate → real apply. Rollback nếu fail.
  5. Skip audit log vì "đang dev": đừng. Audit trail là selling point.

Idea cho future (Phase 8+, sau khi release)

  • Skill library: shared repo của skill đã evolved cho common tasks
  • Federated evolution: nhiều user cùng evolve 1 skill, merge contributions
  • Skill linting: static analysis cho skill quality trước khi deploy
  • Cost optimizer: pick model nhỏ nhất đủ cho task
  • Hybrid human-AI curation: Anthropic skill-creator + Trace2Skill loop
  • Non-English rubrics: hỗ trợ judge tiếng Việt/Nhật/Hàn native
  • Skill A/B testing framework: compare v1 vs v2 trên production traffic

14. Contact / ownership

  • Owner: (bạn fill vào)
  • Repo: (github.com//trace2skill — fill khi tạo)
  • Docs: (sẽ fill ở Phase 7)
  • Discord: (optional, Phase 7)