|
| 1 | +--- |
| 2 | +mode: 'agent' |
| 3 | +description: 'Review and improve the automatic vs intentional tool classification in src/automaticTools.json.' |
| 4 | +tools: ['read_file', 'write_file', 'search_files'] |
| 5 | +--- |
| 6 | + |
| 7 | +# Review Automatic Tool Classification |
| 8 | + |
| 9 | +Audit `src/automaticTools.json` against `src/toolNames.json` to ensure every tool is classified correctly. The classification directly affects fluency/maturity scoring in the extension — tools in `automaticTools.json` are considered **automatic** (Copilot invokes them on its own) and are **excluded** from the user's intentional tool usage score. Tools that are not in that list are counted as **intentional** (the user deliberately enables or configures them). |
| 10 | + |
| 11 | +## Classification Rules |
| 12 | + |
| 13 | +Apply the rules below in priority order. When a tool matches multiple rules, the first matching rule wins. |
| 14 | + |
| 15 | +### Rule 1 — MCP tools are ALWAYS intentional (never automatic) |
| 16 | + |
| 17 | +Any tool whose ID starts with `mcp_`, `mcp.`, or `mcp__` is a user-configured external MCP server tool. These must **never** appear in `automaticTools.json`. |
| 18 | + |
| 19 | +- Examples: `mcp_github_search_code`, `mcp_io_github_git_create_pull_request`, `mcp__workspace__web_fetch` |
| 20 | +- Reason: the user must explicitly install and configure an MCP server; Copilot does not add them autonomously. |
| 21 | + |
| 22 | +### Rule 2 — Write / modify / execute tools are intentional |
| 23 | + |
| 24 | +Tools that write to disk, run commands, install packages, or otherwise have side effects outside the AI context window are **intentional**: |
| 25 | + |
| 26 | +| Category | Examples | |
| 27 | +|---|---| |
| 28 | +| File editing | `edit_file`, `edit_files`, `copilot_editFiles`, `copilot_writeFile`, `copilot_applyPatch`, `copilot_replaceString`, `copilot_multiReplaceString`, `insert_edit_into_file`, `replace_string_in_file`, `multi_replace_string_in_file`, `write`, `edit`, `multiedit`, `apply_patch` | |
| 29 | +| File creation/deletion | `create_file`, `create_directory`, `remove_file`, `copilot_createFile`, `copilot_createDirectory` | |
| 30 | +| Terminal execution | `run_in_terminal`, `run_build`, `run_command_in_terminal`, `copilot_runInTerminal`, `run_vscode_command`, `copilot_runVscodeCommand`, `kill_terminal`, `run_task`, `create_and_run_task`, `copilot_createAndRunTask` | |
| 31 | +| Test / notebook execution | `runTests`, `copilot_runTests1`, `run_notebook_cell`, `copilot_runNotebookCell`, `copilot_editNotebook` | |
| 32 | +| Package / environment management | `install_extension`, `copilot_installExtension`, `install_python_packages`, `create_virtual_environment`, `configure_python_environment` | |
| 33 | +| Symbol refactoring | `vscode_renameSymbol` | |
| 34 | +| Browser interaction | `click_element`, `navigate_page`, `open_browser_page`, `type_in_page`, `run_playwright_code` | |
| 35 | +| Todo writing | `todowrite`, `TodoWrite`, `manage_todo_list` | |
| 36 | +| Web fetching | `fetch_webpage`, `copilot_fetchWebPage`, `webfetch`, `websearch`, `WebSearch`, `vscode-websearchforcopilot_webSearch` | |
| 37 | + |
| 38 | +### Rule 3 — Read / search / inspect tools are automatic |
| 39 | + |
| 40 | +Tools that only read, list, or search without side effects are **automatic**. Copilot calls them transparently during exploration: |
| 41 | + |
| 42 | +| Category | Examples | |
| 43 | +|---|---| |
| 44 | +| File reading | `read_file`, `read`, `view`, `copilot_readFile`, `get_file`, `get_currentfile` | |
| 45 | +| Directory listing | `list_dir`, `ls`, `copilot_listDirectory`, `listFiles` | |
| 46 | +| File / text search | `find_files`, `file_search`, `file_glob_search`, `glob`, `grep`, `grep_search`, `copilot_findFiles`, `copilot_findTextInFiles`, `copilot_findTestFiles`, `test_search` | |
| 47 | +| Code / symbol search | `semantic_search`, `code_search`, `copilot_searchCodebase`, `copilot_getSearchResults`, `get_search_view_results`, `search_workspace_symbols`, `copilot_searchWorkspaceSymbols`, `vscode_listCodeUsages`, `get_symbols_by_name` | |
| 48 | +| Errors & diagnostics | `get_errors`, `copilot_getErrors`, `test_failure`, `copilot_testFailure` | |
| 49 | +| Project metadata | `read_project_structure`, `copilot_readProjectStructure`, `get_project_setup_info`, `copilot_getProjectSetupInfo`, `get_files_in_project`, `get_projects_in_solution`, `get_python_executable_details` | |
| 50 | +| Doc / API info | `get_doc_info`, `copilot_getDocInfo`, `get_vscode_api`, `copilot_getVSCodeAPI` | |
| 51 | +| Changed files | `get_changed_files`, `copilot_getChangedFiles` | |
| 52 | +| Notebook / image reading | `read_notebook_cell_output`, `copilot_readNotebookCellOutput`, `copilot_getNotebookSummary`, `view_image`, `copilot_viewImage` | |
| 53 | +| Terminal output reading | `get_task_output`, `job_output`, `get_terminal_output`, `await_terminal`, `terminal_selection`, `terminal_last_command` | |
| 54 | +| Memory / context | `memory`, `copilot_memory`, `detect_memories` | |
| 55 | +| Tool introspection | `tool_replay`, `copilot_toolReplay`, `tool_search` | |
| 56 | +| Confirmations | `vscode_get_confirmation`, `vscode_get_confirmation_with_options`, `vscode_get_terminal_confirmation`, `vscode_get_modified_files_confirmation` | |
| 57 | +| Clarification questions | `ask_questions`, `AskUserQuestion`, `copilot_askQuestions`, `vscode_askQuestions` | |
| 58 | +| Agent switching | `switch_agent`, `copilot_switchAgent`, `setup.agent` | |
| 59 | +| Internal VS Code variants | `vscode_editFile_internal`, `vscode_fetchWebPage_internal`, `vscode_searchExtensions_internal` | |
| 60 | +| Skill reading | `read_skill`, `skill` | |
| 61 | +| Diagram rendering | `renderMermaidDiagram` | |
| 62 | +| Task subagents | `search_subagent`, `runSubagent` | |
| 63 | + |
| 64 | +### Rule 4 — Single-word tool heuristic (tie-breaker for unknowns) |
| 65 | + |
| 66 | +When a tool ID is a **single word** (no `_`, `.`, `-`) and its name suggests read/lookup behaviour (e.g. `read`, `view`, `grep`, `glob`, `ls`), default to **automatic**. Single-word tools that clearly write or execute (e.g. `write`, `edit`, `task`) are **intentional**. |
| 67 | + |
| 68 | +## Procedure |
| 69 | + |
| 70 | +1. Load `src/toolNames.json` — this is the complete set of known tool IDs. |
| 71 | +2. Load `src/automaticTools.json` — the current automatic tool set. |
| 72 | +3. For **every** tool ID in `toolNames.json`: |
| 73 | + a. Apply Rules 1–4 to decide: **automatic** or **intentional**. |
| 74 | + b. If the tool should be **automatic** and is **missing** from `automaticTools.json`, add it. |
| 75 | + c. If the tool is currently in `automaticTools.json` but should be **intentional** (e.g. it matches Rule 1 or 2), remove it. |
| 76 | +4. Write the updated array back to `src/automaticTools.json`, preserving the existing comment-style grouping (blank lines between logical groups). |
| 77 | + |
| 78 | +## Output |
| 79 | + |
| 80 | +- Modify only `src/automaticTools.json`. |
| 81 | +- Do **not** open a PR. |
| 82 | +- After writing the file, print a concise summary: |
| 83 | + - Number of tools **added** (with IDs) |
| 84 | + - Number of tools **removed** (with IDs and the reason) |
| 85 | + - Total tools in the updated file |
0 commit comments