[pcbeingused333]

💡 Motivation and Context

The posthog-js LangChain handler records $ai_web_search_count, and so do the Python OpenAI, Anthropic and Gemini wrappers, but the Python LangChain callback handler never did. When Anthropic reports usage.server_tool_use.web_search_requests (as a dict or an object), _parse_usage_model now reads it into a new web_search_count field of ModelUsage, and the handler adds $ai_web_search_count to the event, including a count of 0. Nothing is added when the provider reports no count. langchain-anthropic passes the raw Anthropic usage (with server_tool_use) in llm_output for non-streamed calls, which is where the handler reads it; streamed calls only carry the standard usage_metadata, so no count is recorded for them. Only this exact count is handled; posthog-js also infers a single search from citations or grounding metadata for other providers, which is not ported here.

💚 How did you test it?

• TestWebSearchCount in posthog/test/ai/langchain/test_callbacks.py: a response with two web search requests, one with zero, and one without server_tool_use. The first two fail on master and pass with the change; the third guards the omitted case.
• posthog/test/ai/langchain/test_callbacks.py: 83 passed, 8 skipped. ruff format --check passes on the touched directories. The rest of the suite and a live LangChain call with the Anthropic web search tool were not run.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed. (n/a)
• No breaking change or entry added to the changelog.

If releasing new changes

• Ran sampo add to generate a changeset file (added .sampo/changesets/langchain-web-search-count.md)

🤖 Agent context

Autonomy: Fully autonomous

• The gap was found and the change, tests and description were written by Claude Code (Claude Sonnet 5.5) by comparing the Python and JS LangChain handlers. Agent-authored, needs human review.

🤖 Generated with Claude Code

🤖 Generated with Claude Code

[marandaneto]

Automated advisory code review.

[marandaneto]

@PostHog/team-ai-observability do we need any backend/frontend change for this?

[carlos-marchal-ph]

@PostHog/team-ai-observability do we need any backend/frontend change for this?

No, but there were some issues with the implementation that I surfaced on my review

[pcbeingused333]

Thanks, all four points addressed in the latest commit:

• $ai_web_search_count is now emitted only when the count is greater than 0, like posthog/ai/utils.py.
• server_tool_use is read from llm_output["usage"] separately in _parse_usage and merged into whichever ModelUsage wins for tokens, so it is no longer lost when usage_metadata is set on the message.
• The three tests are now one parametrized test_web_search_count using the module-level imports and the mock_client fixture.
• The fixture sets usage_metadata={"input_tokens": 10, "output_tokens": 5, "total_tokens": 15}; with the callbacks.py change reverted the two-searches case fails, and with it the test passes.