Skip to content

fix(ai): capture an error event when a streaming call fails to connect - #1146

Open
pcbeingused333 wants to merge 3 commits into
PostHog:mainfrom
pcbeingused333:fix/stream-connect-error-capture
Open

pcbeingused333 wants to merge 3 commits into
PostHog:mainfrom
pcbeingused333:fix/stream-connect-error-capture

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

For streaming calls the OpenAI chat, OpenAI Responses, Anthropic and Gemini wrappers call the underlying create / generate_content_stream outside the try/finally that captures the event, and only wrap the iteration of the returned stream. A connection-time failure (401 bad key, 429 rate limit, invalid model) is raised before any generator exists, so no $ai_generation event is sent; only errors raised mid-stream are recorded. Non-streaming calls already record these errors through call_llm_and_track_usage.

The streaming call is now wrapped in try/except Exception: on failure the existing _capture_streaming_event is called with an empty stream state and error=exc (which sets $ai_is_error and $ai_http_status), and the exception is re-raised unchanged. Sync and async, all four providers. No issue is linked; the gap is visible from the code path and reproduced by the tests below.

💚 How did you test it?

Eight tests in posthog/test/ai/test_stream_midstream_error.py (OpenAI chat, OpenAI Responses, Anthropic and Gemini, sync and async) make the underlying call raise a 401 error and assert the captured event has $ai_is_error and $ai_http_status == 401; all fail on main (no event captured) and pass with the change. pytest posthog/test/ai: full suite green; ruff check and ruff format --check clean. Not run against live provider APIs.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed. (No docs change needed.)
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Changeset file added: .sampo/changesets/stream-connect-error.md (pypi/posthog: patch).

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

The change and the tests were written by Claude Code (Anthropic's agent) working for the PR author, who directs this contribution work and is the DRI. The agent compared the streaming and non-streaming paths of each wrapper, wrote the failing tests first, then wrapped only the initial call so the per-chunk handling is untouched. No manual testing against the providers was done.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Alex Castillo <alex.castillog33@gmail.com>
@pcbeingused333
pcbeingused333 requested a review from a team as a code owner October 29, 2026 10:49

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated advisory code review.

response = super().create(**kwargs)
except Exception as exc:
accumulator = _AnthropicStreamAccumulator()
self._capture_streaming_event(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: Preserve the provider exception when error telemetry cannot format rejected input — When Anthropic rejects malformed input such as messages=[None] with HTTP 400, the newly added capture call formats that same rejected input before reaching the bare raise. format_anthropic_input calls msg.get(...), raising AttributeError and replacing the original anthropic.BadRequestError; the async handler at posthog/ai/anthropic/anthropic_async.py:169 behaves identically. Previously callers received the actionable provider error and could handle BadRequestError normally. Ensure failures in this telemetry path cannot replace the original exception. Reproduction: reproduced — python -m pytest -p no:cacheprovider posthog/test/ai/test_review_connect.py -k invalid_anthropic --timeout=30 -vv fails both sync/async assertions on the reviewed head with AttributeError: 'NoneType' object has no attribute 'get', while both pass unchanged at the recorded merge base.

@marandaneto
marandaneto requested a review from a team October 29, 2026 11:42

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated advisory code review.

kwargs,
usage_stats,
time.time() - start_time,
accumulated_content,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: Keep the new Gemini handlers compatible with Mypy — Passing the initially empty, unannotated accumulated_content into this new exception handler makes Mypy require an explicit element type before the nested generator populates it. Both the sync and async additions introduce var-annotated errors and fail the required Mypy check; please annotate the accumulated content in both wrappers. Reproduction: reproduced — python -m mypy --no-site-packages --config-file mypy.ini posthog/ai/gemini/gemini.py posthog/ai/gemini/gemini_async.py reports exactly two annotation errors on the reviewed head and no issues on the recorded merge base.

@marandaneto
marandaneto requested a review from a team October 29, 2026 14:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants