Skip to content

Add realtime multimodal token usage - #54

Closed
wjinxu wants to merge 3 commits into
OpenBMB:mainfrom
wjinxu:feat/realtime-usage
Closed

Add realtime multimodal token usage#54
wjinxu wants to merge 3 commits into
OpenBMB:mainfrom
wjinxu:feat/realtime-usage

Conversation

@wjinxu

@wjinxu wjinxu commented Jul 28, 2026

Copy link
Copy Markdown

Summary

  • Add session-cumulative usage and per-chunk chunk_usage to realtime duplex events.
  • Track text, audio, image, video, LLM output, and TTS output tokens.
  • Add chunk_index to correlate text/audio events produced by the same model chunk.
  • Keep cached_tokens at 0.
  • Include final cumulative usage in session lifecycle events.
  • Add missing TTS sampling defaults required by the Python backend.

Testing

  • 58 related unit tests passed.
  • Real audio realtime E2E passed (USAGE_E2E_OK).
  • Real video realtime E2E passed (VIDEO_USAGE_E2E_OK).
  • Verified cumulative usage equals session initialization usage plus chunk usage.

Expose per-chunk and session-cumulative usage so realtime clients can track text, audio, image, and video tokens throughout a duplex session.
@wjinxu
wjinxu force-pushed the feat/realtime-usage branch from e83b804 to 4f9de39 Compare July 29, 2026 05:31
@chmod777john

chmod777john commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

这个是否有按照 readme 里面的 docker 方式打包过镜像, docker compose 启动,然后用 examples 文件夹里面那些 probe 脚本端对端 api 调用测试过。包括视频双工和音频双工。

可以测试一下然后让 ai 写份报告粘贴在此处看看

Capture session totals and deduplicated per-chunk usage so audio and video probes expose API token accounting.
@wjinxu

wjinxu commented Jul 30, 2026

Copy link
Copy Markdown
Author

测试报告:
按 README 的 Docker 方式构建并启动了 Gateway、Worker 和 Python Backend。
使用 audio_probe.py 和 video_probe.py 通过真实 /v1/realtime API 完成音频、视频双工测试。
两种模式均成功返回文本和音频。
usage、chunk_usage、chunk_index 均正确显示,token 总数与各模态明细一致,cached_tokens=0。

Keep TTS sampling behavior outside the realtime usage scope so the change only reports existing token generation.
@wjinxu wjinxu closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants