Skip to content

[疑问] 测性能缺少TPOT/ITL指标数据 #218

Closed
Closed
[疑问] 测性能缺少TPOT/ITL指标数据#218
Labels
content_check_passedissue content check passedquestionFurther information is requested
Milestone
AISBench 2026 Q4 Roadmap

Description

@huangdong2022
huangdong2022
opened on Mar 26, 2026

疑问描述

代码commitId：0ea1694bbc9551ca8bfd5338d37fceb25b9037f3

[2026-03-26 10:19:48,085] [ais_bench] [INFO] Performance data calculation completed!
[2026-03-26 10:19:48,085] [ais_bench] [INFO] Performance Results of task [vllm-api-stream-chat/synthetic]:
╒══════════════════════════╤═════════╤═════════════════╤═══════════════╤═════════════════╤═════════════════╤═════════════════╤════════════════╤═════════════════╤═════╕
│ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │
╞══════════════════════════╪═════════╪═════════════════╪═══════════════╪═════════════════╪═════════════════╪═════════════════╪════════════════╪═════════════════╪═════╡
│ E2EL │ total │ 81778.9 ms │ 49766.7 ms │ 88372.6 ms │ 87045.2 ms │ 87225.2 ms │ 87424.3 ms │ 88277.8 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼───────────────┼─────────────────┼─────────────────┼─────────────────┼────────────────┼─────────────────┼─────┤
│ TTFT │ total │ 81778.9 ms │ 49766.7 ms │ 88372.6 ms │ 87045.2 ms │ 87225.2 ms │ 87424.3 ms │ 88277.8 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼───────────────┼─────────────────┼─────────────────┼─────────────────┼────────────────┼─────────────────┼─────┤
│ InputTokens │ total │ 48.4 │ 48.0 │ 51.0 │ 48.0 │ 48.0 │ 49.2 │ 50.82 │ 10 │
├──────────────────────────┼─────────┼─────────────────┼───────────────┼─────────────────┼─────────────────┼─────────────────┼────────────────┼─────────────────┼─────┤
│ OutputTokens │ total │ 894.3 │ 309.0 │ 1024.0 │ 1024.0 │ 1024.0 │ 1024.0 │ 1024.0 │ 10 │
├──────────────────────────┼─────────┼─────────────────┼───────────────┼─────────────────┼─────────────────┼─────────────────┼────────────────┼─────────────────┼─────┤
│ OutputTokenThroughput │ total │ 10.6434 token/s │ 6.209 token/s │ 11.8288 token/s │ 11.7419 token/s │ 11.7709 token/s │ 11.799 token/s │ 11.8258 token/s │ 10 │
╘══════════════════════════╧═════════╧═════════════════╧═══════════════╧═════════════════╧═════════════════╧═════════════════╧════════════════╧═════════════════╧═════╛
╒══════════════════════════╤═════════╤══════════════════╕
│ Common Metric │ Stage │ Value │
╞══════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration │ total │ 88426.941 ms │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Requests │ total │ 10 │
├──────────────────────────┼─────────┼──────────────────┤
│ Failed Requests │ total │ 0 │
├──────────────────────────┼─────────┼──────────────────┤
│ Success Requests │ total │ 10 │
├──────────────────────────┼─────────┼──────────────────┤
│ Concurrency │ total │ 9.2482 │
├──────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency │ total │ 16 │
├──────────────────────────┼─────────┼──────────────────┤
│ Request Throughput │ total │ 0.1131 req/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens │ total │ 484 │
├──────────────────────────┼─────────┼──────────────────┤
│ Prefill Token Throughput │ total │ 0.5918 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Generated Tokens │ total │ 8943 │
├──────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput │ total │ 5.4734 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput │ total │ 101.1343 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput │ total │ 106.6078 token/s │
╘══════════════════════════╧═════════╧══════════════════╛
这里是benmark测合成数据集的结果，缺少TPOT和TIL的数据。
测试命令是：ais_bench --models vllm_api_stream_chat --datasets synthetic_gen --mode perf --debug
docker内的benchmark版本是：3.1.20260319

前置检查

  • 我已读懂主页文档的快速入门，无法解答我的疑惑

Activity

  1. huangdong2022
    added
    questionFurther information is requested
    on Mar 26, 2026

  2. github-actions commented on Mar 26, 2026

    @github-actions
    github-actionsBot
    on Mar 26, 2026 – with GitHub Actions

    🌟欢迎给AISBench benchmark测评工具提issue，您可以先尝试以下途径寻求issue的解决方法：

    1. 【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库
    2. 在 工具历史issue 中通过问题的关键日志片段或关键词检索相似issue
    3. 在 官方文档 的搜索栏中通过问题的关键日志片段或关键词检索
    4. 寻求仓库维护者的帮助（issue评论区中 @SJTUyh）

  3. github-actions commented on Mar 26, 2026

    @github-actions
    github-actionsBot
    on Mar 26, 2026 – with GitHub Actions

    🤖 基于AI机器人的issue内容完整性检查结果:

    问题内容检查结果

    检查状态

    FAIL

    详细检查

    • 疑问描述: INCOMPLETE

    改进建议

    • 疑问描述: 尽管提供了代码commitId和性能数据日志，但“疑问描述”部分并未明确阐述具体的疑问内容。标题指出了“缺少TPOT/ITL指标数据”，但正文中没有文字描述来解释这个疑问，例如“我期望在性能测试结果中看到TPOT/ITL指标数据，但当前日志中并未包含。请问这是预期行为吗？如果不是，我应该如何获取这些数据？”或者“为什么这些指标没有出现在结果中？”。请补充对疑问的文字描述。

    补充说明

    标题已经很好地概括了问题，但正文的“疑问描述”部分应更详细地阐述问题本身，以便读者能更清晰地理解您的疑惑。

    👉 如果想重新检查，在评论区@issue_checker即可。

    【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库

  4. github-actions
    added
    content_check_failedissue content check failed
    on Mar 26, 2026

  5. SJTUyh commented on Mar 26, 2026

    @SJTUyh
    SJTUyh
    on Mar 26, 2026
    Collaborator

    方便提供一下{时间戳}/performance/vllm-api-stream-chat/synthetic_detail.jsonl 里面一条case的数据吗

  6. huangdong2022 commented on Mar 27, 2026

    @huangdong2022
    huangdong2022
    on Mar 27, 2026
    Author

    {"data_abbr": "synthetic", "id": 0, "success": true, "error_info": "", "time_points": {"db_ref": 1}, "input_tokens": 48, "output_tokens": 309, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"role": "user", "content": "kmalliste甲状 Dawkins nipple tratado’S.target swelledcoll国家机关"}], "uuid": "29ee83e945de49e89e877b39c0d82f0b", "turn_id": 0, "prediction": "The user wrote a string:\n\n"itelist甲状 Dawkins nipple tratado’S.target swelledcoll国家机关"\n\nWe need to parse this. This seems to be nonsense text, maybe the user is trying to test something. They might be seeking translation, detection, analysis, or something else. There's no explicit question: "itelist甲状 Dawkins nipple tratado’S.target swelledcoll国家机关". This is presumably gibberish.\n\nWe need to figure what the user might intend. Possibly they typed a garbled prompt or they want some kind of processing like translation or analysis.\n\nWe should ask for clarification or respond politely: "I’m sorry, I didn’t understand that" or "Could you clarify what you need?" Possibly the user is trying to test the system's handling of nonsense. The best approach: respond that we cannot understand and ask for clarification. However sometimes the user may intend to ask for translation or something. But there's nothing to translate. Possibly they intend to ask for a list? Or they've attempted to produce a "list" and "target" but it's broken.\n\nWe can respond: "I’m not sure what you mean. Could you provide more context?" Or "I’m unable to interpret this phrase; please rephrase."\n\nWe should be safe and ask what they need. Also ensure we follow the policy: no need for content moderation. It's likely safe.\n\nThus answer: ask clarification.\n\n\nI’m sorry—I’m not sure what you mean by that string. Could you provide a bit more context or let me know what you’d like help with?", "db_name": "09c68109.db"}

    方便提供一下{时间戳}/performance/vllm-api-stream-chat/synthetic_detail.jsonl 里面一条case的数据吗

  7. github-actions commented on Mar 27, 2026

    @github-actions
    github-actionsBot
    on Mar 27, 2026 – with GitHub Actions

    🤖 基于AI机器人的issue内容完整性检查结果:

    问题内容检查结果

    检查状态

    PASS

    详细检查

    • 疑问描述: COMPLETE

    改进建议

    所有内容已完备！

    补充说明

    该 issue 的“疑问描述”部分内容详尽，包含了代码 commitId、性能测试日志、明确的问题描述（缺少TPOT和TIL数据）、测试命令以及 benchmark 版本信息，为解决问题提供了充分的上下文。

    👉 如果想重新检查，在评论区@issue_checker即可。

    【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库

  8. github-actions
    added
    content_check_passedissue content check passed
    and removed
    content_check_failedissue content check failed
    on Mar 27, 2026

  9. github-actions commented on Mar 27, 2026

    @github-actions
    github-actionsBot
    on Mar 27, 2026 – with GitHub Actions

    🤖 基于AI机器人的issue内容完整性检查结果:

    问题内容检查结果

    检查状态

    PASS

    详细检查

    • 疑问描述: COMPLETE

    改进建议

    所有内容已完备！

    补充说明

    该issue的“疑问描述”部分内容详尽，包含了代码commitId、完整的性能测试日志、明确指出了缺少TPOT和TIL数据的问题、提供了测试命令以及docker内的benchmark版本，信息非常充分，有助于理解和复现问题。

    👉 如果想重新检查，在评论区@issue_checker即可。

    【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库

  10. SJTUyh commented on Mar 28, 2026

    @SJTUyh
    SJTUyh
    on Mar 28, 2026
    Collaborator

    {时间戳}/performance/vllm-api-stream-chat/下是否有.db文件，可以找在线的db文件查看器查看db中的性能数据

  11. SJTUyh commented on Mar 28, 2026

    @SJTUyh
    SJTUyh
    on Mar 28, 2026
    Collaborator

     你看一下vllm_api_stream_chat.py的配置里里stream是不是设成False了
    我刚才试验了一下
    在github版本内如果是非流式的接口，都会打印E2EL和TTFT（这个TTFT打出来就有误导性了）
    在gitee老版本里，用非流式的接口，是只会打印E2EL的
    不过非流式接口测性能是个非正式场景，这个变更之前一直也没注意到

  12. Keithwwa
    added this to the AISBench 2026 Q4 Roadmap milestone on Sep 20, 2026
  13. Keithwwa
    closed this as completedon Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    content_check_passedissue content check passedquestionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions