Skip to content

[perf][prefix cache] optimize prefix cache planning for large scenarios - #596

Open
jschen069 wants to merge 7 commits into
AISBench:masterfrom
jschen069:1014_fix_prefix
Open

[perf][prefix cache] optimize prefix cache planning for large scenarios#596
jschen069 wants to merge 7 commits into
AISBench:masterfrom
jschen069:1014_fix_prefix

Conversation

@jschen069

@jschen069 jschen069 commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。

PR Type / PR类型

  • Feature（功能新增）
  • Bugfix（Bug 修复）
  • Docs（文档更新）
  • CI/CD（持续集成/持续部署）
  • Refactor（代码重构）
  • Perf（性能优化）
  • Dependency（依赖项更新）
  • Test-Cases（测试用例更新）
  • Other（其他）

Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)

🔍 Motivation / 变更动机

Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。

prefix cache插件生成大规格数据集时，复杂度较高：
原来复杂度较高，为NNB*C
N：请求数量，例如当前场景是 1818。
B：每轮保留的 beam 状态数量，默认最多 256。
C：每条请求可选择的前缀 block 数，近似为平均 cap + 1
新增代码

大规模场景主要包含：

  • 遍历请求并建立 lane：O(N)
  • water-fill 比例分配：固定次数的线性扫描
  • 余数排序：O(N log N)
  • 前缀水位回填：O(N)
  • 理论命中率模拟：O(N)
    O(N log N)

另外降低函数复杂度，保证代码具有可维护性。

📝 Modification / 修改内容

Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。

代码变更文档设计：
https://github.com/jschen069/benchmark/wiki/Prefix-Cache-%E4%BC%98%E5%8C%96%E4%BB%A3%E7%A0%81%E8%AF%B4%E6%98%8E

📐 Associated Test Results / 关联测试结果

Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。

跑大规格数据集生成

scenario.json配置

{
  "schema_version": "1.0",
  "run": {
    "run_id": "gsm8k-prefix-cache-60",
    "random_seed": 42,
    "output_dir": "./outputs/gsm8k-prefix-cache-60",
    "overwrite": false
  },
  "tokenizer": {
    "path": "/home/weights/GLM-5.3",
    "block_size": 128,
    "revision": null,
    "trust_remote_code": false
  },
  "corpus": {
    "path": "ais_bench/datasets/gsm8k/test.jsonl",
    "field": "question",
    "selection": {"mode": "random"}
  },
  "requests": {
    "count": 1818,
    "input_length": {"mode": "explicit", "values": [29580,29628,29677,29726,29775,29823,29872,29921,29969,30018,30067,30115,30164,30213,30261,30310,30359,30407,30456,30505,30554,30602,30651,30700,30748,30797,30846,30894,30943,30992,31040,31089,31138,31186,31234,31284,31332,31381,31430,31478,31527,31576,31624,31673,31722,31770,31819,31868,31917,31965,32013,32063,32111,32160,32209,32257,32306,32355,32403,32452,32501,32549,32598,32647,32695,32744,32793,32840,32890,32939,32987,33036,33085,33133,33182,33231,33279,33328,33377,33426,33474,33523,33572,33620,33668,33718,33766,33815,33864,33912,33961,34010,34058,34107,34156,34204,34253,34302,34350,34399,34447,34496,34545,34594,34642,34691,34740,34788,34837,34886,34935,34983,35032,35081,35129,35178,35227,35274,35324,35373,35421,35470,35519,35567,35616,35665,35713,35762,35811,35859,35908,35957,36005,36053,36103,36151,36200,36249,36297,36346,36395,36444,36492,36541,36590,36638,36687,36736,36784,36833,36881,36930,36979,37028,37076,37125,37174,37222,37271,37320,37368,37417,37466,37514,37563,37612,37660,37708,37758,37806,37855,37904,37952,38001,38050,38099,38147,38196,38245,38293,38342,38391,38439,38487,38537,38585,38634,38683,38731,38780,38829,38877,38926,38975,39023,39072,39121,39169,39218,39267,39314,39364,39413,39461,39510,39559,39608,39656,39705,39754,39802,39851,39900,39948,39997,40046,40093,40143,40192,40240,40289,40338,40386,40435,40484,40532,40581,40630,40678,40727,40776,40824,40873,40921,40970,41019,41068,41117,41165,41214,41263,41311,41360,41409,41457,41506,41555,41603,41652,41701,41748,41798,41847,41895,41944,41993,42041,42090,42139,42187,42236,42285,42333,42382,42431,42479,42527,42577,42626,42674,42723,42772,42820,42869,42918,42966,43015,43064,43112,43161,43210,43258,43307,43355,43404,43453,43502,43550,43599,43648,43696,43745,43794,43842,43891,43940,43988,44037,44086,44134,44183,44232,44281,44329,44378,44427,44475,44524,44573,44621,44670,44719,44767,44816,44865,44913,44961,45011,45059,45108,45157,45205,45254,45303,45351,45400,45449,45497,45546,45595,45644,45692,45741,45789,45838,45887,45936,45984,46033,46082,46130,46179,46228,46276,46325,46374,46422,46471,46520,46567,46617,46666,46714,46763,46812,46860,46909,46958,47006,47055,47104,47153,47201,47250,47299,47347,47395,47445,47493,47542,47591,47639,47688,47737,47785,47834,47883,47931,47980,48029,48077,48126,48174,48223,48272,48321,48369,48418,48467,48515,48564,48613,48662,48710,48759,48808,48856,48905,48954,49001,49051,49100,49148,49197,49246,49294,49343,49392,49440,49489,49538,49586,49635,49684,49732,49781,49829,49878,49927,49976,50024,50073,50122,50171,50219,50268,50317,50365,50414,50463,50511,50560,50608,50657,50706,50755,50803,50852,50901,50949,50998,51047,51095,51144,51193,51241,51290,51339,51387,51435,51485,51533,51582,51631,51680,51728,51777,51826,51874,51923,51972,52020,52069,52118,52166,52214,52264,52312,52361,52410,52458,52507,52556,52604,52653,52702,52750,52799,52848,52896,52945,52994,53041,53091,53140,53188,53237,53286,53335,53383,53432,53481,53529,53578,53627,53675,53724,53773,53821,53869,53919,53967,54016,54065,54113,54162,54211,54259,54308,54357,54405,54454,54503,54551,54600,54648,54697,54746,54795,54844,54892,54941,54990,55038,55087,55136,55184,55233,55282,55330,55379,55428,55475,55525,55574,55622,55671,55720,55768,55817,55866,55914,55963,56012,56060,56109,56158,56206,56254,56304,56353,56401,56450,56499,56547,56596,56645,56693,56742,56791,56839,56888,56937,56985,57034,57082,57131,57180,57229,57277,57326,57375,57423,57472,57521,57569,57618,57667,57715,57764,57813,57862,57909,57959,58008,58056,58105,58154,58202,58251,58300,58348,58397,58446,58494,58543,58592,58640,58688,58738,58786,58835,58884,58932,58981,59030,59078,59127,59176,59224,59273,59322,59371,59419,59468,59516,59565,59614,59663,59711,59760,59809,59857,59906,59955,60003,60052,60101,60149,60198,60247,60295,60343,60393,60441,60490,60539,60587,60636,60685,60733,60782,60831,60880,60928,60977,61026,61074,61122,61172,61220,61269,61318,61366,61415,61464,61512,61561,61610,61658,61707,61756,61804,61853,61902,61949,61999,62048,62096,62145,62194,62242,62291,62340,62389,62437,62486,62535,62583,62632,62681,62728,62778,62827,62875,62924,62973,63021,63070,63119,63167,63216,63265,63313,63362,63411,63459,63508,63556,63605,63654,63703,63751,63800,63849,63898,63946,63995,64044,64092,64141,64190,64238,64287,64336,64383,64433,64482,64530,64579,64628,64676,64725,64774,64822,64871,64920,64968,65017,65066,65114,65162,65212,65260,65309,65358,65407,65455,65504,65553,65601,65650,65699,65747,65796,65845,65893,65942,65990,66039,66088,66137,66185,66234,66283,66331,66380,66429,66477,66526,66575,66623,66672,66721,66768,66818,66867,66915,66964,67013,67062,67110,67159,67208,67256,67305,67354,67402,67451,67500,67548,67596,67646,67694,67743,67792,67840,67889,67938,67986,68035,68084,68132,68181,68230,68278,68327,68376,68423,68473,68522,68571,68619,68668,68717,68765,68814,68863,68911,68960,69009,69057,69106,69155,69202,69252,69301,69349,69398,69447,69495,69544,69593,69641,69690,69739,69787,69836,69885,69933,69982,70030,70080,70128,70177,70226,70274,70323,70372,70420,70469,70518,70566,70615,70664,70712,70761,70809,70858,70907,70956,71004,71053,71102,71150,71199,71248,71296,71345,71394,71442,71491,71540,71589,71636,71686,71735,71783,71832,71881,71929,71978,72027,72075,72124,72173,72221,72270,72319,72367,72416,72464,72513,72562,72611,72659,72708,72757,72805,72854,72903,72951,73000,73049,73098,73146,73195,73243,73292,73341,73390,73438,73487,73536,73584,73633,73682,73730,73779,73875,73970,74066,74161,74257,74352,74448,74543,74639,74735,74830,74926,75022,75117,75213,75308,75404,75499,75595,75691,75786,75882,75977,76072,76168,76264,76360,76455,76551,76646,76742,76838,76933,77029,77124,77220,77315,77411,77507,77602,77697,77793,77889,77984,78080,78176,78271,78367,78462,78558,78653,78749,78845,78940,79036,79131,79227,79322,79418,79514,79609,79705,79800,79896,79992,80087,80183,80278,80374,80469,80565,80661,80756,80851,80947,81043,81139,81234,81330,81425,81521,81616,81712,81808,81903,81999,82094,82190,82285,82381,82476,82572,82668,82763,82859,82955,83050,83146,83241,83337,83432,83528,83624,83719,83815,83910,84005,84101,84197,84293,84388,84484,84579,84675,84771,84866,84962,85057,85153,85248,85344,85440,85535,85630,85726,85822,85917,86013,86109,86204,86300,86395,86491,86586,86682,86778,86873,86969,87064,87160,87255,87351,87447,87542,87638,87733,87829,87925,88020,88116,88211,88307,88402,88498,88594,88689,88784,88880,88976,89072,89167,89263,89358,89454,89549,89645,89741,89836,89932,90027,90123,90218,90314,90409,90505,90601,90696,90792,90888,90983,91079,91174,91270,91365,91461,91557,91652,91748,91843,91938,92034,92130,92226,92321,92417,92512,92608,92704,92799,92895,92990,93086,93181,93277,93373,93468,93563,93659,93755,93850,93946,94042,94137,94233,94328,94424,94519,94615,94711,94806,94902,94997,95093,95188,95284,95380,95475,95571,95666,95762,95858,95953,96049,96144,96240,96335,96431,96527,96622,96717,96813,96909,97005,97100,97196,97291,97387,97482,97578,97674,97769,97865,97960,98056,98151,98247,98342,98438,98534,98629,98725,98821,98916,99012,99107,99203,99298,99394,99490,99585,99681,99776,99871,99967,100063,100159,100254,100350,100445,100541,100637,100732,100828,100923,101019,101114,101210,101306,101401,101496,101592,101688,101783,101879,101975,102070,102166,102261,102357,102452,102548,102644,102739,102835,102930,103026,103121,103217,103313,103408,103504,103599,103695,103791,103886,103982,104077,104173,104268,104364,104460,104555,104650,104746,104842,104938,105033,105129,105224,105320,105415,105511,105607,105702,105798,105893,105989,106084,106180,106275,106371,106467,106562,106658,106754,106849,106945,107040,107136,107231,107327,107423,107518,107614,107709,107805,107899,107996,108092,108187,108283,108378,108474,108570,108665,108761,108856,108952,109047,109143,109239,109334,109429,109525,109621,109716,109812,109908,110003,110099,110194,110290,110385,110481,110577,110672,110768,110863,110959,111054,111150,111246,111341,111437,111532,111628,111724,111819,111915,112010,112106,112201,112297,112393,112488,112583,112679,112775,112871,112966,113062,113157,113253,113348,113444,113540,113635,113731,113826,113922,114017,114113,114208,114304,114400,114495,114591,114687,114782,114878,114973,115069,115164,115260,115356,115451,115547,115642,115738,115832,115929,116025,116120,116216,116311,116407,116502,116598,116694,116789,116885,116980,117076,117172,117267,117362,117458,117554,117649,117745,117841,117936,118032,118127,118223,118318,118414,118510,118605,118701,118796,118892,118987,119083,119179,119274,119370,119465,119561,119657,119752,119848,119943,120039,120134,120230,120326,120421,120516,120612,120708,120804,120899,120995,121090,121186,121281,121377,121473,121568,121664,121759,121855,121950,122046,122141,122237,122333,122428,122524,122620,122715,122811,122906,123002,123097,123193,123289,123384,123480,123575,123671,123765,123862,123958,124053,124149,124244,124340,124435,124531,124627,124722,124818,124913,125009,125105,125200,125295,125391,125487,125582,125678,125774,125869,125965,126060,126156,126251,126347,126443,126538,126634,126729,126825,126920,127016,127112,127207,127303,127398,127494,127590,127685,127781,127876,127972,128067,128163,128259,128354,128449,128545,128641,128737,128832,128928,129023,129119,129214,129310,129406,129501,129597,129692,129788,129883,129979,130074,130170,130266,130361,130457,130553,130648,130744,130839,130935,131030,131126,131222,131317,131413,131508,131604,131698,131795,131891,131986,132082,132177,132273,132368,132464,132560,132655,132751,132846,132942,133038,133133,133228,133324,133420,133515,133611,133707,133802,133898,133993,134089,134184,134280,134376,134471,134567,134662,134758,134853,134949,135045,135140,135236,135331,135427,135523,135618,135714,135809,135905,136000,136096,136192,136287,136382,136478,136574,136670,136765,136861,136956,137052,137147,137243,137339,137434,137530,137625,137721,137816,137912,138007,138103,138199,138294,138390,138486,138581,138677,138772,138868,138963,139059,139155,139250,139346,139441,139537,139631,139728,139824,139919,140015,140110,140206,140301,140397,140493,140588,140684,140779,140875,140971,141066,141161,141257,141353,141448,141544,141640,141735,141831,141926,142022,142117,142213,142309,142404,142500,142595,142691,142786,142882,142978,143073,143169,143360,143360,143624,143888,144152,144415,144679,144943,145207,145471,145735,145999,146262,146525,146790,147054,147318,147582,147845,148109,148373,148637,148901,149165,149429,149692,149956,150220,150484,150748,151011,151276,151539,151803,152067,152331,152595,152859,153123,153386,153650,153914,154178,154442,154706,154970,155233,155496,155761,156025,156289,156553,156816,157080,157344,157608,157872,158136,158400,158663,158927,159191,159455,159718,159983,160247,160510,160774,161038,161302,161566,161830,162094,162357,162621,162885,163149,163413,163677,163941,164203,164468,164732,164996,165260,165524,165787,166051,166315,166579,166843,167107,167371,167634,167898,168162,168425,168690,168954,169218,169481,169745,170009,170273,170537,170801,171065,171328,171592,171856,172120,172384,172648,172911,173175,173439,173703,173967,174231,174495,174758,175022,175286,175550,175814,176078,176342,176605,176869,177133,177396,177661,177925,178189,178452,178716,178980,179244,179508,179772,180036,180299,180563,180827,181091,181355,181618,181883,182146,182410,182674,182938,183202,183466,183729,183993,184257,184521,184785,185049,185313,185576,185840,186368,186368,186822,187278,187733,188188,188644,189099,189554,190009,190464,190919,191374,191829,192284,192740,193195,193650,194105,194560]},
    "output_length": {"mode": "fixed", "value": 1024}
  },
  "output": {
    "output_key": null
  },
  "prefix_cache": {
    "mode": "cold",
    "target_hit_rate": 0.89,
    "seed_blocks": 1,
    "minimum_non_shared_length": 128,
    "groups": {"count": 6, "assignment": {"mode": "weights", "weights": [7.26, 4.57, 4.46, 9.85, 36.30, 37.56]}, "overrides": {}},
    "order": {"strategy": "sequential"}
  },
  "service": {
    "inference_url": "http://127.0.0.1:8005/v1/completions",
    "metrics_url": "http://127.0.0.1:8005/metrics",
    "reset_url": "http://127.0.0.1:8005/reset_prefix_cache",
    "model": "qwen3",
    "dp_size": 1,
    "assume_empty_cache": false,
    "engine_label_map": {},
    "timeout_seconds": 30,
    "api_key": "",
    "poll_interval_seconds": 5.0
  },
  "validation": {"target_warning_pp": 1.0, "actual_warning_pp": 5.0},
  "aisbench": {
    "config": "./plugins/prefix_cache/config_examples/prefix_cache_perf.py",
    "work_dir": "./outputs/default",
    "extra_args": {"--num-warmups": 0},
    "dataset": {
      "abbr": null,
      "input_columns": ["question", "max_out_len"],
      "output_column": "answer",
      "prompt_template": "{question}",
      "pred_role": "BOT"
    },
    "model": {
      "abbr": null,
      "attr": "service",
      "stream": true,
      "max_out_len": 1,
      "retry": 2,
      "batch_size": 1,
      "generation_kwargs": {
        "temperature": 0,
        "ignore_eos": true
      }
    }
  },
  "multimodal": {
    "mmmu_parquet_dir": "../../../../MMMU",
    "scenarios": ["single_1080p"]
  }
}

执行结果：

root@hostname-tqptc:/workspace/benchmark# ais-bench-prefix-cache inspect --scenario ./scenario.json
{
  "run_id": "gsm8k-prefix-cache-60",
  "mode": "cold",
  "requested_target_hit_rate": 0.89,
  "effective_target_hit_rate": 0.8899996398449369,
  "theoretical_hit_rate": 0.8899996398449369,
  "reachable_min": 0.0,
  "reachable_max": 0.9948276821729792,
  "target_reachable": true,
  "trajectory": {
    "strategy": "scalable_lane_convergent",
    "status": "BEST_EFFORT",
    "issues": [
      "global cumulative hit rate is not monotonic",
      "global cumulative hit rate overshoots its final rate"
    ]
  },
  "group_reachability": {
    "group-0": {
      "reachable_min": 0.0,
      "reachable_max": 0.9858748879009815
    },
    "group-1": {
      "reachable_min": 0.0,
      "reachable_max": 0.9823733740169912
    },
    "group-2": {
      "reachable_min": 0.0,
      "reachable_max": 0.9825965983458596
    },
    "group-3": {
      "reachable_min": 0.0,
      "reachable_max": 0.9899738802933102
    },
    "group-4": {
      "reachable_min": 0.0,
      "reachable_max": 0.9952429366909438
    },
    "group-5": {
      "reachable_min": 0.0,
      "reachable_max": 0.996403457502522
    }
  },
  "groups": {
    "group-0": 132,
    "group-1": 83,
    "group-2": 81,
    "group-3": 179,
    "group-4": 660,
    "group-5": 683
  },
  "input_tokens": {
    "min": 29580,
    "max": 194560,
    "mean": 86016.0,
    "p50": 73827.0,
    "p90": 143360.0,
    "p95": 167146.59999999998,
    "p99": 186368.0,
    "bins": [
      {
        "min": 29580,
        "max": 46033,
        "count": 339
      },
      {
        "min": 46082,
        "max": 62535,
        "count": 339
      },
      {
        "min": 62583,
        "max": 79036,
        "count": 286
      },
      {
        "min": 79131,
        "max": 95571,
        "count": 173
      },
      {
        "min": 95666,
        "max": 112010,
        "count": 172
      },
      {
        "min": 112106,
        "max": 128545,
        "count": 173
      },
      {
        "min": 128641,
        "max": 144943,
        "count": 161
      },
      {
        "min": 145207,
        "max": 161566,
        "count": 63
      },
      {
        "min": 161830,
        "max": 177925,
        "count": 62
      },
      {
        "min": 178189,
        "max": 194560,
        "count": 50
      }
    ],
    "total": 156377088
  },
  "output_tokens": {
    "min": 1024,
    "max": 1024,
    "mean": 1024.0,
    "p50": 1024.0,
    "p90": 1024.0,
    "p95": 1024.0,
    "p99": 1024.0,
    "bins": [
      {
        "min": 1024,
        "max": 1024,
        "count": 1818
      }
    ],
    "total": 1861632
  },
  "dp_route_counts": {
    "0": 1818
  },
  "sends_requests": false,
  "planning_only": true,
  "content_validation_deferred_to_prepare": true,
  "log": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/log/gsm8k-prefix-cache-60_20261015_014024.inspect.log",
  "manifest": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/result/gsm8k-prefix-cache-60_20261015_014024.manifest.json"
}
root@hostname-tqptc:/workspace/benchmark# ais-bench-prefix-cache prepare --mode text --scenario ./scenario.json
Generate prompts [##############################] 1818/1818 100%
{"full": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/result/gsm8k-prefix-cache-60_20261015_014024.full.jsonl", "requests": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/result/gsm8k-prefix-cache-60_20261015_014024.requests.jsonl", "manifest": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/result/gsm8k-prefix-cache-60_20261015_014024.manifest.json", "analysis": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/result/gsm8k-prefix-cache-60_20261015_014024.analysis.json", "log": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261015_014024/log/gsm8k-prefix-cache-60_20261015_014024.prepare.log"}

gsm8k 生成固定数据集测试，

root@hostname-tqptc:/workspace/benchmark# ais-bench-prefix-cache inspect --scenario ./scenario.json
{
  "run_id": "gsm8k-prefix-cache-60",
  "mode": "cold",
  "requested_target_hit_rate": 0.6,
  "effective_target_hit_rate": 0.6041666666666666,
  "theoretical_hit_rate": 0.6041666666666666,
  "reachable_min": 0.0,
  "reachable_max": 0.7291666666666666,
  "target_reachable": true,
  "trajectory": {
    "strategy": "target_convergent",
    "status": "PASS",
    "issues": []
  },
  "group_reachability": {
    "group-0": {
      "reachable_min": 0.0,
      "reachable_max": 0.7291666666666666
    }
  },
  "groups": {
    "group-0": 6
  },
  "input_tokens": {
    "min": 1024,
    "max": 1024,
    "mean": 1024.0,
    "p50": 1024.0,
    "p90": 1024.0,
    "p95": 1024.0,
    "p99": 1024.0,
    "bins": [
      {
        "min": 1024,
        "max": 1024,
        "count": 6
      }
    ],
    "total": 6144
  },
  "output_tokens": {
    "min": 32,
    "max": 32,
    "mean": 32.0,
    "p50": 32.0,
    "p90": 32.0,
    "p95": 32.0,
    "p99": 32.0,
    "bins": [
      {
        "min": 32,
        "max": 32,
        "count": 6
      }
    ],
    "total": 192
  },
  "dp_route_counts": {
    "0": 6
  },
  "sends_requests": false,
  "planning_only": true,
  "content_validation_deferred_to_prepare": true,
  "log": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/log/gsm8k-prefix-cache-60_20261014_063323.inspect.log",
  "manifest": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.manifest.json"
}
root@hostname-tqptc:/workspace/benchmark# ais-bench-prefix-cache prepare --mode text --scenario ./scenario.json
Generate prompts [##############################] 6/6 100%
{"full": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.full.jsonl", "requests": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.requests.jsonl", "manifest": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.manifest.json", "analysis": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.analysis.json", "log": "/workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/log/gsm8k-prefix-cache-60_20261014_063323.prepare.log"}


root@hostname-tqptc:/workspace/benchmark# ais-bench-prefix-cache run --scenario ./scenario.json
[2026-10-14 06:37:36,560] [ais_bench] [INFO] Current exp folder: /workspace/benchmark/outputs/default/20261014_063719
[2026-10-14 06:37:36,623] [ais_bench] [INFO] Starting inference tasks...
[2026-10-14 06:37:36,625] [ais_bench] [INFO] Partitioned into 1 tasks.
[2026-10-14 06:37:36,625] [ais_bench] [INFO] Merging datasets with the same model and inferencer...
[2026-10-14 06:37:36,657] [ais_bench] [INFO] Launch TasksMonitor, PID: 4927, Refresh interval: 0.5, Run in background: False
[2026-10-14 06:37:36,658] [ais_bench] [INFO] Start launch task state board ...
+----------------------------------------------------------------------------------+-----------+-------------------------------------------------+-------------+----------+-------------------------------------------------------------------------------------------------+------------------------------------------------+
| Task Name                                                                        |   Process | Progress                                        | Time Cost   | Status   | Log Path                                                                                        | Extend Parameters                              |
+==================================================================================+===========+=================================================+=============+==========+=================================================================================================+================================================+
| gsm8k-prefix-cache-60_20261014_063323-vllm/gsm8k-prefix-cache-60_20261014_063323 |      4933 | [##############################] 6/6 [2.0 it/s] | 0:00:05     | finish   | logs/infer/gsm8k-prefix-cache-60_20261014_063323-vllm/gsm8k-prefix-cache-60_20261014_063323.out | {'POST': 6, 'RECV': 6, 'FINISH': 6, 'FAIL': 0} |
+----------------------------------------------------------------------------------+-----------+-------------------------------------------------+-------------+----------+-------------------------------------------------------------------------------------------------+------------------------------------------------+
[2026-10-14 06:38:02,694] [ais_bench] [INFO] Inference tasks completed.
[2026-10-14 06:38:02,696] [ais_bench] [INFO] Summarizing performance results...
Calculating performance details: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00,  6.61it/s]
/usr/local/python3.12.13/lib/python3.12/site-packages/plotly/offline/offline.py:150: UserWarning: 
Unrecognized config options supplied: ['showLink']
  warnings.warn(
[2026-10-14 06:38:05,168] [ais_bench] [INFO] Request timeline and concurrency chart saved to /workspace/benchmark/outputs/default/20261014_063719/performances/gsm8k-prefix-cache-60_20261014_063323-vllm/gsm8k-prefix-cache-60_20261014_063323_plot.html
[2026-10-14 06:38:05,168] [ais_bench] [INFO] The gsm8k-prefix-cache-60_20261014_063323_plot has been saved in /workspace/benchmark/outputs/default/20261014_063719/performances/gsm8k-prefix-cache-60_20261014_063323-vllm/gsm8k-prefix-cache-60_20261014_063323_plot.html
[2026-10-14 06:38:05,172] [ais_bench] [INFO] Converting performance results for stage...
[2026-10-14 06:38:05,172] [ais_bench] [INFO] Performance results conversion completed!
[2026-10-14 06:38:05,173] [ais_bench] [INFO] Starting metrics calculation...
[2026-10-14 06:38:05,180] [ais_bench] [INFO] Starting common metrics calculation...
[2026-10-14 06:38:05,181] [ais_bench] [INFO] Adding units to metrics...
[2026-10-14 06:38:05,181] [ais_bench] [INFO] Performance data calculation completed!
[2026-10-14 06:38:05,182] [ais_bench] [INFO] Performance Results of task [gsm8k-prefix-cache-60_20261014_063323-vllm/gsm8k-prefix-cache-60_20261014_063323]: 
╒══════════════════════════╤═════════╤═══════════════════╤═══════════════════╤════════════════════╤═══════════════════╤═══════════════════╤════════════════════╤════════════════════╤═════╕
│ Performance Parameters   │ Stage   │ Average           │ Min               │ Max                │ Median            │ P75               │ P90                │ P99                │  N  │
╞══════════════════════════╪═════════╪═══════════════════╪═══════════════════╪════════════════════╪═══════════════════╪═══════════════════╪════════════════════╪════════════════════╪═════╡
│ E2EL                     │ total   │ 585.7 ms          │ 573.1 ms          │ 624.4 ms           │ 578.5 ms          │ 583.5 ms          │ 604.6 ms           │ 622.4 ms           │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ TTFT                     │ total   │ 109.8 ms          │ 101.6 ms          │ 129.6 ms           │ 107.1 ms          │ 109.9 ms          │ 120.0 ms           │ 128.7 ms           │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ TPOT                     │ total   │ 15.4 ms           │ 15.2 ms           │ 16.0 ms            │ 15.2 ms           │ 15.3 ms           │ 15.6 ms            │ 15.9 ms            │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ ITL                      │ total   │ 14.9 ms           │ 0.0 ms            │ 23.6 ms            │ 15.1 ms           │ 15.2 ms           │ 15.4 ms            │ 18.1 ms            │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ InputTokens              │ total   │ 1024.0            │ 1024.0            │ 1024.0             │ 1024.0            │ 1024.0            │ 1024.0             │ 1024.0             │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ OutputTokens             │ total   │ 32.0              │ 32.0              │ 32.0               │ 32.0              │ 32.0              │ 32.0               │ 32.0               │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ OutputTokenThroughput    │ total   │ 54.6813 token/s   │ 51.2486 token/s   │ 55.8367 token/s    │ 55.3146 token/s   │ 55.5977 token/s   │ 55.748 token/s     │ 55.8279 token/s    │  6  │
├──────────────────────────┼─────────┼───────────────────┼───────────────────┼────────────────────┼───────────────────┼───────────────────┼────────────────────┼────────────────────┼─────┤
│ PrefillTokenThroughput   │ total   │ 9383.5244 token/s │ 7898.4354 token/s │ 10078.9048 token/s │ 9558.3414 token/s │ 9864.1433 token/s │ 10002.4379 token/s │ 10071.2581 token/s │  6  │
╘══════════════════════════╧═════════╧═══════════════════╧═══════════════════╧════════════════════╧═══════════════════╧═══════════════════╧════════════════════╧════════════════════╧═════╛
╒═════════════════════════╤═════════╤═══════════════════╕
│ Common Metric           │ Stage   │ Value             │
╞═════════════════════════╪═════════╪═══════════════════╡
│ Benchmark Duration      │ total   │ 3516.8899 ms      │
├─────────────────────────┼─────────┼───────────────────┤
│ Total Requests          │ total   │ 6                 │
├─────────────────────────┼─────────┼───────────────────┤
│ Failed Requests         │ total   │ 0                 │
├─────────────────────────┼─────────┼───────────────────┤
│ Success Requests        │ total   │ 6                 │
├─────────────────────────┼─────────┼───────────────────┤
│ Concurrency             │ total   │ 0.9993            │
├─────────────────────────┼─────────┼───────────────────┤
│ Max Concurrency         │ total   │ 1                 │
├─────────────────────────┼─────────┼───────────────────┤
│ Request Throughput      │ total   │ 1.7061 req/s      │
├─────────────────────────┼─────────┼───────────────────┤
│ Total Input Tokens      │ total   │ 6144              │
├─────────────────────────┼─────────┼───────────────────┤
│ Total Generated Tokens  │ total   │ 192               │
├─────────────────────────┼─────────┼───────────────────┤
│ Input Token Throughput  │ total   │ 1746.9981 token/s │
├─────────────────────────┼─────────┼───────────────────┤
│ Output Token Throughput │ total   │ 54.5937 token/s   │
├─────────────────────────┼─────────┼───────────────────┤
│ Total Token Throughput  │ total   │ 1801.5918 token/s │
╘═════════════════════════╧═════════╧═══════════════════╛
[2026-10-14 06:38:05,189] [ais_bench] [INFO] Performance Result files located in /workspace/benchmark/outputs/default/20261014_063719/performances/gsm8k-prefix-cache-60_20261014_063323-vllm.
[2026-10-14 06:38:09,534] [ais_bench_prefix_cache] [INFO] Prefix Cache Results of task [gsm8k-prefix-cache-60_20261014_063323]:
╒══════════════════════════════╤════════╕
│ Prefix Cache Metric          │ Value  │
╞══════════════════════════════╪════════╡
│ Overall Target Hit Rate      │ 60.00% │
├──────────────────────────────┼────────┤
│ Overall Theoretical Hit Rate │ 60.42% │
├──────────────────────────────┼────────┤
│ Overall Actual Hit Rate      │ 60.42% │
├──────────────────────────────┼────────┤
│ Theory vs Actual Difference  │ 0.00%  │
├──────────────────────────────┼────────┤
│ Theory vs Target Difference  │ 0.42%  │
╘══════════════════════════════╧════════╛
[INFO] Detailed analysis is available at: /workspace/benchmark/outputs/gsm8k-prefix-cache-60_20261014_063323/result/gsm8k-prefix-cache-60_20261014_063323.analysis.json

⚠️ BC-breaking (Optional) / 向后不兼容变更（可选）

Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。

⚠️ Performance degradation (Optional) / 性能下降（可选）

If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。

🌟 Use cases (Optional) / 使用案例（可选）

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。

✅ Checklist / 检查列表

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。
  • The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。
  • All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects. / 如果此拉取请求对下游或其他相关项目有潜在影响，应在那些项目中测试此 PR。
  • CLA has been signed and all committers have signed the CLA in this PR. / CLA 已签署，且本 PR 中的所有提交者均已签署 CLA。

👥 Collaboration Info / 协作信息

  • Suggested Reviewers / 建议审核人: @xxx
  • Relevant Module Owners / 相关模块负责人: @xxx
  • Other Collaboration Notes / 其他协作说明：

🌟 Useful CI Command / 实用的CI命令

Command / 命令 Introduction / 介绍
/gemini review Performs a code review for the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 执行代码审核。
/gemini summary Provides a summary of the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 提供摘要。
/gemini help Displays a list of available commands of Gemini. / 显示 Gemini 可用命令的列表。
/readthedocs build Triggers a build of the documentation for the current pull request in its current state by Read the Docs. / 触发当前拉取请求在当前状态下由 Read the Docs 构建文档。
/pr_check 手动重新触发 PR 合入质量检查工作流（PR Quality Check），并在 PR 上评论即可生效。 / Manually re-runs the PR Quality Check workflow by commenting on the pull request.

@jschen069
jschen069 deployed to smoke-test-approval October 14, 2026 07:54 — with GitHub Actions Active
@github-actions github-actions Bot added no_issue PR 未关联任何 Issue，合入前必须关联 Issue no_review PR 未包含任何检视意见，合入前必须包含检视意见 labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：no_issue、no_review

  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。
  • 无检视意见：请至少邀请一位 reviewer 提交检视意见（Approve / Request changes / Comment 均可）。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@deepsource-io

deepsource-io Bot commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in 5a63d93...3676b17 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

Important

Some issues found as part of this review are outside of the diff in this pull request and aren't shown in the inline review comments due to GitHub's API limitations. You can see those issues on the DeepSource dashboard.

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
Python Oct 16, 2026 8:43a.m. Review ↗
Shell Oct 16, 2026 8:43a.m. Review ↗
Secrets Oct 16, 2026 8:43a.m. Review ↗
Docker Oct 16, 2026 8:43a.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

deepsource-io[bot]
deepsource-io Bot reviewed Oct 14, 2026
View reviewed changes
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
@jschen069
jschen069 deployed to smoke-test-approval October 14, 2026 09:15 — with GitHub Actions Active
@github-actions github-actions Bot added large_pr PR 新增代码超过 1000 行，建议拆分为更小的 PR review_opened PR 中存在未闭环的检视意见，所有检视意见必须闭环后才能合入 and removed no_review PR 未包含任何检视意见，合入前必须包含检视意见 labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、no_issue、review_opened

  • 超大 PR：新增代码 1205 行，超过 1000 行上限，建议拆分。
  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。
  • 检视意见未闭环：还存在 8 条未 Resolve 的检视意见，请全部 Resolve 后再合入。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

deepsource-io[bot]
deepsource-io Bot reviewed Oct 14, 2026
View reviewed changes
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
deepsource-io[bot]
deepsource-io Bot reviewed Oct 14, 2026
View reviewed changes
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
Comment thread plugins/prefix_cache/ais_bench_prefix_cache/pipeline.py Outdated
@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、no_issue、review_opened

  • 超大 PR：新增代码 1232 行，超过 1000 行上限，建议拆分。
  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。
  • 检视意见未闭环：还存在 14 条未 Resolve 的检视意见，请全部 Resolve 后再合入。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@github-actions

github-actions Bot commented Oct 14, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、no_issue、review_opened

  • 超大 PR：新增代码 1318 行，超过 1000 行上限，建议拆分。
  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。
  • 检视意见未闭环：还存在 14 条未 Resolve 的检视意见，请全部 Resolve 后再合入。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@jschen069
jschen069 deployed to smoke-test-approval October 14, 2026 09:44 — with GitHub Actions Active
@github-actions github-actions Bot removed the review_opened PR 中存在未闭环的检视意见，所有检视意见必须闭环后才能合入 label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、no_issue

  • 超大 PR：新增代码 1318 行，超过 1000 行上限，建议拆分。
  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@jschen069
jschen069 deployed to smoke-test-approval October 15, 2026 07:00 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 15, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、no_issue

  • 超大 PR：新增代码 1337 行，超过 1000 行上限，建议拆分。
  • 未关联 Issue：请在 PR 描述中关联 Issue，例如 Fixes #123 或 Relates to #123。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@github-actions github-actions Bot added the perf label Oct 16, 2026
@jschen069 jschen069 changed the title Optimize prefix cache planning for large scenarios [Perf][prefix cache] optimize prefix cache planning for large scenarios Oct 16, 2026
@jschen069 jschen069 changed the title [Perf][prefix cache] optimize prefix cache planning for large scenarios [perf][prefix cache] optimize prefix cache planning for large scenarios Oct 16, 2026
@jschen069 jschen069 linked an issue Oct 16, 2026 that may be closed by this pull request
[issue] prefix cache性能优化，当生成大规格数据集时，缩短生成时间 #603
Open
3 of 4 tasks
@github-actions

github-actions Bot commented Oct 16, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr

  • 超大 PR：新增代码 1337 行，超过 1000 行上限，建议拆分。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

@github-actions github-actions Bot removed the no_issue PR 未关联任何 Issue，合入前必须关联 Issue label Oct 16, 2026
SJTUyh
SJTUyh reviewed Oct 16, 2026
View reviewed changes

@SJTUyh SJTUyh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 性能改造本身在“硬约束”上是安全的：solve_prefix_lengths 仍先算可达上下界、按 Block 取最近目标，并用与 Manifest 相同的模拟器校验最终命中 token 精确（_validated_prefix_theory），新增 scalable 路径的构造数学上也自洽（lane 内 uppers=[0]+caps[:-1]、prefix_i=h[i+1] 恰好复现 h，首个 lane 请求 hit=0），前缀长度仍是 block 对齐且不超过 length-minimum_non_shared，结果确定可复现。但确实存在多处“功能/语义层面的不兼容或退化”：(1) inspect 由“临时构造数据”改为纯配置级规划——不再加载/校验 tokenizer 与语料（tokenizer_loader 形参保留但被忽略），theoretical_hit_rate 改为计划级数值（与 effective_target_hit_rate 相同），语料缺失、tokenizer 不可加载、round-trip 失败只会在 prepare 才暴露；(2) 大规模场景的 shared_prefix_tokens 与工件内容按设计改变，且当 lane 内长度/容量不单调（如 global_shuffle / within_group_shuffle 排序）时 scalable 路径返回 None 并直接落到旧的 exact_lane_fallback，beam 因估算超阈值被跳过，平滑优化整体失效；(3) 新增的全局单调性判定会让多 lane cold 场景恒为 BEST_EFFORT（PR 自述的 1818 请求场景输出即为 BEST_EFFORT + 两条 issues），并连带把已有场景 analysis.json 的 validation.status 从 PASS 变成 PASS_WITH_WARNING；(4) 文档只补充了定性的“近线性/分 lane 构造”，没有给出任何规模—耗时规格（本次未能核实文档其他章节是否已有实测数据，也未能取到 pipeline.py 尾部完整源码，inspect 的其余改动是依据 diff 判断的）。

Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
"target_convergent",
)

estimated_transitions = 256 * sum(cap + 1 for cap in caps)

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 算法选择用一个与实现重复的魔数决定是否完全放弃 beam：estimated_transitions = 256 * sum(cap + 1 for cap in caps)（256 与 _target_convergent_cold_prefixes 的 beam_width 默认值重复），超过 2_000_000 就只走 scalable 路径。问题是 scalable 路径要求每个 lane 的 cap/输入长度单调不减（_cold_lane_layout 第 743-746 行），否则返回 None，随后 solve_prefix_lengths 会直接落到 exact_lane_fallback（第 1201-1207 行）——也就是本 PR 想淘汰的“前缀集中在前若干条再回填”的旧构造；而对 order.strategy: global_shuffle / within_group_shuffle（或 group 长度 override 让 lane 内长度抖动）的大规模配置，lane 通常不单调，因此大规模不规则场景既拿不到 scalable 平滑解，也不会再尝试 beam（文档只写了“小型不规则场景保留受限 beam search”，未覆盖这一类）。建议：scalable 失败时用更小 beam 宽度重试一次（或先按 lane 内长度排序再走 scalable），把 256/2_000_000 改为由 beam_width 派生的常量，并把回退原因（lane 非单调/容量不足/状态爆炸）写入日志与 trajectory.issues，便于用户区分“真的不可平滑”与“被阈值跳过”。

Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py Outdated
mode: str,
) -> list[str]:
"""Collect global and per-lane trajectory diagnostics."""
issues = _rate_trajectory_issues("global", rows)

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] _prefix_trajectory_issues 对全局累计命中率无条件做单调性/超调判定（第 1006-1009 行），但 cold 多 lane 场景中每条 lane 的首条请求按定义必然 miss（hit=0），只要请求顺序让多个 lane 交错，全局累计命中率就会出现回落、“超过最终值”的超调，于是 trajectory.status 恒为 BEST_EFFORT：PR 描述里给出的 1818 请求 / 6 group / dp_size=1 场景输出正是 status=BEST_EFFORT，issues 为 global cumulative hit rate is not monotonic 与 ... overshoots its final rate。这会带来两个后果：a) _prepare_validation 会因此恒定追加 TRAJECTORY_BEST_EFFORT，把已有（本来正常）场景的 analysis.json 的 validation.status 从 PASS 变成 PASS_WITH_WARNING，新告警失去区分度；b) 文档声称“大规模 cold 场景累计命中率平缓上升并精确落到有效目标”，与实测输出不一致。建议把“每条 lane 首条必然 miss”的可达下界计入基线（或只按 lane / DP rank 内部判定单调性，全局只判定超调），并同步修改文档措辞。

Comment thread plugins/prefix_cache/ais_bench_prefix_cache/generation.py
tuple[tuple[int, ...], int],
tuple[tuple[float, float, float, float], tuple[int, ...]],
] = {(initial_watermarks, 0): ((0.0, 0.0, 0.0, 0.0), ())}
future_capacity = _ColdFutureCapacity(caps, lane_requests)

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 这里每轮求解都会新建 _ColdFutureCapacity，但它的 lane_units 是用 @lru_cache(maxsize=65_536) 直接装饰的实例方法（定义在模块级类里）：该缓存挂在函数对象上、键里包含 self，因此会把每个实例连同其持有的 caps/lane_requests 列表长期留在进程内（最多 65536 条，批量/长驻调用时表现为内存滞留）。旧实现是把 @lru_cache 加在函数内部创建的闭包上，调用返回后即可回收。建议改为实例级普通 dict 缓存，或在 total_units 内部用局部闭包 + lru_cache，使缓存随求解结束释放。

Comment thread plugins/prefix_cache/README.md Outdated
- 理论值减目标值的带符号偏差和绝对偏差。

目标高于 `reachable_max` 或低于 `reachable_min` 时，求解器直接选择对应边界解；目标位于区间内时，按 Block 单位选择最近可达命中量。最终 hit token 精度是硬约束，在此基础上再优化请求过程：warmup 按累计输入比例均衡分配前缀；cold 按 `(Prefix Group, DP rank)` 独立水位搜索，依次最小化累计命中率的最大超调、总超调和回落，再尽快贴近目标。因此不会再把大部分前缀集中到前几条、最后用零/短前缀回调。
目标高于 `reachable_max` 或低于 `reachable_min` 时，求解器直接选择对应边界解；目标位于区间内时，按 Block 单位选择最近可达命中量。最终 hit token 精度是硬约束，在此基础上再优化请求过程：warmup 按累计输入比例均衡分配前缀；cold 的小型、不规则场景使用受限 beam search，大型且 lane 内长度单调的场景按 `(Prefix Group, DP rank)` 分解，先按容量和输入 token 分配目标，再在线性规模内构造逐步增长的水位。后者不再枚举每条请求的全部前缀候选，累计命中率平缓上升到目标，不会先明显超过目标再用短前缀回调。

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 这段新增说明只给出了定性描述（“按 (Prefix Group, DP rank) 分 lane 近线性构造、不再枚举每条请求的全部前缀候选”），没有任何性能规格：多大规模（请求数 × 输入长度）下 inspect/prepare 分别需要多少时间、多少内存、走哪条路径。既然本 PR 的目标就是“大规格数据集生成”，建议补充一张实测规格表（可直接用 PR 描述里的 1818 请求、平均约 86k 输入 token、block_size=128、6 个 group 的场景，给出 inspect 与 prepare 的耗时和机器规格），并说明 estimated_transitions > 2_000_000 时会从 beam 切换到 scalable 构造、超过该规模后耗时曲线是否仍近线性；docs/source_en 与 docs/source_zh_cn 中同一段也建议补充。另外该段的“累计命中率平缓上升并精确落到有效目标”与该场景实测的 BEST_EFFORT（见 generation.py 的轨迹判定）不一致，建议按实现修正措辞。

Comment thread docs/source_en/advanced_tutorials/prefix_cache.md
- [Prefix Cache plugin README](../../../plugins/prefix_cache/README_en.md)
- [Complete Scenario field reference](../../../plugins/prefix_cache/config_examples/scenario.example_en.md)
- [Multimodal benchmark MULTIMODAL.md](../../../plugins/prefix_cache/MULTIMODAL.md)
- [Prefix Cache plugin README](https://github.com/AISBench/benchmark/blob/master/plugins/prefix_cache/README_en.md)

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 文档内链由相对路径改为指向 https://github.com/AISBench/benchmark/blob/master/... 的绝对链接（zh_cn 文档与 README 同样改动）：这样任何分支/发布版本的文档都会跳到 master 的最新内容，离线阅读或内网构建时也无法解析。如果原相对链接只是构建期告警，建议保留相对链接，或改用不带硬编码分支的仓库地址（至少在正文中说明链接指向 master）。

Comment thread plugins/prefix_cache/tests/test_core.py Outdated
self.assertEqual(result.strategy, "scalable_lane_convergent")
self.assertEqual(result.trajectory_status, "PASS")
self.assertFalse(result.trajectory_issues)
self.assertLess(elapsed, 5.0)

@SJTUyh SJTUyh Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] 把墙钟断言写进单元测试容易在共享 CI / 容器 CPU 抖动时误报（self.assertLess(elapsed, 5.0)），同时 self.assertEqual(result.strategy, "scalable_lane_convergent") 与内部阈值/beam 宽度强耦合，后续微调 estimated_transitions 常数会让功能测试连带失败。建议把耗时断言放宽或拆成单独的 perf smoke（可加标记/重试），策略断言改为断言行为属性（命中 token 精确、累计命中率不超调、与 simulate_theory 一致），而不是断言内部策略字符串。

@github-actions github-actions Bot added the review_opened PR 中存在未闭环的检视意见，所有检视意见必须闭环后才能合入 label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过，PR 已被禁止合入。

未通过项：large_pr、review_opened

  • 超大 PR：新增代码 1453 行，超过 1000 行上限，建议拆分。
  • 检视意见未闭环：还存在 6 条未 Resolve 的检视意见，请全部 Resolve 后再合入。

请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。

This branch had an error being deployed

1 failed deployment
smoke-test-approval — 3676b176 Deployed Oct 16, 2026 by jschen069 via run_smoke_test_cases #804
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deepsource-io deepsource-io[bot] deepsource-io[bot] left review comments
@SJTUyh SJTUyh SJTUyh left review comments
@Libotry Libotry Awaiting requested review from Libotry
@Keithwwa Keithwwa Awaiting requested review from Keithwwa

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

large_pr PR 新增代码超过 1000 行，建议拆分为更小的 PR perf review_opened PR 中存在未闭环的检视意见，所有检视意见必须闭环后才能合入

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[issue] prefix cache性能优化，当生成大规格数据集时，缩短生成时间

2 participants

@jschen069 @SJTUyh