[BUG] Collector timeout fallback at 4 minutes causes duplicate dispatch when actual collection takes longer

[hengyuss]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Now the collector will be assigned a timeout if it runs over 4 minutes, causing the same job to be executed twice when the actual collection exceeds 4 minutes.

Expected Behavior

The logic in dispatchCollectData should be executed only once, whether triggered by normal collection completion or timeout fallback — whichever comes first.

Steps To Reproduce

No response

Environment

HertzBeat version(s):

Debug logs

No response

Anything else?

i will fix it

[habiibullahm]

Hi @hengyuss @tomsun28 — I'd like to help with this if it's still available.

I looked at CommonDispatcher:

• In-flight collects are tracked in metricsTimeoutMonitorMap with DURATION_TIME = 240_000L (4 minutes)
• monitorCollectTaskTimeout removes the map entry and calls dispatchCollectData with a TIMEOUT payload
• When the real MetricsCollect finishes later, it calls dispatchCollectData again; remove(monitorKey) returns null so metrics recording is skipped, but sendMetricsData / cyclic re-schedule (and next-level scheduling) can still run — that matches the duplicate-dispatch behavior

Proposed approach:

1. Make dispatchCollectData once-wins (e.g. early-return on completion when the monitor entry was already removed by timeout, with care for subtask/prometheus keys), and/or cancel/mark the in-flight collect on timeout so MetricsCollect.fastFailed() stops the late path
2. Add a regression test: timeout fires, then late success → assert dispatchCollectData side effects happen only once

Quick questions before I open a PR:

• @hengyuss you noted “i will fix it” — are you still working on this? Does open PR fix: dispatch timeout metrics for all priorities with staggered delay #4196 (for [BUG] Availability (priority=0) failure blocks all lower priority metrics collection, causing monitoring pipeline stall #4194) also intend to cover [BUG] Collector timeout fallback at 4 minutes causes duplicate dispatch when actual collection takes longer #4203, or is that separate?
• If this is free, is the once-wins / early-return approach acceptable?

Happy to take it if you’re not actively working on it. Thanks!

[hengyuss]

Hi @hengyuss @tomsun28 — I'd like to help with this if it's still available.嗨——如果还有机会的话，我愿意帮忙处理这件事。

I looked at CommonDispatcher:我看了看 CommonDispatcher ：

* In-flight collects are tracked in `metricsTimeoutMonitorMap` with `DURATION_TIME = 240_000L` (4 minutes)飞行过程中的收集数据通过 `metricsTimeoutMonitorMap` 进行记录，更新频率为 `DURATION_TIME = 240_000L` （4 分钟一次）。

* `monitorCollectTaskTimeout` removes the map entry and calls `dispatchCollectData` with a `TIMEOUT` payload `monitorCollectTaskTimeout` 删除该地图条目，然后使用 `TIMEOUT` 作为参数来调用 `dispatchCollectData` 。

* When the real `MetricsCollect` finishes later, it calls `dispatchCollectData` again; `remove(monitorKey)` returns null so metrics recording is skipped, but `sendMetricsData` / cyclic re-schedule (and next-level scheduling) can still run — that matches the duplicate-dispatch behavior当真正的 `MetricsCollect` 稍后完成时，它会再次调用 `dispatchCollectData` ；由于 `remove(monitorKey)` 返回了 null 值，因此会跳过指标记录步骤。不过， `sendMetricsData` 的循环重新调度功能以及更高级别的调度操作仍然可以正常运行——这符合“重复调度”的行为模式。

Proposed approach: 拟采用的方法：

1. Make `dispatchCollectData` once-wins (e.g. early-return on completion when the monitor entry was already removed by timeout, with care for subtask/prometheus keys), and/or cancel/mark the in-flight collect on timeout so `MetricsCollect.fastFailed()` stops the late path让 `dispatchCollectData` 实现“一次成功即可”机制（例如：当监控条目因超时而被移除时，能够立即返回结果；同时要妥善处理与子任务/ Prometheus 相关的键值对）。另外，当出现超时情况时，应取消或标记正在进行的收集操作，这样 `MetricsCollect.fastFailed()` 就能停止后续的处理流程。

2. Add a regression test: timeout fires, then late success → assert `dispatchCollectData` side effects happen only once添加一个回归测试：当超时发生时，系统应触发相应的处理机制；之后，如果操作最终成功，那么应确认 `dispatchCollectData` 的副作用仅发生一次。

Quick questions before I open a PR:在提交 PR 之前，有一些问题需要先解答：

* [@hengyuss](https://github.com/hengyuss) you noted “i will fix it” — are you still working on this? Does open PR [fix: dispatch timeout metrics for all priorities with staggered delay #4196](https://github.com/apache/hertzbeat/pull/4196) (for [[BUG] Availability (priority=0) failure blocks all lower priority metrics collection, causing monitoring pipeline stall #4194](https://github.com/apache/hertzbeat/issues/4194)) also intend to cover [[BUG] Collector timeout fallback at 4 minutes causes duplicate dispatch when actual collection takes longer #4203](https://github.com/apache/hertzbeat/issues/4203), or is that separate?你提到“我会去解决这个问题”——你还在继续处理这件事吗？ [fix: dispatch timeout metrics for all priorities with staggered delay #4196](https://github.com/apache/hertzbeat/pull/4196) 发起的开放 PR（针对 [[BUG] Availability (priority=0) failure blocks all lower priority metrics collection, causing monitoring pipeline stall #4194](https://github.com/apache/hertzbeat/issues/4194) ）是否也打算涵盖 [[BUG] Collector timeout fallback at 4 minutes causes duplicate dispatch when actual collection takes longer #4203](https://github.com/apache/hertzbeat/issues/4203) 的问题？还是说这两者是分开处理的？

* If this is free, is the once-wins / early-return approach acceptable?如果这是免费的，那么“一次获胜/尽早获得回报”的策略是否可行呢？

Happy to take it if you’re not actively working on it. Thanks!如果你不打算主动处理这件事的话，那我就收下吧。谢谢！

PMC want to discuss a solution for this issue， If you have a better solution, you can give it a try.