Add worker telemetry for key metrics (KV Cache Utilization, queue depth) at a configurable interval

[jamesyan2028]

Currently Opal does not have a global view of per worker metrics for KVC utilization and queue depth. These metrics are important since they are consumed by other services (such as autoscaling) to make decisions. This pull starts and extends and implements _periodic_infra_updates() in vllm_worker.py so each worker periodically reports its KVC utilization and queue depth to the router. In addition, this pull request also adds computing and storing pluggable p90/p95/p99 TTFT and ITL over a sliding window of most recent requests in the router at the same interval that KVC utilization and queue depth metrics are pushed, to be used for future autoscaling decisions,

To simulate telemetry reporting in a distributed inference environment, each worker pushes their per-worker statistics to the router every self.periodic_infra_update_time virtual seconds as a SystemEvent dataclass. This configurable parameter is to simulate telemetry program (e.g. Prometheus) scrape/push intervals. The router gathers these per-worker SystemEvents and aggregates them into a MetricsSnapshot dataclass.

Previously in the router, a router process would be responsible for polling and draining the event_queue() of incoming events from workers. If the queue was empty, the router process would sleep for periodic_infra_update_collection_time number of seconds before waking up again to check the queue. This pull requests removes periodic_infra_update_collection_time, and instead choses to have the workers push their own telemetry every self.periodic_infra_update_time virtual seconds.

The router then stores the most recent MetricsSnapshot in self._latest_metrics, which can be queried by other services. A log of all past MetricsSnapshots taken during the simulation is also added to stage_statistics.py.