Repository navigation

solidify the metrics to mlflow - #1

Closed

ashtarkb wants to merge 393 commits into

mainfrom

mlflow-metrics

Closed

solidify the metrics to mlflow#1
ashtarkb wants to merge 393 commits into
mainfrom
mlflow-metrics

This pull request is big! We're only showing the most recent 250 commits

Commits on Jul 17, 2026
Commits on Jul 18, 2026
Commits on Jul 20, 2026
- Merge branch 'main' into feat/benchmark-matrix-isolation
  albertoperdomo2
  authored
  7f1382c
- Merge pull request #132 from kpouget/llmd-dry-run
  kpouget
  authored
  7c237f6
- Merge branch 'main' into feat/benchmark-matrix-isolation
  albertoperdomo2
  authored
  3bc2997
- Merge pull request #129 from albertoperdomo2/feat/benchmark-matrix-isolation
  openshift-merge-bot[bot]
  authored
  8df998c
- feat: add nightly pipeline to forge using clusterless pipeline
  
  ashtarkb
  and
  cursoragent
  committed
  c02d8e3
- fix: use openshift origin-cli image for oc in cronjob
  
  ashtarkb
  and
  cursoragent
  committed
  625cf60
- refactor: collapse nightly pipeline into single task with notifications
  
  ashtarkb
  and
  cursoragent
  committed
  e16dc56
- fix: use correct cluster name for nightly job target
  
  ashtarkb
  and
  cursoragent
  committed
  318657b
- fix: correct pipeline name to forge-full
  
  ashtarkb
  and
  cursoragent
  committed
  9370ef2
- fix: rename _mesh_ready_spec to propagate across DSL tasks
  
  ashtarkb
  and
  cursoragent
  committed
  fdc49f1
- Add Slack failure alert when RHAIIS pipeline crashes
  
  Harshith-umesh
  and
  cursoragent
  committed
  b983a1d
- Append FJOB suffix to deployment name for parallel job isolation
  
  Harshith-umesh
  and
  cursoragent
  committed
  3948bd1
- Switch RHAIIS Slack notifications to topsail bot on psap-rhaiis-alerts channel
  
  Harshith-umesh
  and
  cursoragent
  committed
  d4c7600
Commits on Jul 21, 2026
- Truncate deployment name to respect KServe 63-char hostname limit
  
  Harshith-umesh
  and
  cursoragent
  committed
  7850ad4
- Fix deployment name truncation to preserve unique FJOB suffix
  
  Harshith-umesh
  and
  cursoragent
  committed
  e30e0eb
- Add AGENTS.md with security guidelines for secret handling
  ashtarkb
  committed
  0119b54
- Merge branch 'main' into agents-md
  ashtarkb
  authored
  0eb3bc3
- Apply suggestion from @kpouget
  kpouget
  authored
  dca57ba
- Apply suggestion from @kpouget
  kpouget
  authored
  b328d55
- Merge pull request #134 from ashtarkb/agents-md
  kpouget
  authored
  1fa202a
- Remove slack-webhook-url, use topsail bot exclusively for notifications
  
  Harshith-umesh
  and
  cursoragent
  committed
  cee42b3
- Refactor: extract analysis, notifications; configurable ISVC labels and S3 settings
  
  Harshith-umesh
  and
  cursoragent
  committed
  57f0848
Commits on Jul 22, 2026
- Fix config collision: move rhaiis.s3 to config.d/rhaiis.yaml
  
  Harshith-umesh
  and
  cursoragent
  committed
  069859a
- refactor: config-driven nightly pipeline with reusable MLflow verifier
  ashtarkb
  committed
  c3a89ab
- style: apply ruff formatting
  ashtarkb
  committed
  cf1ff57
- fix: Increase number of attempts
  albertoperdomo2
  committed
  abfe64a
- Merge pull request #137 from albertoperdomo2/fix/fournos-launcher-delay
  openshift-merge-bot[bot]
  authored
  5738c61
- small fix to run in the smoke cluster
  ashtarkb
  committed
  a7cea8a
- [core] dsl: show the function duration in the completion banner
  kpouget
  committed
  6196820
- [core] library: postprocess: skip in the postprocessing on keyboard interrupt
  kpouget
  committed
  71c91ae
- [core] library: ci: better handle early exception
  kpouget
  committed
  aaed126
- [core] library: config: more more reliable if 'extends: None'
  kpouget
  committed
  69062f6
- [core] dsl: utils: k8s: allow oc to save to file
  kpouget
  committed
  7affb82
- [core] dsl: utils: k8s: use stdout_dest
  kpouget
  committed
  c9d19f4
- [agentic_tools] mcp: toolbox: deploy_mock_servers/main: use stdout_dest
  kpouget
  committed
  5caaa6c
- [guidellm] toolbox: run_guidellm_benchmark: use stdout_dest
  kpouget
  committed
  5a63729
- [agentic_tools] locust: toolbox: generate_prompts/main: use stdout_dest
  kpouget
  committed
  9cbfac0
- [kserve] toolbox: prepare_hf_model_cache: use oc stdout_dest
  kpouget
  committed
  9790aa0
- [llm_d] toolbox: capture_prepare_state: use stdout_dest
  kpouget
  committed
  101b4b8
- [mcp_gateway] toolbox: use stdout_dest
  kpouget
  committed
  d032774
- [mcp_gateway] toolbox: install_platform: use stdout_dest
  kpouget
  committed
  83cdf21
- [core] Add the config changelog
  kpouget
  committed
  885f618
- Merge pull request #130 from kpouget/stdout_dest
  kpouget
  authored
  c78e432
- [llm_d] start working on P/D testing
  kpouget
  committed
  80b3ac2
- [llm_d] orchestration: render_inference_service: add support for P/D
  kpouget
  committed
  545b173
- [llm_d] orchestration: work on the P/D deployment
  kpouget
  committed
  b6cf2c8
- [llm_d] tests: add reference profiles
  kpouget
  committed
  fc629cd
- [llm_d] orchestration: keep working on P/D
  kpouget
  committed
  4badfcc
- [llm_d] orchestration: manifests: scheduler_config: unify
  kpouget
  committed
  7527409
- [core] library: env: create link from /tmp/forge_last to the current ARTIFACT_DIR
  kpouget
  committed
  c76f038
- [core] library: env: use '%Y%m%d-%H%M-%S' if '%Y%m%d-%H%M' already exists
  kpouget
  committed
  a4cbb64
- [llm_d] work on P/D
  kpouget
  committed
  50dd143
- [llm_d] CHANGELOG-orchestration: add a changelog
  kpouget
  committed
  0c0f6a2
- [core] library: postprocess: return the status dict if postprocessing is disabled
  kpouget
  committed
  a6099df
- [llm_d] orchestration: use the config file to define the labels
  kpouget
  committed
  103e33c
- [llm_d] finish removing the managed namespace
  kpouget
  committed
  bdf4935
- [kserve] toolbox: prepare_hf_model_cache: remove the useless namespace_is_managed flag
  kpouget
  committed
  20f1854
- [llm_d] relocatethe runtime.kserve.dry_run flag
  kpouget
  committed
  e133a3a
- Fix result extraction truncation and PVC cleanup for parallel jobs
  
  Harshith-umesh
  and
  cursoragent
  committed
  d52fc25
- Refactor dashboard CSV to use Caliper postprocessing pipeline
  
  Harshith-umesh
  and
  cursoragent
  committed
  fdf73c3
- Populate all dashboard CSV fields from Caliper KPIs
  
  Harshith-umesh
  and
  cursoragent
  committed
  4d6e58d
- Handle large benchmark files: compress before oc cp
  
  Harshith-umesh
  and
  cursoragent
  committed
  d15938f
- Fix KPI validation: add required run_id field
  
  Harshith-umesh
  and
  cursoragent
  committed
  2b55199
- Fix dashboard CSV: use GPU chip name for accelerator, include run_uuid
  
  Harshith-umesh
  and
  cursoragent
  committed
  af3b991
- fix: sort extra curves by request_rate and convert latency units to ms
  
  Harshith-umesh
  and
  cursoragent
  committed
  c77bd80
Commits on Jul 23, 2026
Commits on Jul 24, 2026
- [projects] Fix ruff in MD files
  kpouget
  committed
  3c629b8
- ruff: pin the version to 0.16.0
  kpouget
  committed
  635e05e
- Merge pull request #141 from kpouget/ruff
  kpouget
  authored
  0506991
Commits on Jul 26, 2026
- cluster: add capture_prometheus toolbox for extracting time-bounded metrics
  
  ashtarkb
  and
  cursoragent
  committed
  ece24b9
- addres few code rabbit issues
  ashtarkb
  committed
  6502de3
Commits on Jul 27, 2026
Commits on Jul 28, 2026
- [core] dsl: task: log the end time
  kpouget
  committed
  d57c523
- [core] dsl: runtime: reduce the log level
  kpouget
  committed
  407ca50
- [core] library: export: improve the test status
  kpouget
  committed
  b3f7964
- [core] library: config: write_variables_override: accept comma separated presets
  kpouget
  committed
  7635cda
- [core] library: env: add a MuteStdOut context
  kpouget
  committed
  0c9ce60
- Merge pull request #140 from kpouget/core
  kpouget
  authored
  ba1b681
- fix: address PR review comments (coderabbit + colleague)
  
  Harshith-umesh
  and
  cursoragent
  committed
  8aa9b77
- Merge branch 'main' into port/model-furnace-features
  Harshith-umesh
  authored
  3f2ecca
Commits on Jul 29, 2026
- Merge remote-tracking branch 'upstream/main' into nightly-pipeline
  
  ashtarkb
  and
  cursoragent
  committed
  0dced36
- Address PR review: stricter error handling in mlflow verifier, fix duplicate directive
  
  ashtarkb
  and
  cursoragent
  committed
  44902bf
- fix: sort imports in mlflow_verifier to satisfy ruff I001
  
  ashtarkb
  and
  cursoragent
  committed
  c4bc7af
- style: apply ruff formatting to mlflow_verifier
  
  ashtarkb
  and
  cursoragent
  committed
  024bd90
- Merge pull request #128 from Harshith-umesh/port/model-furnace-features
  openshift-merge-bot[bot]
  authored
  ce6a8f8
- [caliper] orchestration: postprocess: re-enable the hierarchical KPI format
  kpouget
  committed
  08bf719
- Merge pull request #146 from kpouget/kpi
  kpouget
  authored
  68c3880
- feat: add SGLang and TRT-LLM engine support
  
  Harshith-umesh
  and
  cursoragent
  committed
  b61e356
- fix: remove runAsUser:0 from vLLM/SGLang ServingRuntime
  
  Harshith-umesh
  and
  cursoragent
  committed
  cf28237
- fix: add USER env var back for vLLM/SGLang ServingRuntime
  
  Harshith-umesh
  and
  cursoragent
  committed
  ea8f555
- refactor: replace image_pull_secret string with image_pull_secrets list
  
  Harshith-umesh
  and
  cursoragent
  committed
  fa0dab0
- refactor: move manifest building to orchestration layer
  
  Harshith-umesh
  and
  cursoragent
  committed
  6c1c397
- fix: write hierarchical KPIs to separate file to preserve JSONL
  
  Harshith-umesh
  and
  cursoragent
  committed
  70a55ea
Commits on Jul 30, 2026
- add default trtllm_config structure to base config
  
  Harshith-umesh
  and
  cursoragent
  committed
  0f061ab
- Add artifact export and MLflow config to nightly pipeline
  
  ashtarkb
  and
  cursoragent
  committed
  b69e3eb
- Decouple verifier experiment from export experiment
  
  ashtarkb
  and
  cursoragent
  committed
  1f64f22
- Handle verifier bootstrap: treat failures as no previous version
  
  ashtarkb
  and
  cursoragent
  committed
  f088034
- [kserve] toolbox: deploy_llmisvc: correctly resolve the kserve address
  kpouget
  committed
  42ad31a
- [kserve] toolbox: deploy_llmisvc: improve wait_old_pods_gone
  kpouget
  committed
  74f94fd
- [kserve] toolbox: prepare_hf_model_cache: wait for the pod to start running in a dedicated task
  kpouget
  committed
  ce0d075
- [kserve] toolbox: capture_llmisvc_state: capture more information
  kpouget
  committed
  ca2bcec
- [kserve] toolbox: deploy_llmisvc: capture more information
  kpouget
  committed
  049bdcc
- [kserve] toolbox: prepare_hf_model_cache: wait 1h for the download completion
  kpouget
  committed
  35efb51
- [kserve] toolbox: deploy_llmisvc: add flag to wait_pods_scheduled
  kpouget
  committed
  d8f7538
- [kserve] toolbox: deploy_llmisvc: capture_final_llmisvc_yaml: better error handling
  kpouget
  committed
  b957741
- [kserve] toolbox: deploy_llmisvc: abort on Pod restart
  kpouget
  committed
  e9c25dd
- [kserve] toolbox: deploy_llmisvc: improve the URL resolution
  kpouget
  committed
  3d80550
- [kserve] toolbox: deploy_llmisvc: also wait on SchedulingGated
  kpouget
  committed
  02247d6
- [kserve] toolbox: deploy_llmisvc: expose the try_resolve_endpoint_url method
  kpouget
  committed
  ef3abf5
- [kserve] toolbox: deploy_llmisvc: save the deploy/rs/pod description
  kpouget
  committed
  8ef80c1
- [guidellm] toolbox: run_smoke_request: save the pod info in a file
  kpouget
  committed
  96b00a0
- [guidellm] toolbox: run_smoke_request: don't wait for the pod deletion
  kpouget
  committed
  68b26a8
- [guidellm] toolbox: run_guidellm_benchmark: give more time
  kpouget
  committed
  bcde3d0
- [guidellm] toolbox: run_guidellm_benchmark: set the owner of the PVC
  kpouget
  committed
  b4e8c2c
- [caliper] engine: parameter_matrix: don't truncate long parameters
  kpouget
  committed
  c5547eb
- [llm_d] tests: test_deployment_profiles: add a 60s timeout
  kpouget
  committed
  b7f46d8
- Merge pull request #138 from kpouget/helpers
  openshift-merge-bot[bot]
  authored
  e7ff52b
- [caliper] postprocess: helpers: visualization_utils: update the size/scale of the plots
  kpouget
  committed
  b4577af
- [caliper] cli: commands: don't swallow the stack trace
  kpouget
  committed
  5ebeae8
- [caliper] engine: parse: show the parsing duration
  kpouget
  committed
  b80af82
- [caliper] orchestration: postprocess: show the stack trace
  kpouget
  committed
  ae6b755
- [caliper] improve the --include-filter behavior
  kpouget
  committed
  ec6b56d
- Revert "[caliper] remove unnecessary reference to the hierachical format"
  kpouget
  committed
  9283f38
- [caliper] engine: traverse: include hierarchical label parsing
  kpouget
  committed
  9901b0b
- fix: add init container to fix PVC permissions for GuideLLM
  
  Harshith-umesh
  and
  cursoragent
  committed
  802a178
- fix: make fsGroup opt-in for GuideLLM job pods
  
  Harshith-umesh
  and
  cursoragent
  committed
  27be386
- chore: add missing trtllm_config defaults for Nemotron-H
  
  Harshith-umesh
  and
  cursoragent
  committed
  a5de6ca
- fix: include trtllm_config in dashboard runtime_args
  
  Harshith-umesh
  and
  cursoragent
  committed
  f221ef1
- chore: use recommended sglang serve entrypoint
  
  Harshith-umesh
  and
  cursoragent
  committed
  c4d582e
Commits on Jul 31, 2026
- docs: update README with multi-engine support
  
  Harshith-umesh
  and
  cursoragent
  committed
  f88e6a7
- Merge pull request #150 from kpouget/caliper
  kpouget
  authored
  5812b6c
- [cluster] toolbox: create_sa_kubeconfig: new toolbox script
  kpouget
  committed
  618599e
- [core] dsl: utils: k8s: add the handled secretly arg to flag the calls handling secret data
  kpouget
  committed
  39fe524
- AGENTS.md: adjust the secret-handling guidelines
  kpouget
  committed
  c5ce243
- [core] dsl: shell: add support for input and timeout
  kpouget
  committed
  cec558c
- Merge pull request #139 from kpouget/cluster
  kpouget
  authored
  342ca3e
- fix: address PR review comments
  
  Harshith-umesh
  and
  cursoragent
  committed
  89f906a
- Merge branch 'main' into sglang-trtllm
  Harshith-umesh
  authored
  f6de064
- fix: use format.py from PR #136 for csv-export v2 support
  
  Harshith-umesh
  and
  cursoragent
  committed
  f5274f7
- fix: preserve per-KPI labels (rate_index) in hierarchical JSON
  
  Harshith-umesh
  and
  cursoragent
  committed
  e10f721
- fix: restore run_path when flattening hierarchical KPIs
  
  Harshith-umesh
  and
  cursoragent
  committed
  b95b5c2
- Merge pull request #149 from Harshith-umesh/sglang-trtllm
  kpouget
  authored
  e8921fb
- [kserve] toolbox: capture_llmisvc_state: capture more information
  kpouget
  committed
  445a560
- [kserve] toolbox: deploy_llmisvc: raise an exception if name isn't k8s compliant
  kpouget
  committed
  3938c15
- [kserve] toolbox: deploy_llmisvc: wait_pods_scheduled: ensure the LLMISVC exists
  kpouget
  committed
  88ff61b
- [guidellm] toolbox: run_guidellm_benchmark: give more time
  kpouget
  committed
  0711987
- [guidellm] toolbox: run_guidellm_benchmark: compress and trim the benchmarks.json file
  kpouget
  committed
  2de3103
- [guidellm] postprocess: guidellm: improve the plotting
  kpouget
  committed
  244d836
- [rhoai] library: deploy: improve the error message
  kpouget
  committed
  cd1f5e7
- [rhoai] toolbox: wait_datasciencecluster_ready: better detect deployment errors
  kpouget
  committed
  5edff8c
- [cluster] toolbox: deploy_custom_catalog: wait for the Pod to start running
  kpouget
  committed
  875babc
- [llm_d] orchestration: render_inference_service: make the deployment k8s compliant
  kpouget
  committed
  f5457e7
- [llm_d] tests: reference_deployments: normalize the guidellm names
  kpouget
  committed
  37a5125
- [core] dsl: utils: k8s: add the text flag for the oc command
  kpouget
  committed
  d75dbfe
- [core] dsl: shell: add text field for the run command
  kpouget
  committed
  f9a05ed
- [llm_d] orchestration: config: disable kpis_to_csv and analyze
  kpouget
  committed
  fdb42e0
- Merge pull request #151 from kpouget/inference
  kpouget
  authored
  cd8c7c8
Commits on Aug 3, 2026

Commits

Commits on Jul 17, 2026

Commits on Jul 18, 2026

Commits on Jul 20, 2026

Commits on Jul 21, 2026

Commits on Jul 22, 2026

Commits on Jul 23, 2026

Commits on Jul 24, 2026

Commits on Jul 26, 2026

Commits on Jul 27, 2026

Commits on Jul 28, 2026

Commits on Jul 29, 2026

Commits on Jul 30, 2026