Fix test result propagation to radiator and Slack#47
Merged
maximusunc merged 2 commits intoJul 12, 2026
Conversation
Several test results were not making it to the reporting services intact: - Performance tests were created in the Information Radiator but never finished, so they showed up as perpetually incomplete. run_tests now finishes each performance test with a terminal status (FAILED if the query can't be generated or the run raises, PASSED otherwise). - Performance failures were overwritten per host in the ResultCollector, so when a suite exercised more than one target only the last target's failures reached Slack. Failures are now accumulated by error key, summing occurrences across hosts. - Agents that returned no response for an asset were written to the CSV as SKIPPED but omitted from the per-agent JSON stats, so the two Slack artifacts disagreed and undercounted. Skipped agents are now counted in the JSON stats too. - ARS timeouts (status code 598) were compared against the string "598" and so were never labeled "Timed out"; compare as a string of the code. Adds regression tests for each fix.
When an acceptance test case is skipped (its query never really ran), its assets were not being reported as SKIPPED. Instead each agent kept its incidental status - FAILED for ARAs that errored, NO_RESULTS for an ARS that returned an empty message - which inflated FAILED/NO_RESULTS counts and left 0 SKIPPED for the ARAs, disagreeing with the skipped test-level status shown in the Information Radiator. Two gaps caused this: - When the test-level status resolves to SKIPPED (no ARS result), the per-agent results were still recorded from the report. collect_acceptance_ result now takes force_skipped so every agent is recorded SKIPPED, and the radiator labels are uploaded as SKIPPED for all agents to match (previously labels were not uploaded at all for skipped tests, leaving the per-agent columns stale/blank). - When an asset had no query response at all (eg query generation failed), it was finished as SKIPPED in the radiator but never passed to the collector, so it vanished from the CSV and per-agent JSON stats. It is now recorded as SKIPPED for every agent. Adds regression tests covering the forced-skip stats/labels and an end-to-end skipped test case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several test results were not making it to the reporting services intact:
Performance tests were created in the Information Radiator but never
finished, so they showed up as perpetually incomplete. run_tests now
finishes each performance test with a terminal status (FAILED if the
query can't be generated or the run raises, PASSED otherwise).
Performance failures were overwritten per host in the ResultCollector,
so when a suite exercised more than one target only the last target's
failures reached Slack. Failures are now accumulated by error key,
summing occurrences across hosts.
Agents that returned no response for an asset were written to the CSV
as SKIPPED but omitted from the per-agent JSON stats, so the two Slack
artifacts disagreed and undercounted. Skipped agents are now counted in
the JSON stats too.
ARS timeouts (status code 598) were compared against the string "598"
and so were never labeled "Timed out"; compare as a string of the code.
Adds regression tests for each fix.