Improve first thought by RedTanny · Pull Request #291 · RHEcosystemAppEng/exploit-iq-agent

RedTanny · 2026-07-13T07:40:58Z

Summary

Refactors L1 intel gathering so the checker extracts CVE understanding and patch intel separately (per-file), then feeds focused context into Source Grep observations. Also includes related checker bugfixes and OSIDB identify improvements from this branch.

Why

The old single-prompt intel extraction mixed CVE narrative with full-patch analysis, which hurt keyword quality on large patches and gave observations little file-specific guidance. Splitting the flow improves search targeting and keeps large patches within the context budget.

High-level feature (intel refactor)

Prompt 1: CVE understanding (no patch required) — vulnerability type, component, ordered grep keywords
Prompt 2/3: Per-file patch analysis — functions/patterns per changed file, with token-budget batching for large patches
Result model: IntelGatheringResult aggregates per-file intel and converts back to VulnerabilityIntel for existing prompts/reports
Focused observations: when Source Grep targets a file, prepend that file’s intel into the observation context
New flow is now the sole intel path (dual-path flag removed)

Testing

Added unit coverage for the refactor and related empty/error tool handling:

test_file_change_summary.py — patch → per-file summaries / token estimates
test_cve_understanding_prompt.py — Prompt 1 formatting and keyword rules
test_patch_analysis_prompt.py — Prompt 2/3 formatting and functions summary helper
test_intel_gathering_result.py — lookup, aggregation, focused-context formatting
test_new_intel_flow.py — file priority + token-budget batch split
test_focused_observations.py — file filter extraction + focused context injection
test_check_empty_output.py — tool-error detection via status/prefix (not mid-content "error:")
tests/test_package_identifier.py — OSIDB affectedness / identify regression coverage

Bug fixes

Missing ADVISORY URL patterns / OSIDB in URL mining (b31b9d9): advisory/OSIDB URLs were missed when mining commit/advisory links for intel.
Weak hints continued into repo search (8ab9421): weak reference hints no longer keep searching repos; flow goes to the agent instead.
Noisy prompt regression (cafce28): removed extra prompt/intel fields that added noise and made the LLM less reliable.
“Patch file:” in report (47f8c01): dropped confusing “Patch file:” labeling from the checker report viewer.
Source Grep tool-call guidance (6c0576d): clarified grep tool instructions/schema so the LLM calls Source Grep correctly.
Identify regression (4132d2b): fixed PackageIdentifier affectedness regression after OSIDB support.

Also in this branch

OSIDB affectedness support in PackageIdentifier (55ac04d)
Stricter tool-error handling in observation/build agents so successful greps containing "error:" in source are not treated as tool failures

Test plan

Unit tests listed above
Local CVE run: Phoenix Initial_Intelligence_Gathering shows file_changes + CVE understanding
File-filtered Source Grep: observation span has focused_context_used=true
Spot-check known CVE verdicts for unexpected flips

vbelouso · 2026-07-13T07:41:19Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

zvigrinberg

@RedTanny Thank you for the PR.
Please see my comments.

TamarW0 · 2026-07-20T10:36:26Z

/retest

exploit-iq-pac · 2026-07-20T10:36:30Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
vulnerability-analysis-on-pr	`CEL expression evaluation error: expression "event == \"pull_request\" &&\n!body.pull_request.draft &&\n(target_branch == \"main\" \|\| target_branch == \"rh-aiq-main\") &&\n(\"src/\".pathChanged() \|\| \"metrics_lib/\".pathChanged() \|\| \"pyproject.toml\".pathChanged() \|\| \"uv.lock\".pathChanged() \|\| \"Dockerfile\".pathChanged() \|\| \".dockerignore\".pathChanged())\n" failed to evaluate: no such key: pull_request`

TamarW0 · 2026-07-20T10:40:31Z

/test vulnerability-analysis-on-pr

etsien · 2026-07-21T17:31:49Z

Just FYI, with the multi-prompting framework coming, these prompts will be moved to centralized locations, with separate copies per model and if necessary, per environment (I know of Java-specific prompts for certain tool usage chains).

So please be aware of this coming change, and a big rebase/merge.

zvigrinberg · 2026-07-21T18:10:59Z

Just FYI, with the multi-prompting framework coming, these prompts will be moved to centralized locations, with separate copies per model and if necessary, per environment (I know of Java-specific prompts for certain tool usage chains).

So please be aware of this coming change, and a big rebase/merge.

@etsien But the RPM analysis is out of scope for multi repo... ( this could be a future enhancement POST-GA that if the TPA guys wants to prioritize it will happen). So it's expected to be quite smooth rebase.

… to agent

… and llm didn't follow prompt less accurate

Implements issues 01-06 of intel gathering refactor: - FileChangeSummary model and extraction (01) - file_changes span output (02) - CVE_UNDERSTANDING_PROMPT (03) - PATCH_ANALYSIS_PROMPT (04) - IntelGatheringResult object (05) - Dual-path intel gathering with enable_new_intel_flow config (06) Co-authored-by: Cursor <cursoragent@cursor.com>

…bservations

Co-authored-by: Cursor <cursoragent@cursor.com>

…l LLM calls. Steer forced-finish and empty-grep from FILE_CHANGES/identify signals, keep focused patch context out of NEW OUTPUT, and avoid cartesian comprehension that overflowed context on large single-file diffs. Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg · 2026-07-28T10:53:17Z

+    "Intercepted an error from tool",
+)
+# Source Grep query: pattern[,file_filter] — last comma segment is the file filter.
+GREP_QUERY_FILE_SEPARATOR = ","


@RedTanny Maybe this one and GREP_QUERY_FILE_SEPARATOR of cve_package_code_agent.py could be the same one ( impore it there from here)?

@zvigrinberg
fixed
@RedTanny
CodeReview GREP_QUERY_FILE_SEPARATOR dup

zvigrinberg · 2026-07-28T11:03:01Z

+            # Fallback: compute target_version_in_vulnerable_range from NVD if not set by Identify phase
+            if vulnerability_intel.target_version_in_vulnerable_range is None and fixed_ver:
+                try:
+                    from packaging import version as pkg_version


@RedTanny Are you sure it's the appropriate lib ( packaging.version) to compare the versions? AFAIK it's only tailored to python versions as per PEP-440.

@zvigrinberg
fixed
@RedTanny
CodeReview version package

zvigrinberg · 2026-07-28T11:07:35Z

+    normalized_line = _normalize_pattern_text(line)
+    if not normalized_line:
+        return False
+    return pattern in normalized_line or normalized_line in pattern


@RedTanny IMO It's too much permissive, it allows, for instance , to generic pattern like write to pass through if the line contains a much more specific method/function, like write_in_spool_if_ack , but it's clear that there is no affinity at all between these two.

@zvigrinberg
fixed
@RedTanny
CodeReview _pattern_matches_line

- Added test_l1_file_changes_step1_prompt.py to verify Step-1 prefers highest-removal FILE_CHANGES file - Added test_l2_kernel_makefile_pattern_prompt.py to verify L2 kernel Makefile grep uses path tokens Co-authored-by: Cursor <cursoragent@cursor.com>

- Added test_l1_intel_grep_guard.py to verify intel allowlist validation - Tests cover hallucinated grep patterns, allowed substrings, and file filter scopes Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg

@RedTanny LGTM Approved , Great job.
Thank you.

RedTanny requested review from TamarW0 and zvigrinberg July 13, 2026 07:40

zvigrinberg added enhancement New feature or request optimization labels Jul 13, 2026

zvigrinberg requested changes Jul 13, 2026

View reviewed changes

RedTanny and others added 12 commits July 26, 2026 07:28

BugFix missing ADVISORY patterns and support for osidb in url minning

f9b2cb9

BugFix when weak hints are found don't continue to repo search but go…

5ff556b

… to agent

Bugfix: reggression bug caused when adding to prompt cause more nosie…

2d7d09f

… and llm didn't follow prompt less accurate

remove Patch file: from report confuses the viewer

b2810a2

make LLM know how to call the grep tool correctly

74e6db7

feat: add OSIDB affectedness support to PackageIdentifier

e5fe33a

fix regression issue identify

a079b22

feat: switch intel gathering to per-file 3-prompt flow with focused o…

d87ba78

…bservations

fix: align OSIDB NOTAFFECTED test with continue-to-verify status

f2a6b86

Co-authored-by: Cursor <cursoragent@cursor.com>

Issue1 and issue 2 remove unused code

b1c8ca6

CodeReview 3: do exact compare

97f5d75

RedTanny force-pushed the improve-first-thought branch from 2601f5a to 97f5d75 Compare July 26, 2026 08:21

fix rebase broke test fix it

a756ee6

zvigrinberg reviewed Jul 27, 2026

View reviewed changes

Comment thread src/vuln_analysis/functions/code_agent_graph_defs.py

zvigrinberg reviewed Jul 27, 2026

View reviewed changes

Comment thread src/vuln_analysis/utils/rpm_checker_prompts.py Outdated

RedTanny and others added 3 commits July 27, 2026 17:39

CodeReview duplicate grep in prompt

cb48671

fix last issues

8d827a9

zvigrinberg reviewed Jul 28, 2026

View reviewed changes

Comment thread src/vuln_analysis/functions/react_internals.py

RedTanny and others added 5 commits July 28, 2026 15:41

test: add unit tests for L1IntelGrepGuard

f7e3214

- Added test_l1_intel_grep_guard.py to verify intel allowlist validation - Tests cover hallucinated grep patterns, allowed substrings, and file filter scopes Co-authored-by: Cursor <cursoragent@cursor.com>

CodeReview GREP_QUERY_FILE_SEPARATOR dup

322654e

CodeReview _pattern_matches_line

2120258

CodeReview version package

02accb7

RedTanny requested a review from zvigrinberg July 28, 2026 13:55

zvigrinberg approved these changes Jul 28, 2026

View reviewed changes

RedTanny merged commit 6184325 into RHEcosystemAppEng:main Jul 28, 2026
2 checks passed

Uh oh!

Conversation

RedTanny commented Jul 13, 2026

Summary

Why

High-level feature (intel refactor)

Testing

Bug fixes

Also in this branch

Test plan

Uh oh!

vbelouso commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

zvigrinberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

exploit-iq-pac Bot commented Jul 20, 2026

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

etsien commented Jul 21, 2026

Uh oh!

zvigrinberg commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

zvigrinberg Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

RedTanny Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

RedTanny Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

RedTanny Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zvigrinberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vbelouso commented Jul 13, 2026 •

edited

Loading