feat(examples): CV screening demo with feedback-to-deploy walkthrough

[mmabrouk]

What this is

A complete CV screening example that demonstrates the core Agenta loop: a prompt screens a candidate, the result lands in observability as a trace, a human gives feedback on it, and that feedback drives a prompt change that you evaluate and deploy. The prompt scores a CV against an IT Manager job spec on three booleans with reasons (tech_match, experience_match, overall_match) plus the list of missing requirements.

A recruiter uses a small Streamlit app (PDF upload to Markdown to the prompt fetched from the registry). An AI engineer works the prompt, the test set, and the evaluations in Agenta. The two sides stay split: screening.py owns the AI logic and tracing, app.py is only UI.

The walkthrough

The example is built around one story: the company's working language is German, but the prompt's job spec never says so.

1. The recruiter screens a strong IT manager who does not speak German. The app says "Advance to interview" (the miss), so they submit a thumbs-down with a comment.
2. The AI engineer filters traces by the user-feedback annotation, opens the bad trace, and opens its span in the playground. It lands on the exact prompt revision with the CV pre-filled.
3. They add Fluent German (the company's working language) to the must-have requirements and rerun. overall_match flips to false and German shows up in missing_requirements, while tech_match and experience_match stay true.
4. They add the CV to the test set as a new case with only expected_overall_match = false, leaving the other two expected columns empty.
5. They run an evaluation comparing the deployed prompt against the new one. The old prompt fails the new case; the new prompt passes it and keeps every original case passing, because every other candidate in the test set speaks German.
6. They deploy the new revision. The Streamlit app picks it up on the next screening, no code change.

Walkthrough video

What's in the example

• config.py: the job spec, the prompt, and the structured-output JSON schema.
• prepare_testset.py: builds data/testset.csv (27 curated CVs from the public opensporks/resumes dataset, all German speakers) and can upload it to Agenta. The demo candidate is deliberately excluded so the walkthrough adds it.
• setup_app.py: idempotent setup. Creates the app and a default variant, sets the completion URI on every revision, deploys with the environment reference map, and archives any stray auto-named variant.
• generate_traces.py: seeds screening traces (no feedback) before a demo.
• screening.py, app.py: the AI logic and the Streamlit UI.
• make_sample_pdfs.py, data/sample_cvs/: four sample CV PDFs, including the no-German demo candidate and a German-speaking strong match.
• AGENTS.md: setup steps and the SDK/endpoint gotchas found while building this (completion URI, variant naming, deploy reference map).

Notes

• The demo beats are verified, not assumed. Borderline CVs are model-unstable, so the labels follow the model's stable consensus, and the German requirement is added at the top of the must-haves where the flip is reliable (5/5 runs).
• Pre-ran the before/after evaluation over the full test set twice: before scores 0.96 (only the new German case fails), after scores 1.00 with no regressions.
• The code evaluator that scores each expected_* column is in Readme.md. Test it with a real evaluation run, not the evaluator test panel, which does not pass test set columns (AGE-3825).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jun 15, 2026 8:38am

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces a complete, production-ready CV screening example for the Python SDK. It includes shared configuration with a structured JSON schema for classification results, scripts to prepare a curated test set from external resume data, an Agenta deployment script, and an interactive Streamlit demo that fetches prompts from Agenta, runs LLM screening, and collects user feedback as trace annotations.

Changes

CV Screening Example

Layer / File(s)	Summary
Overview, Documentation, and Infrastructure examples/python/Readme.md, examples/python/cv-screening/Readme.md, examples/python/cv-screening/.env.example, examples/python/cv-screening/requirements.txt, examples/python/cv-screening/data/.gitignore	Root README adds CV screening to use cases table. New cv-screening README documents the full end-to-end workflow. Environment template defines AGENTA_API_KEY, AGENTA_HOST, and OPENAI_API_KEY. Dependencies include Agenta SDK, OpenAI client, Streamlit, test-data tools, and PDF generation.
Shared Configuration and Agenta Deployment examples/python/cv-screening/config.py, examples/python/cv-screening/create_app.py	config.py defines app/variant slugs, system/user prompts with {cv} template, a strict JSON schema enforcing scores (1–5), requirement lists, classification enum, and reasoning. create_app.py initializes the Agenta client, creates the service completion app, publishes the prompt variant with the schema-based LLM config, and deploys to production.
Test Set Preparation and Sample Generation examples/python/cv-screening/Readme.md (test set docs), examples/python/cv-screening/prepare_testset.py, examples/python/cv-screening/make_sample_pdfs.py	README documents test set construction from external resume dataset with curated IT-manager classifications. prepare_testset.py downloads a public Hugging Face parquet, applies hand-curated resume ID mappings, converts resume HTML to Markdown, writes data/testset.csv, and optionally uploads to Agenta. make_sample_pdfs.py renders selected test CVs as PDF files under data/sample_cvs/ with text normalization and FPDF styling.
Interactive Demo with PDF Upload and Feedback examples/python/cv-screening/Readme.md (demo walkthrough), examples/python/cv-screening/app.py	README walks through setup, Agenta deployment, test-set upload, playground iteration, Streamlit demo run, and feedback collection. app.py provides a Streamlit dashboard: PDF upload and Markdown conversion, fetches the production prompt from Agenta (with local fallback), invokes OpenAI chat completion with the schema, captures Agenta trace invocation IDs, renders classification banner and per-area score metrics with progress bars, requirement lists, and a feedback form (thumbs up/down + optional comment) that posts to Agenta trace data and triggers a success rerun.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant StreamlitApp as Streamlit App
  participant Agenta
  participant OpenAI
  
  User->>StreamlitApp: Upload CV PDF
  StreamlitApp->>StreamlitApp: Convert PDF to Markdown
  StreamlitApp->>Agenta: Fetch production prompt config
  Agenta-->>StreamlitApp: Return prompt + LLM config
  User->>StreamlitApp: Click "Screen CV" button
  StreamlitApp->>OpenAI: Call chat completion<br/>with prompt + schema
  OpenAI-->>StreamlitApp: Return structured JSON<br/>(scores, requirements, classification)
  StreamlitApp->>StreamlitApp: Render classification banner<br/>+ score metrics + requirements
  StreamlitApp->>Agenta: Capture trace invocation ID
  User->>StreamlitApp: Submit feedback<br/>(thumbs up/down + comment)
  StreamlitApp->>Agenta: POST feedback as trace annotation
  Agenta-->>StreamlitApp: Success response (200/202)
  StreamlitApp-->>User: Show feedback confirmation

Loading

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 21.05% which is insufficient. The required threshold is 60.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately captures the main feature being added: a CV screening demo with feedback integration and evaluation walkthrough workflow.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset, detailing the walkthrough story, components, and implementation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/cv-classifier-demo-oug3jb

Warning

Review ran into problems

🔥 Problems

Stopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a @coderabbit review after the pipeline has finished.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

examples/python/cv-screening/requirements.txt (1)

1-19: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Pin dependency versions to avoid pulling vulnerable packages.

The requirements file specifies no version constraints, which means pip install will fetch the latest versions of all packages and their transitive dependencies. OSV Scanner has flagged numerous critical and high-severity vulnerabilities in transitive dependencies that could be pulled in, including:

• aiohttp: 23 CRITICAL issues (SSRF, header injection, DoS, credential leaks)
• gitpython: 9 CRITICAL issues (RCE, path traversal, arbitrary code execution)
• litellm: 13 CRITICAL issues (SSTI, SQL injection, SSRF, eval-based RCE)
• pillow: 6 CRITICAL issues (arbitrary code execution, buffer overflow, DoS)
• pyarrow: 3 CRITICAL issues (arbitrary code execution)

While this is example code, users may run it in environments connected to real data or networks. Unpinned dependencies create a supply-chain risk.

🔒 Recommendation

Generate a pinned requirements.txt by running:

pip install -r requirements.txt
pip freeze > requirements.txt

Then review the frozen versions and update any packages flagged by pip-audit or OSV Scanner. Alternatively, specify minimum safe versions inline:

 # Agenta SDK + LLM client
-agenta
-openai
-python-dotenv
+agenta>=0.28.0
+openai>=1.0.0
+python-dotenv>=1.0.0

For the remaining packages, apply the same pattern after verifying secure minimum versions.

🧹 Nitpick comments (1)

examples/python/cv-screening/Readme.md (1)

20-22: 💤 Low value

Add language identifier to code fence.

The code fence starting at line 20 lacks a language identifier, triggering a markdownlint warning (MD040). While this is ASCII art rather than code, specifying text or leaving it as triple-backticks with no syntax highlighting improves consistency.

📝 Proposed fix

-```
+```text
 PDF upload ──> Markdown (markitdown) ──> prompt fetched from Agenta ──> LLM ──> structured scores

</details>

<!-- cr-comment:v1:63b9a63971a5e8574a05aef6 -->

</blockquote></details>

</blockquote></details>

---

<details>
<summary>ℹ️ Review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Organization UI

**Review profile**: CHILL

**Plan**: Pro Plus

**Run ID**: `7c1b7401-bc27-47de-b94d-d0d734c5558f`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between aed2d47357cc8d88347011835c7cc1f3f7f08ea7 and c28d1a2dca9c1982a3b8885929de57447d80e256.

</details>

<details>
<summary>⛔ Files ignored due to path filters (4)</summary>

* `examples/python/cv-screening/data/sample_cvs/candidate_chef.pdf` is excluded by `!**/*.pdf`
* `examples/python/cv-screening/data/sample_cvs/candidate_it_manager.pdf` is excluded by `!**/*.pdf`
* `examples/python/cv-screening/data/sample_cvs/candidate_it_supervisor.pdf` is excluded by `!**/*.pdf`
* `examples/python/cv-screening/data/testset.csv` is excluded by `!**/*.csv`

</details>

<details>
<summary>📒 Files selected for processing (10)</summary>

* `examples/python/Readme.md`
* `examples/python/cv-screening/.env.example`
* `examples/python/cv-screening/Readme.md`
* `examples/python/cv-screening/app.py`
* `examples/python/cv-screening/config.py`
* `examples/python/cv-screening/create_app.py`
* `examples/python/cv-screening/data/.gitignore`
* `examples/python/cv-screening/make_sample_pdfs.py`
* `examples/python/cv-screening/prepare_testset.py`
* `examples/python/cv-screening/requirements.txt`

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->