Skip to content

Add a research-backed acceptance harness for keep (poc zero-trust access stack google) #74

@haasonsaas

Description

@haasonsaas

Summary

Convert the repo's latent product contract into a repeatable benchmark suite with explicit pass/fail evidence.

This issue was generated from an org-wide EvalOps mining pass on 2026-05-10 07:57 UTC. It combines live GitHub repo signals with a per-repo arXiv search. Treat the research links as grounding for a concrete implementation, not as a request for a literature review.

Repo Evidence

  • Repository description: PoC zero-trust access stack with Google SSO, Envoy, OPA, and device attestation
  • Tree signals: 1 docs files, 4 workflows, 0 proto files, 6 test-like files.
  • README.md:195 includes latent-spec language: - Device posture updates are unauthenticated; real attestation agent verification is not implemented. - Secrets are sourced from environment variables/ConfigMaps—no dedicated KMS integration.
  • README.md:295 includes latent-spec language: ## Future Enhancements
  • agent/internal/posture/posture_test.go:141 includes latent-spec language: if ts != TrustStatusUnknown { t.Fatalf("zero value should be TrustStatusUnknown, got %v", ts) }
  • agent/internal/posture/posture_test.go:300 includes latent-spec language: {emptyString, DefaultRules}, {decimalValue, DefaultRules}, // Should fail for non-integer }

Research Grounding

Repo axes: security, governance, evaluation, tooling

Search keywords: device, input, posture, authz, access, make, envoy, inventory, opa, services, use, google

  • arXiv:2511.15759v1 Securing AI Agents Against Prompt Injection Attacks (Badrinath Ramakrishnan, Akshaya Balaji), 2025.
  • arXiv:2509.14285v4 A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks (S M Asif Hossain, Ruksat Khan Shayoni, Mohd Ruhul Ameen, Akif Islam, M. F. Mridha, Jungpil Shin), 2025.
  • arXiv:2504.18575v3 WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, Kamalika Chaudhuri), 2025.
  • arXiv:2410.23308v1 Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures (Victoria Benjamin, Emily Braca, Israel Carter, Hafsa Kanchwala, Nava Khojasteh, Charly Landow), 2024.
  • arXiv:2510.03705v1 Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods (Yulin Chen, Haoran Li, Yuan Sui, Yangqiu Song, Bryan Hooi), 2025.
  • arXiv:2510.21203v1 The Nuclear Analogy in AI Governance Research (Sophia Hatz), 2025.
  • arXiv:2012.09344v2 Machine Learning for Detecting Data Exfiltration: A Review (Bushra Sabir, Faheem Ullah, M. Ali Babar, Raj Gaire), 2020.
  • arXiv:2410.05451v3 SecAlign: Defending Against Prompt Injection with Preference Optimization (Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, Chuan Guo), 2024.
  • arXiv:2408.00925v1 WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes (Victor Valbuena), 2024.
  • arXiv:2505.11717v4 WebInject: Prompt Injection Attack to Web Agents (Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, Neil Zhenqiang Gong), 2025.

What To Build

  • Define the smallest representative keep golden workflow and capture expected inputs, outputs, and evidence artifacts.
  • Add fixtures for a successful path, an ambiguous/degraded path, and a failure path.
  • Publish a command that local agents and CI can run before shipping related changes.

Acceptance Criteria

  • A short design note names the repo-specific workflow, threat or correctness model, and the research assumptions being adopted.
  • A runnable check, fixture, or verifier exercises the new contract in CI or an equivalent local command documented in the repo.
  • The implementation emits or stores enough evidence for a downstream agent/operator to cite inputs, decisions, and outputs.
  • At least one negative/degraded-mode case is covered so failures are observable rather than silently accepted.
  • Documentation links the new behavior to the relevant EvalOps platform primitive or explicitly records why this repo remains standalone.

Notes

  • Generated issue 1/5 for evalops/keep by evalops_org_miner.py.
  • Before implementation, confirm the sampled latent-spec snippets still match main; this issue intentionally cites exact file paths/lines where the mining pass saw them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions