feat: KAITO Guard - move guardrail from RAGEngine to Istio gateway layer #226

Summary

Sink the 8 guardrail scanners validated in RAGEngine down to the Istio gateway layer via an Envoy ext_proc service. Reuse the existing PoC's SSE streaming logic, evolving from hard-coded patterns to a configurable production system that protects all LLM traffic at the gateway.

Phase-by-Phase PR Breakdown

Phase 1: Foundation - Streaming Infrastructure (3 PRs) ✅ IN PROGRESS

Scope: SSE streaming pipeline + cross-event pattern detection

• PR-1.1: Skeleton + Dependencies (~80 LOC, 3 tests)
  • Basic gRPC server structure, dependencies, project setup
• PR-1.2: SSE Parser (~100 LOC, 16 tests)
  • split_sse_events(), parse_sse_event(), is_done_event()
  • Handle incomplete events, malformed JSON, OpenAI delta format
• PR-1.3: Logical Stream + Holdback (~90 LOC, 34 tests)
  • LogicalStreamBuffer: Accumulate delta.content across SSE events
  • release_safe_prefix(): Cross-boundary pattern detection via holdback mechanism

Total Phase 1: ~270 LOC + 53 tests, 100% pass ✅

Dependency: None. Phase 1 is self-contained.

Deliverable: Production-ready streaming pipeline that can parse SSE, buffer content, and detect patterns split across event boundaries.

Phase 2: Scanner Framework (4 PRs) 🔄 PLANNED

Scope: Pluggable pattern detection architecture

• PR-2.1: Scanner Interface + Registry (~110 LOC, 23 tests)

  • Scanner abstract base class
  • ScanResult dataclass with validation
  • ScannerRegistry for lifecycle management

• PR-2.2: GuardEngine Coordinator (~110 LOC, 18 tests)

  • Orchestrate LogicalStreamBuffer + Holdback + ScannerRegistry
  • process_delta(): Stream incoming content
  • flush(): End-of-stream final scan
  • scan_history(), get_detections()

• PR-2.3: BanSubstrings Reference Scanner (~80 LOC, 33 tests)

  • Reference implementation showing Scanner interface
  • Exact substring matching (case-sensitive)
  • Multi-pattern support with priority order

• PR-2.4: Additional Scanners (Regex, Secrets, etc.)

  • Regex pattern matching
  • Secret detection + optional redaction (partial/hash modes)
  • Sensitive data (PII: email, phone, credit card, IP)

Total Phase 2: ~310 LOC + 74 tests (estimated)

Dependency: Phase 1 (requires SSE parser, logical stream, holdback)

Decision: These Scanners can be:

• Self-contained in production-stack (recommended for Phase 1-3)
• OR integrated with kaito#2378 guardrails_core later (Phase 4+)

Phase 3: Server Integration (2 PRs) 🔄 PLANNED

Scope: Envoy ext_proc gRPC integration + E2E

• PR-3.1: gRPC Server + ext_proc Adapter (~150 LOC)

  • Implement ExternalProcessorServicer.Process()
  • Handle ProcessingRequest/ProcessingResponse
  • Connect GuardEngine to Envoy request/response flow

• PR-3.2: E2E Tests + Local Deployment (~100 LOC, 20+ tests)

  • End-to-end integration tests with mock Envoy
  • Docker Compose for local testing
  • Example EnvoyFilter configuration

Total Phase 3: ~250 LOC + 20+ tests (estimated)

Dependency: Phase 1-2

Deliverable: Production-ready Envoy ext_proc service deployable on Kubernetes

Phase 4-8: Advanced Features 🔄 FUTURE

Phase 4: Output Redaction + Blocking (redact pattern, replace with [REDACTED])

Phase 5: Metrics + Monitoring (Prometheus metrics, structured logging)

Phase 6: Config Management (YAML policy, hot reload, k8s ConfigMap watcher)

Phase 7: Performance Optimization (caching, async scanners, batch processing)

Phase 8: Production Hardening (timeouts, circuit breakers, fallback behavior)

Progress Summary

Current PR Status

Phase 1 PRs (Open):

• PR PR-1.1: Envoy ext_proc gRPC server skeleton #227: Skeleton + Dependencies
• PR PR-1.2: SSE Parser (~100 LOC + 100 tests) #228: SSE Parser
• PR PR-1.3: Logical Stream + Holdback (~90 LOC + 100 tests) #229: Logical Stream + Holdback

Review Notes:

• Each PR is independently reviewable but should merge in order (PR-1.1 → PR-1.2 → PR-1.3)
• No external dependencies beyond Python stdlib + grpcio/protobuf/pydantic
• All 53 tests pass locally

Architecture

Request path (routing):
```
Client ───────► Envoy Gateway ───────► BBR/EPP ───────► vLLM
```

Response path (output guarding):
```
vLLM
│
│ SSE streaming
▼
Client ◄────── Envoy Gateway ◄──────────────────────────┤
│ │
│ ext_proc gRPC (FULL_DUPLEX) │
▼ │
KAITO Guard (Output Guard) ◄──────────────┘

           Phase 1-2: Stream processing + scanning
           Phase 3: gRPC integration + responses
           Phase 4+: Advanced features

```

Key Design Decisions

• Service: Envoy ext_proc via Python for dev velocity + RAGEngine reuse. Go remains fallback option.

• Streaming: Both Streaming (incremental) and Buffered (full content) scanner modes supported.

• Holdback Mechanism: Prevent pattern leakage across SSE event boundaries by holding back (pattern_length - 1) bytes.

• Scanner Architecture: Interface-based + config-driven. New scanner types can be added without modifying core.

• Failure Mode: Per-policy fail-open/fail-closed, with secure defaults.

• Configuration: request_body_mode: NONE; response_body_mode: FULL_DUPLEX_STREAMED (output guard only).

Related Issues

• kaito#2378: Decouple RAGEngine guardrail core from llm-guard (parallel effort, Phase 4+)

Labels

• area/guardrail
• area/gateway
• type/feature
• priority/high