Task: weigh unknown_audience_ttl 0 (Keycloak load and outage cost) against a restart-free invalidation #253

Background

PR #242 sets token-exchange-aiac's unknown_audience_ttl to "0" on the outbound leg (k8s/opa-kind-enable.sh; PRD D36). The plugin's default, 5 minutes, remembers an audience that has no Keycloak client. So a caller that called a target before the target's client existed would pass its calls through un-exchanged (the target answers 401) until the cache expired or the caller restarted. With 0 a new target needs no restart of its callers.

The cost, raised in the review of #242:

• Load and latency. Every call to an in-cluster host that has no client first sends a token-exchange request to Keycloak.
• Availability. During a Keycloak outage such a call gets 503 upstream.token-exchange-failed, although it never needed an exchange. With the default, a warm host would ride out a short outage.

AIAC accepts this for now: its agents rarely call in-cluster hosts that have no client. The LLM endpoint and the telemetry collector are outside the cluster, and the plugin never offers those to Keycloak (host_not_in_cluster).

Task

Find a way to keep a non-zero unknown_audience_ttl without restarting callers when a target appears. Options to weigh:

1. A restart-free invalidation in cortex. The plugin drops one cached audience when its client is created, through an admin endpoint on the sidecar or a watch of Keycloak events. That needs a cortex change.
2. A short non-zero TTL (for example 30 s). A new target's first calls still pass through un-exchanged for up to that long; PRD D36 lists this option as rejected for that reason.
3. Keep 0, and record the load and availability cost as accepted (it is in D36 today).

Files

• k8s/opa-kind-enable.sh: unknown_audience_ttl.
• k8s/authbridge-pipeline-check.py: the check that requires a zero duration.
• docs/specs/PRD.md D36: the TTL bullet.
• docs/testing/uc1-onboarding-pipeline.md: the Testing Decisions entry on the TTL.

Acceptance criteria

• A decision between the options above, recorded in PRD D36.
• If the TTL becomes non-zero: a caller of a new target is exchanged without a restart, shown by the system lane.