Skip to content

fix(tools): give federated transport errors a non-empty message - #6727

Open
Ewertonslv wants to merge 3229 commits into
IBM:mainfrom
Ewertonslv:fix/6592-empty-transport-error-message
Open

Ewertonslv wants to merge 3229 commits into
IBM:mainfrom
Ewertonslv:fix/6592-empty-transport-error-message

Conversation

@Ewertonslv

Copy link
Copy Markdown

Summary

Failures from the anyio/httpx transport family reached the caller as a bare
Tool invocation failed: — an empty detail — so a caller could not tell a closed
stream from a read timeout from a broken resource.

Closes #6592

Root cause

The outer except BaseException handler of invoke_tool builds the caller-facing
message as error_message = str(root_cause) (mcpgateway/services/tool_service.py:7029
on main@5273986b3). str() of that whole transport family is '':

anyio 4.14.0, httpx 0.28.1
ClosedResourceError   ''
EndOfStream           ''
BrokenResourceError   ''
httpx.ReadTimeout     ''

The exception class is logged server-side but never reaches the message. The empty
string flows to all three consumers of error_message: the error span, the
ToolResult handed to the post-invoke hooks, and the final ToolInvocationError.

A second, related gap: a read timeout raised inside the MCP SDK's TaskGroup arrives
wrapped in an ExceptionGroup, so it misses the per-transport
except (asyncio.TimeoutError, httpx.TimeoutException) lanes that emit the typed
Tool invocation timed out after {effective_timeout}s message, and lands in the same
outer handler — where it also stringifies to ''.

Fix

Two changes at the one construction site:

  • str(root_cause) or "<module>.<qualname>" — byte-identical for every exception whose
    str() is already non-empty (e.g. httpx.HTTPStatusError for a 503), so nothing that
    works today changes.
  • A group-wrapped timeout is restated in the vocabulary the per-transport lanes already
    emit, so callers key on one wording regardless of how the timeout arrived.

This is the shape proposed by the reporter in the issue.

Tests

Three tests added to tests/unit/mcpgateway/services/test_tool_service.py, mirroring the
existing test_invoke_tool_error_exception_group_unwrapping:

  • test_invoke_tool_error_empty_str_falls_back_to_class_name — anyio.ClosedResourceError
    now names its class instead of yielding an empty detail.
  • test_invoke_tool_group_wrapped_timeout_uses_timeout_vocabulary — an ExceptionGroup-wrapped
    httpx.ReadTimeout is reported as a timeout.
  • test_invoke_tool_error_non_empty_str_is_unchanged — regression guard: an exception carrying
    a real message still produces exactly Tool invocation failed: upstream said no.

Before/after (stash only the fix, keep the tests):

without the fix:  F F .
    assert 'ClosedResourceError' in 'Tool invocation failed: '
    assert 'Tool invocation timed out after' in 'Tool invocation failed: '
with the fix:     . . .

The regression test passes in both states, as it should.

Verification

  • pytest tests/unit/mcpgateway/services/test_tool_service.py — passes (exit 0)
  • pytest tests/unit/mcpgateway/services/test_tool_service_coverage.py — passes (exit 0)
  • ruff check — All checks passed
  • ruff format --diff — 2 files already formatted

Note on overlap

PR #6181 also touches tool_service.py error handling, but at a different level — it
changes the envelope (returning CallToolResult(isError=True) instead of raising),
while this changes the content of error_message. A textual merge conflict in that
region is possible; happy to rebase on whichever lands first.

prakhar-singh1928 and others added 30 commits July 13, 2026 15:40
* updated python dependencies

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* updated uv.lock/pyproject.toml

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

---------

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
…ker (IBM#5406)

* feat(plugins): elect a single primary worker for non-hook plugin init

Under multiple worker processes (gunicorn workers > 1) every worker loads
the plugin manager and runs each plugin's initialize(). A non-hook plugin
that performs a side effect on startup or in a background task therefore
runs that work once per worker, duplicating it N times.

Add an opt-in helper, is_primary_worker(), so a side-effecting plugin can
gate its work to exactly one worker:

    from mcpgateway.utils.primary_worker import is_primary_worker

    async def initialize(self) -> None:
        if not is_primary_worker():
            return
        ...  # runs on one worker only

Election is file-lock based: the first process to acquire the lock is the
primary and holds it for its lifetime; other workers get False. The OS
releases the lock when the holding process exits, so if the primary is
recycled (max_requests) the next caller takes over. The lock path is
port-scoped (override via PRIMARY_WORKER_LOCK_PATH) so co-located gateway
instances don't collide while workers of one instance share a lock.

Opt-in and minimal: no changes to main.py, gunicorn config, plugin schema,
or the hook path. Plugins that don't call the helper behave exactly as
before, and hook plugins still load and execute on every worker. Election
is per host.

Includes unit tests and a local 2-worker e2e (make test-primary-worker-e2e)
that boots a real gateway with a gated marker plugin and asserts the side
effect runs once.

Closes IBM#5139

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
* refactor(csp): modernize Content Security Policy configuration

Update CSP headers to follow modern web standards and best practices.
Migrate from Tailwind CDN to precompiled CSS for improved performance
and compatibility with CSP Level 3 recommendations.

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>

* docs(csp): update CSP comments to reference CSP Level 3 standards

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>

* fix(lint): remove unnecessary f-string prefix from style-src CSP directive

The style-src CSP directive doesn't contain any interpolated variables,
so the f-string prefix is unnecessary and triggers linting warnings.

Fixes ruff F541 and pylint W1309 violations.

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>

* feat: migrate Tailwind config to external CSS for strict CSP compliance

- Remove inline Tailwind config script from login.html (51 lines)
- Add auth-animations.css link to login, change-password-required, and admin templates
- Enhance CSP documentation with clickjacking mitigation details
- Add CDN allowlist rationale (Font Awesome, CodeMirror, Chart.js, Marked, DOMPurify)
- Update CSP migration doc to mark HTMX unsafe-eval as RESOLVED

Addresses review comments from PR IBM#5111.

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>

* docs(csp): address security review feedback on CSP migration documentation

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>

* Resolve conflict

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* fixed the pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed the pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed the pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Rakhi Dutta <rakhibiswas@yahoo.com>
Signed-off-by: Gabriel Costa <gabrielcg@proton.me>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Gabriel Costa <gabrielcg@proton.me>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
…ce name (IBM#5158)

* fix: show meaningful 409 message for duplicate resource name

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix: address regressions in resource name conflict handling

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix: resolve lint, alembic branch, and diff-cover failures in CI

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix for additional changes

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix for test failures

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix: address review feedback on resource name conflict handling

Batch the bulk-register name-conflict lookup instead of querying per
resource, remove an unreachable team_id guard, map ResourceValidationError
to 422 in the create endpoint, and add migration/e2e test coverage for
the duplicate-name constraint.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: enforce resource name uniqueness for public/private and rename paths

team_id is NULL for public/private resources, and SQL treats NULLs as
distinct in unique indexes, so the DB backstop only ever fired for
team-scoped rows. Wrap team_id in COALESCE(team_id, '') in both resource
name-uniqueness indexes so public/private duplicates are actually blocked,
and add matching name pre-checks for private-scope create/bulk-import and
for the resource rename path (update_resource had none before), which was
silently allowed to collide.

Also fix migration idempotency: SQLAlchemy's Inspector.get_indexes() drops
expression-based indexes from reflection, so the existing-index check
needs to read the DB catalog directly instead.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(resources): propagate ResourceValidationError unwrapped, add diff-cover tests

register_resource's generic except-Exception handler was rewrapping
ResourceValidationError (team-scoped create without team_id) into a plain
ResourceError, so the REST layer's dedicated 422 handler for
ResourceValidationError could never fire and callers got a 400 instead.

Also adds unit/integration coverage for the team- and private-visibility
name-conflict branches, the update_resource team-scoped rename conflict
path, and the update_resource ContentPatternError branch, to satisfy the
diff-cover threshold.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
* fix(a2a): pass allowlisted headers to tool pre-invoke hooks

Signed-off-by: King Star <mcxin.y@gmail.com>

* docs: cover A2A passthrough headers

Signed-off-by: King Star <mcxin.y@gmail.com>

* fix(a2a): keep sensitive headers out of tool hooks

Signed-off-by: King Star <mcxin.y@gmail.com>

* fix(a2a): honor global header allowlist for plugin updates

Signed-off-by: King Star <mcxin.y@gmail.com>

* fix(a2a): align unset passthrough headers

Signed-off-by: King Star <mcxin.y@gmail.com>

* test(a2a): use complete plugin results

Signed-off-by: King Star <mcxin.y@gmail.com>

---------

Signed-off-by: King Star <mcxin.y@gmail.com>
* Package all js libs

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Compress (gzip) bundle

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Divide JS into chunks

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Edit docs

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Remove ui_airgapped flag from HTML files

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Update docs (again)

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Undo Containerfile.lite restore

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Cleanup CDN sri management

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Address comments

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Reduce lazy-load scope

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Add CSS import fallback

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Fix secrets baseline

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Rebase

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Fix failing tests

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

---------

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>
* Fix templated resource proxy reads

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: address templated resource review feedback

Signed-off-by: lucarlig <luca.carlig@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: Gabriel Costa <gabrielcg@proton.me>
* feat: add MCP Apps extension substrate

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* docs: document MCP Apps extension substrate

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: linearize MCP Apps migration

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: cover MCP Apps extension branches

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* feat: fixing mcp apps rendering

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: linting cleanop

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: formatting and linting

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* docs: clarify MCP Apps UI behavior

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* feat: code improvements

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: breaking test

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: refresh branch after rebase

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: linearize MCP Apps migration after rebase

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: stabilize MCP Apps CI failures

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: harden mcp apps security substrate

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: refresh mcp apps branch after rebase

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: cover mcp apps ci gaps

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: support mcp app helper tools in clients

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: stabilize router and wheel checks

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: satisfy pylint for app tool proxy

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: refresh validation metadata

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: harden MCP Apps tool visibility

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: align MCP Apps metadata contract

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: fix MCP Apps UI resource metadata assertion

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: keep pyproject aligned with main

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* feat: addressed comments

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: breaking test

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: address MCP Apps review feedback

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: keep app tools behind AppBridge

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: trim unused MCP Apps registry helpers

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: refresh secrets baseline

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: keep app-only tools out of model listings

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: handle AppBridge resource lookup failures

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: finalize MCP Apps branch validation

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

---------

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>
…otifications now relayed (IBM#5603)

The POST-correlated notification relay fix closes GAP-001 (server-initiated
log notifications) and GAP-002 (progress notifications) on gateway_proxy
and gateway_virtual targets.

- Remove xfail_on from test_log_message_reaches_client and
  test_progress_notifications_delivered
- Move GAP-001 and GAP-002 to Closed gaps in COMPLIANCE_GAPS.md
- Update GAP-011 cross-reference to exclude closed gaps

Signed-off-by: Jonathan Springer <jps@s390x.com>
* test: retry live RBAC per-server access

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* feat: make dataplane publisher snapshot interval configurable

The dataplane publisher interval was a hardcoded 60s module constant,
so runtime-created users and servers stayed invisible to the dataplane
for up to a full minute. Expose it as
DATAPLANE_PUBLISHER_INTERVAL_SECONDS (default 60, unchanged) and derive
the UserConfig key TTL as interval + 10, preserving the current 60/70
relationship. Functional test stacks can now run a short interval while
load benchmarks keep the default.

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: cover a full publisher interval in RBAC allow-path retries

A 5 x 100ms retry window cannot cover the dataplane publisher snapshot
cycle (default 60s), so the allow-path tests still failed on the split
stack. Switch the helper to a deadline-based wait that covers one full
publish interval plus slack by default (75s), tunable via
MCP_E2E_PUBLISHER_SYNC_DEADLINE for stacks that run a short
DATAPLANE_PUBLISHER_INTERVAL_SECONDS.

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: drop publisher sync deadline env var

The deadline only bounds how long a genuine failure takes; passing runs
return as soon as the publisher converges, so a fixed 75s covering one
default publish interval plus slack works on every stack and the knob
adds nothing.

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: make stdin reader test deterministic

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: trim RBAC retry comments

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: assert allow-path tools lists are non-empty

The per-server allow-path checks only asserted that tools/list
succeeded, so a deployment that silently drops the server's backend
passed them with an empty list. Assert the visibility servers actually
advertise their associated tools.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
(cherry picked from commit 3eea93d)

* fix: apply dataplane publisher interval dynamically

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: make publisher sync deadline configurable

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* chore: refresh secrets baseline

Signed-off-by: lucarlig <luca.carlig@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Co-authored-by: Lang-Akshay <akshay.shinde26@ibm.com>
* fix(output-length-guard): enforce limits on resource.text in content array

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* test(output-length-guard): fix always-skipped double-truncation test

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* test(output-length-guard): trim docstring for double-truncation test

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* chore: merge main and update .secrets.baseline line numbers

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Lang-Akshay <akshay.shinde26@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
…BM#5622)

* Update secrets and reset some of content from .secrets.baseline file

Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>

* Updates with linting formatting

Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>

* Update to pass grype issues with container build

Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>

---------

Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>
IBM#5515)

The compliance virtual_server fixture connected to /servers/{id}/mcp
immediately after creation. On split deployments that route is served
from asynchronously published dataplane config, so early gateway_virtual
rows failed with 404/400 before the first snapshot landed. Probe the
route until it accepts (75s deadline, one default publish interval plus
slack); control-plane-only deployments return on the first probe.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
…ts that run on Go (IBM#5596)

* fix: remove premature exit in Containerfile and restore Go linting targets

- Containerfile: Remove stray "exit 0" inside the ENABLE_RUST=false
  fallback block that caused a buildkit warning and would short-circuit
  the build when ENABLE_RUST=true with a trailing backslash.

- Makefile: Add explicit GOPATH, GOMODCACHE, GOCACHE, and GOBIN
  environment variables to all Go-based linting targets (actionlint,
  reviewdog, chart-testing, kube-linter) so they function correctly
  regardless of the host Go toolchain configuration.

Signed-off-by: Jonathan Springer <jps@s390x.com>

* Remove LINT_GO_TOOLCHAIN

Signed-off-by: Jonathan Springer <jps@s390x.com>

---------

Signed-off-by: Jonathan Springer <jps@s390x.com>
Co-authored-by: Lang-Akshay <akshay.shinde26@ibm.com>
… connections (IBM#4724)

* fix(redis): raise maxclients from 10K to 30K to support 10 replicas. (IBM#4630)
- Update maxclients to 30,000 in all docker-compose files
- Add redis.maxclients configuration to Helm chart (values.yaml, schema, configmap)
- Update capacity formula comments: replicas × workers × pool < maxclients
- Add documentation about server-side vs client-side connection limits
- Fixes IBM#4630: Redis maxclients exceeded at 10 replicas

Formula: 10 replicas × 24 workers × 100 pool = 24,000 connections
With 20% headroom: 28,800 → 30,000 maxclients

This addresses the root cause (server-side capacity) rather than
degrading client-side performance by reducing connection pools.

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

Change maxclients to 15000 and connection pool size to 50

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

Replace accidentally deleted config: line

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

* Pass maxclients as fallback and fix math calculation error in comment

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

update baseline secrets

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

* add updates for docs

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

Update secrets baseline and resolve conflict

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

fixed pre-commit

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

* fix pre-commit

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

* fix dead anchor in configuration.md, add helm-unittest for deployment-redis

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

---------

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>
* Fix playwright tests

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Adress comments

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Fix some more failing tests

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Address comments

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

* Address comments

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>

---------

Signed-off-by: Gabriel Costa <gabrielcg@proton.me>
…s/{tool_id} (IBM#5314)

* fix(schemas): extend ToolCreate/ToolUpdate assemble_auth to handle auth_headers array for authheaders auth type

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix tests

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* chore: update .secrets.baseline line numbers after rebase

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix(schemas): unify authheaders array validation across tool/gateway/a2a schemas

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>

* fix: reject invalid auth_headers with 422 and share header validation

Signed-off-by: Marek Dano <Marek.Dano@ibm.com>

* fix(schemas): cap auth_headers length before iterating and correct unset semantics

Signed-off-by: Marek Dano <Marek.Dano@ibm.com>

* chore: update .secrets.baseline line numbers after schemas.py changes

Signed-off-by: Marek Dano <Marek.Dano@ibm.com>

* fix(admin): treat non-list auth_headers form value as no headers instead of 500

Signed-off-by: Marek Dano <Marek.Dano@ibm.com>

* chore: update .secrets.baseline line numbers after test_admin.py changes

Signed-off-by: Marek Dano <Marek.Dano@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Shoumi <shoumimukherjee@gmail.com>
Signed-off-by: Marek Dano <Marek.Dano@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Marek Dano <Marek.Dano@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
IBM#5611)

* feat(plugins): extend S4 metrics allowlist for 5 new cpex-* plugins

Extends _SAFE_STRING_FIELD_NAMES / _SAFE_NUMERIC_FIELD_NAMES so
secrets_detection, encoded_exfil_detection, url_reputation, rate_limiter,
and retry_with_backoff metrics fields survive sanitization instead of
being silently dropped as not-yet-allowlisted field names.

Part of IBM#5554.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* feat(plugins): bump cpex-secrets-detection floor to 0.3.7, document metrics contract

cpex-secrets-detection 0.3.7 is the first release that emits
result.metadata["secrets_detection"] for the S4 metrics pipeline; bump the
pyproject.toml floor to match, following the cpex-pii-filter>=0.3.6 comment
convention from IBM#5470.

Deliberately do NOT flip plugins/config.yaml's SecretsDetection mode from
"disabled" to "sequential". Per cpex.framework.PluginMode / manager.py
load_plugins(), a "disabled" plugin is never instantiated or registered --
its hooks never run at all. This block's config already has
block_on_detection: true and min_findings_to_block: 1, so enabling it would
newly start blocking tool calls/prompts/resources on secret-shaped content
for every existing deployment using the default config -- a materially
bigger change than "add metrics", and bigger than the pii_filter precedent
(which stays disabled by design, with block_on_detection: false even in its
documented enabled state). Add version: "0.3.7" plus a comment documenting
this decision instead.

Extend the internal-observability.md plugin-metrics doc with a per-plugin
field reference table (pii_filter baseline + secrets_detection's
total_detections/total_masked/total_blocked/secret_types) and note that
metrics require both the version floor and an enabled mode.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* feat(plugins): bump cpex-encoded-exfil-detection floor to 0.3.6, document metrics contract

cpex-encoded-exfil-detection 0.3.6 is the first release that emits
result.metadata["encoded_exfil_detection"] for the S4 metrics pipeline; bump
the pyproject.toml floor to match, following the cpex-pii-filter>=0.3.6 and
cpex-secrets-detection>=0.3.7 comment convention from IBM#5470 and the prior
secrets_detection pin bump.

Deliberately do NOT flip plugins/config.yaml's EncodedExfilDetector mode from
"disabled" to "sequential". Per cpex.framework.PluginMode / manager.py
load_plugins(), a "disabled" plugin is never instantiated or registered --
its hooks never run at all. This block's config already has
block_on_detection: true and min_findings_to_block: 1, so enabling it would
newly start blocking tool calls/prompts/resources on encoded-payload-shaped
content for every existing deployment using the default config -- the same
materially bigger change documented for secrets_detection. Add
version: "0.3.6" plus a comment documenting this decision instead, and clean
up the stale commented-out `# mode: "sequential"` line that was left over
from before this decision was settled.

Extend the internal-observability.md plugin-metrics doc with
encoded_exfil_detection's total_detections/encoding_types/redacted fields in
the per-plugin field reference table, and note its 0.3.6 metrics floor
alongside pii_filter/secrets_detection in the dependency section.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* feat(plugins): bump cpex-url-reputation floor to 0.3.5, document metrics contract

cpex-url-reputation 0.3.5 is the first release that emits
result.metadata["url_reputation"] for the S4 metrics pipeline; bump the
pyproject.toml floor to match, following the cpex-pii-filter>=0.3.6,
cpex-secrets-detection>=0.3.7, and cpex-encoded-exfil-detection>=0.3.6 comment
convention from IBM#5470 and the prior secrets_detection/encoded_exfil_detection
pin bumps.

Deliberately do NOT flip plugins/config.yaml's URLReputationPlugin mode from
"disabled" to "sequential". Per cpex.framework.PluginMode / manager.py
load_plugins(), a "disabled" plugin is never instantiated or registered -- its
hooks never run at all. This block's config already has blocked_domains and
blocked_patterns entries, so enabling it would newly start blocking resource
fetches for every existing deployment using the default config -- the same
materially bigger change documented for secrets_detection and
encoded_exfil_detection. Add version: "0.3.5" plus a comment documenting this
decision instead.

Extend the internal-observability.md plugin-metrics doc with
url_reputation's total_checked/total_blocked/reputation_categories fields in
the per-plugin field reference table, using the actual violation-reason slugs
the plugin emits (blocked_domain, insecure_scheme, high_entropy_domain,
illegal_tld, unicode_spoofing, blocked_pattern, malformed_url,
malformed_domain, internal_error, other) rather than the malware/phishing
placeholders from an earlier draft of the plan, and note its 0.3.5 metrics
floor alongside the other three plugins.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* feat(plugins): bump cpex-rate-limiter floor to 0.1.7, document metrics contract

Bumps the cpex-rate-limiter dependency pin to >=0.1.7, the first release
that emits result.metadata["rate_limiter"] (per-call allowed/throttled
0-or-1 values plus backend) when a trace is active, and documents the
metrics contract in plugins/config.yaml and the internal observability
docs, following the same pattern established for SecretsDetection,
EncodedExfilDetector, and URLReputationPlugin. mode remains "disabled".

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* feat(plugins): bump cpex-retry-with-backoff floor to 0.3.6, document metrics contract

Bumps the cpex-retry-with-backoff dependency pin to >=0.3.6, the first
release that emits result.metadata["retry_with_backoff"] (retry_count,
retry_delay_ms per-attempt) when a trace is active, and documents the
metrics contract in plugins/config.yaml and the internal observability
docs, following the same pattern established for SecretsDetection,
EncodedExfilDetector, URLReputationPlugin, and RateLimiterPlugin.
mode remains "disabled".

Also drops the pre-existing <0.3.2 upper bound. git log -S confirms it
was introduced in IBM#5332 ("fix(ci): pin retry-with-backoff to 0.3.1")
as a deliberate CI-compat pin, not a documented runtime incompatibility
-- no CI config, workflow comment, or test file references an actual
incompatibility with versions 0.3.2-0.3.5. Since the new floor (0.3.6)
is well past the old ceiling, the ceiling is superseded and removed.
Note: this could not be verified against the real published 0.3.6
package in this sandbox; a real CI run is still needed before merge.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(plugins): add integration + e2e metrics coverage for 5 new plugins

Adds real-DB integration tests for rate_limiter, retry_with_backoff,
encoded_exfil_detection, and url_reputation in
test_plugin_metrics_consumer_integration.py (10/10 passing locally with
--with-integration), following the existing pii_filter test class's
fixtures and style exactly.

Adds a secrets_detection e2e test (TestOtelSecretsDetectionMetadataE2E)
in test_otel_plugin_metadata_e2e.py as the second Rust-core parity check
alongside pii_filter, via a new traced_app_secrets_detection fixture that
force-enables SecretsDetection with block_on_detection=false/redact=true
(mirroring PIIFilter's redact-not-block default) so record_plugin_metrics
actually gets a chance to run. Verified this test genuinely passes after
dev-installing cpex-secrets-detection into the local .venv (git-ignored,
not part of this commit); without that package installed it fails at
fixture setup with the same ModuleNotFoundError-derived AssertionError
the existing pii_filter tests already hit in this environment -- that
gap is pre-existing and out of scope here.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(deps): extend exclude-newer-package cutoffs for 5 bumped cpex floors

Tasks 2-6 each bumped one cpex-* plugin's floor version in
[project.optional-dependencies].plugins to the first release emitting
result.metadata for the S4 metrics pipeline, but none updated the
separate [tool.uv.exclude-newer-package] table, which still capped
resolution at 2026-06-29T23:59:59Z for all five. That made the new
floors (cpex-secrets-detection>=0.3.7, cpex-encoded-exfil-detection>=0.3.6,
cpex-url-reputation>=0.3.5, cpex-rate-limiter>=0.1.7,
cpex-retry-with-backoff>=0.3.6) unreachable to the resolver, so
`uv lock`/`uv run` failed before any test could run -- caught by Task 8's
verification pass.

Extend each package's cutoff to 2026-07-13T23:59:59Z, following the
cpex-pii-filter precedent (extended cutoff + explanatory comment naming
the version, its result.metadata field, and the dependent test files).
Verified against the sibling cpex-plugins repo's git history: the
version-bump commits for all five packages landed today (2026-07-13),
matching the new cutoff date.

Note: uv lock still cannot fully resolve in this sandbox -- confirmed via
direct PyPI queries that 4 of the 5 target versions (all but
cpex-secrets-detection, which is separately blocked upstream) have not
actually been published yet, independent of this cutoff fix.

Signed-off-by: Suresh Kumar Moharajan <suresh12.dmm@gmail.com>
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(plugins): correct bool-bypass doc claim and cover redacted field

The "Per-plugin metadata fields" preamble in internal-observability.md
claimed every field must appear in _SAFE_STRING_FIELD_NAMES or
_SAFE_NUMERIC_FIELD_NAMES or it's silently dropped. That's inaccurate
for bool-typed fields: _sanitize_plugin_metrics() accepts any bool
value unconditionally, before either allowlist check runs, which is
exactly why encoded_exfil_detection's redacted field survives despite
not being in either allowlist. Correct the doc to call out the bool
exception explicitly.

Also add unit coverage proving the redacted bool field survives
sanitization (previously untested), and fix a cosmetic type mismatch
in the url_reputation integration test where reputation_categories
was passed as a plain string instead of the documented list[str].

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* docs(plans): add implementation plan for issue IBM#5554 gateway rollout

Records the plan this branch executed.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(deps): bump exclude-newer-package cutoffs to 2026-07-14

cpex-plugins#131 merged and published all 5 packages today
(2026-07-14, ~13:46-14:27 UTC) rather than on 2026-07-13 as
originally guessed when the cutoff dates were set — uv's own
resolver error named the exact mismatch:
"cpex-encoded-exfil-detection v0.3.6 was published at
2026-07-14T13:46:18Z" vs a 2026-07-13T23:59:59Z cutoff. Regenerated
uv.lock against the corrected cutoffs; all 5 floors now resolve.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(tests): resolve detect-secrets false positive on AWS key placeholder

detect-secrets dedupes identical secret values within a file to a
single finding keyed by the last occurrence's line, which was masking
the actual issue: RAW_SECRET_VALUE (AWS's own published documentation
placeholder key, deliberately AWS-key-shaped to exercise the real
secrets_detection plugin) was never actually audited. Mark it inline
per detect-secrets' own suggested fix; drop the redundant restatement
of the literal value in a nearby comment.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(tests): update pre-existing rate_limiter/encoded_exfil unit tests for the metrics migration

These gateway-side tests import the real cpex_rate_limiter/
cpex_encoded_exfil_detection packages directly and never actually ran
until now (dependency resolution against the real published versions
was broken until the previous commit) -- so they'd never caught up
with the OTel metrics migration:

- rate_limiter: asserted the old unconditional flat metadata write
  (result.metadata["remaining"]/["reset_in"]), removed when the
  gateway-side review (cpex-plugins#131) trimmed the metrics dict to
  only allowed/throttled/backend. Fixed the packaged-core call-signature
  assertions to include the extensions arg the real 0.1.7 wrapper
  always passes.
- encoded_exfil_detection: asserted the old flat
  "encoded_exfil_redacted"/"encoded_exfil_count" keys, removed in the
  0.3.6 migration in favor of the namespaced, trace-gated
  result.metadata["encoded_exfil_detection"] dict. Updated the
  redaction tests (no trace context passed) to expect empty metadata,
  and modernized the two "returns metadata" tests to pass real trace
  context and assert the new schema, preserving their original intent.

Verified: both files pass in full (94 passed, 2 xfailed).
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix(plugins): bump cpex-retry-with-backoff to 0.3.7 (retry-detection fix)

0.3.6 added the result.metadata["retry_with_backoff"] emission this PR's
metrics wiring depends on, but shipped a retry-detection regression: the
plugin's is_failure read the tool_post_invoke result via PyDict::get_item
(C-level dict storage), which silently misses every key on the gateway's
CopyOnWriteDict payload-isolation wrapper (its data lives in a Python-level
overlay) -- every retryable failure looked like a success and the plugin
never retried a single call. This is why the retry-with-backoff static and
binding E2E legs were red on this PR.

0.3.7 (IBM/cpex-plugins#137, closing IBM#124) fixes retry detection by reading
the result through the Python mapping protocol instead. Bumps the pyproject
floor, the exclude-newer-package cutoff, the plugins/config.yaml version
field, and refreshes uv.lock (uv lock resolved cleanly against the new
floor; only cpex-retry-with-backoff's pin changed).

Verified: installed 0.3.7 from PyPI (not a local build) via `uv sync
--extra plugins --extra redis` and re-ran
tests/live_gateway/plugins/test_retry_with_backoff_e2e.py end-to-end
against the real fast-time-server image -- both static and binding legs
pass.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore: untrack docs/superpowers planning doc

Removes the implementation-plan doc from git tracking without deleting it
locally -- it's a planning artifact, not something that belongs in the repo
history.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): extend real observability e2e coverage to the 4 remaining plugins

Reviewer feedback on this PR: only secrets_detection (and pre-existing pii_filter) had
real traced-HTTP-pipeline coverage in tests/e2e/test_otel_plugin_metadata_e2e.py; the other
4 plugins this branch also wires metrics for (rate_limiter, retry_with_backoff,
encoded_exfil_detection, url_reputation) only had the fake-metadata-dict integration test
in tests/integration/plugins/test_plugin_metrics_consumer_integration.py, which proves the
DB-write/sanitization logic but not that the real plugin actually emits that shape over a
real traced request.

Adds one real-e2e test per plugin, each driving the real cpex-* Rust-backed plugin
(installed from PyPI per this branch's pyproject.toml floors) through a genuinely traced
HTTP request against the real gateway request pipeline, asserting the resulting
plugin.metrics.<name> span reaches GET /observability/traces/{trace_id}:

- encoded_exfil_detection: tool_post_invoke, mirrors the secrets_detection pattern exactly
  (block_on_detection=false, redact=true), asserts total_detections/encoding_types and that
  the raw encoded payload never leaks into the response.
- retry_with_backoff: tool_post_invoke against a persistently-failing mocked upstream, so the
  gateway genuinely retries (real backoff delays, real recursive re-invocation) up to
  max_retries -- asserts multiple spans with strictly increasing retry_count. This is the
  exact regression IBM/cpex-plugins#124 introduced and IBM#137 (0.3.7) fixed: against the buggy
  0.3.6, this test would see exactly one span with retry_count=0 (no retry ever detected).
- url_reputation: resource_pre_fetch, not tool_post_invoke -- registers a real resource and
  reads it back (POST/GET /resources) rather than tools/call. Along the way, discovered
  /resources/* isn't in the shipped settings.observability_include_paths allowlist, so this
  fixture patches it in for itself (and clears path_filter.py's lru_caches on setup/teardown)
  -- otherwise resource reads are never traced under the shipped defaults.
- rate_limiter: tool_pre_invoke, requires a real reachable Redis (the plugin fails open with
  empty metadata when Redis is unreachable, so there's no way to prove real metrics without
  one). tests/e2e/ has no Redis service in CI (unlike plugin-integration.yml's dedicated
  legs), so this test skips cleanly via a reachable_redis_url fixture (mirrors the existing
  otel_memory_exporter graceful-skip-when-extra-missing pattern) rather than failing CI.

Factored the ~90-line real-temp-DB/dedicated-FastAPI-app/dependency-override-auth boilerplate
shared by 3 of the 4 new fixtures into _build_traced_app() (asynccontextmanager) -- the
existing traced_app/traced_app_secrets_detection fixtures are deliberately left untouched
per their own docstrings' stated reasoning for staying self-contained; url_reputation's
resource-based flow doesn't fit the tool-registration shape at all and gets its own
dedicated fixture.

Verified: all 4 new tests pass individually and together with the existing 3
(tests/e2e/test_otel_plugin_metadata_e2e.py -- 6 passed, 1 pre-existing unrelated skip for
the optional opentelemetry-sdk extra), the rate_limiter test both passes against a real
Redis and skips cleanly with none reachable, and the full tests/e2e/ directory (74 files)
runs clean with only 3 pre-existing/unrelated skips and zero failures. pre-commit passes on
the changed file.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Signed-off-by: Suresh Kumar Moharajan <suresh12.dmm@gmail.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
…de dependabot (IBM#5623)

* Remove renovate json file and update security.md to include dependabot

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

* Update security.md wording to be accurate

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>

---------

Signed-off-by: Claudia Gray <claudiag082004@gmail.com>
…rker across instances (IBM#5430)

* feat(plugins): add redis backend for cross-instance primary-worker election

Adds an opt-in Redis backend so a single primary can be elected across multiple
gateway instances/replicas, not just per host. The default stays filelock
(per-host), so existing single-host deployments are unchanged.

- mcpgateway/services/leader_election.py: PrimaryWorkerElector with two
  backends. filelock is passive (OS releases on exit, no task); redis uses a
  lease (SET NX EX) renewed by an atomic compare-and-renew Lua heartbeat,
  released by an if-owner Lua, with a follower loop that re-acquires on expiry.
- is_primary_worker() reads the elector's cached flag for the redis backend and
  fails closed if it isn't started (a redis deployment never silently degrades
  to per-host scope).
- main.py lifespan starts the elector before plugin init (awaits the initial
  election) and stops it on shutdown; only the redis backend needs an elector.
- Config (conservative defaults): PRIMARY_WORKER_ELECTION_BACKEND=filelock,
  PRIMARY_WORKER_REDIS_KEY, PRIMARY_WORKER_LEASE_TTL,
  PRIMARY_WORKER_HEARTBEAT_INTERVAL, PRIMARY_WORKER_REDIS_UNAVAILABLE_POLICY=
  fail_closed (filelock_fallback optional).
- Tests with fakeredis cover single-primary, renew, release, follower takeover,
  fail-closed and filelock-fallback; e2e script is backend-aware.

Election under the redis backend is best-effort under network partitions; keep
side effects idempotent.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): prove hooks run on all workers and default path skips the elector

- e2e: add an ungated hook plugin and assert it loads on every worker (one
  marker line per worker), alongside the gated non-hook plugin asserting one.
  The e2e runner is backend-aware, so this holds under filelock and redis —
  election only gates the opt-in side effect, never the hook path.
- unit: assert the default election backend is filelock and that the filelock
  path never consults the elector (locks the non-hook default behavior).

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): add multi-instance e2e showcase for redis primary-worker election

Scales the standard docker-compose gateway to 2 replicas with the redis election
backend and asserts exactly one primary is elected across the containers — the
cross-instance guarantee a per-host file lock cannot provide.

- docker-compose.yml: expose PRIMARY_WORKER_ELECTION_BACKEND and
  PRIMARY_WORKER_REDIS_UNAVAILABLE_POLICY as env-configurable settings,
  defaulting to filelock / fail_closed (no behavior change by default).
- plugins/primary_worker_multiinstance: a non-hook marker plugin that RPUSHes
  <host>:<pid> to a shared Redis list when primary, so the count is observable
  across containers (a per-container file marker cannot be).
- run_primary_worker_multiinstance.sh + make test-primary-worker-multiinstance:
  redis up -> clear key -> up --scale gateway=2 -> wait healthy -> assert the
  list has exactly one entry. Not part of `make test` (needs Docker + a
  branch-built image).

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* docs(plugins): tighten election comments and drop spike wording

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): add KEEP_UP flag to the multi-instance e2e for redis inspection

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): count distinct primaries via a redis set (robust to re-init)

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): cover redis elector lifecycle, heartbeat, and singleton paths

Raise diff coverage on the redis backend above the CI gate: exercise the
maintenance loop (lost-lease demotion via compare-and-renew, mid-heartbeat
error -> fail-closed), the own-client build/close path, the started/instance_id
properties, and the module-level start/get/stop singleton helpers. Brings
leader_election.py to 100% line coverage.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* fix(plugins): demote on mid-heartbeat redis error under filelock_fallback

The redis elector's maintenance loop only demoted a stale primary for the
fail_closed policy. Under filelock_fallback, a Redis error mid-heartbeat left
_is_primary=True with no election performed: the lease then expired unrenewed,
a follower acquired it via SET NX EX, and two processes reported primary -- the
split-brain this backend prevents.

Route the maintenance-loop error through _handle_redis_unavailable() for both
policies, matching initial-connect behavior: fail_closed demotes; filelock_fallback
re-elects per host (and demotes if the lock is held). Make _acquire_filelock()
idempotent (reuse one FileLock, short-circuit when already held) so the loop can
call it each heartbeat during an outage without churning the lock.

Add regression tests for filelock_fallback + mid-heartbeat error: demotes when the
file lock is held elsewhere, re-elects when it is free.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* refactor(plugins): address non-blocking review notes on the redis elector

- Dedup the port-scoped lock-path helper: reuse _lock_path() from
  mcpgateway.utils.primary_worker instead of a hand-mirrored copy (aliased on
  import, so call sites are unchanged). No circular import — primary_worker only
  imports leader_election lazily.
- Honor explicit falsy options: use 'x if x is not None else settings.y' rather
  than 'x or settings.y' for backend/redis_url/redis_key/lease_ttl/heartbeat/policy,
  so an explicit 0 or "" isn't silently replaced by the settings default.
- Make stop() unconditionally best-effort: also swallow non-CancelledError errors
  from the cancelled maintenance task (matching the redis-release block) so a task
  error can't interrupt the rest of the shutdown sequence.

Adds a shutdown-robustness test; leader_election.py stays at 100% line coverage.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* fix(plugins): make elector start()/stop() idempotent on the redis backend

start() had no guard for already-started: a second call on the redis backend
re-issued SET NX (which fails since we already hold the key) and demoted the
active primary to False, and overwrote self._task with a new maintenance loop,
orphaning the first (unreachable by stop()). This contradicts the documented
"second start reuses the same singleton" contract; the existing singleton test
only covered the idempotent filelock path, which masked it.

Guard start() with an early return when already started. Also drop the owned
Redis client in stop() (self._redis = None in a finally) so a repeat stop()
doesn't aclose() an already-closed client; injected clients are left as-is.

Add a redis-backend start()-idempotency regression test (is_primary stays True,
maintenance task not replaced) and extend the own-client test to assert a second
stop() is a no-op. leader_election.py stays at 100% coverage.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* chore(plugins): warn on bad heartbeat/lease-ttl ratio and document redis election caveats

Add a startup model-validator (redis backend only) that warns when
PRIMARY_WORKER_HEARTBEAT_INTERVAL >= PRIMARY_WORKER_LEASE_TTL/2, since the lease
would expire before renewal and cause continuous re-election. Warns rather than
raises, matching the existing config-validator style (non-breaking).

Document three redis-backend caveats in configuration.md: namespace the lease key
when sharing one Redis across deployments, keep heartbeat < lease_ttl/2, and note
that a boot-time Redis outage doesn't auto-recover (restart the worker).

Related to IBM#5139

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* fix(plugins): log exception type, not str(exc), in elector redis warnings (CWE-532)

A redis.asyncio connection error can stringify the connection target, so logging
str(exc) could leak the password from REDIS_URL/PRIMARY_WORKER_REDIS_URL into the
application log. Log type(exc).__name__ instead across all elector warnings
(redis-unavailable, maintenance error, stop cleanup, task-cancel).

Add a regression test that raises a ConnectionError with an embedded DSN mid-
heartbeat and asserts the secret never reaches the log while the exception type
still does.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test(plugins): add concurrent-start, stop-release-error, and heartbeat-boundary tests

Harden coverage per review:
- test_redis_concurrent_start_elects_one_primary: many electors start() via
  asyncio.gather, exactly one wins the lease (SET NX is atomic).
- test_stop_swallows_redis_release_error: the lease-release eval raising during
  stop() is swallowed (best-effort shutdown), no propagation.
- test_primary_worker_heartbeat_warns_at_exact_boundary: heartbeat == lease_ttl/2
  warns (the requirement is strictly less than).

leader_election.py stays at 100% coverage.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* chore: reindex .secrets.baseline after rebase onto main

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

---------

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
* docs: add generic MCP extension framework ADR

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* docs: align extension framework ADR with security governance

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

---------

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>
* fix: bind join requests to team context

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: align join request not found handling

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: cover join request error branches

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: strengthen join request review handling

Signed-off-by: lucarlig <luca.carlig@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
…check errors (IBM#5654)

* fix(services): rebuild fork-poisoned FileLock and back off on health-check errors

GatewayService's FileLock is constructed once, before gunicorn --preload
forks worker processes. Newer filelock releases detect reuse of a
pre-fork lock from a different PID and raise instead of silently
working. The health-check loop's generic exception handler logged that
error and immediately re-entered the loop with no delay, producing an
unbounded busy loop that starves the worker of the CPU it needs to
serve requests (observed as CI jobs hanging until the workflow timeout,
with the gateway log growing to millions of lines).

Track the PID the lock was built for and rebuild it when a worker
detects it's running under a different PID than its creator. Also add
the same backoff used by the Timeout branch to the generic exception
handler, so any other unexpected lock error can no longer busy-loop.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore(deps): cap filelock below the fork-unsafe 3.30.0 release

filelock 3.30.0 (published 2026-07-16T03:53:56Z) added a fork-safety
guard that raises RuntimeError when a lock object is reused from a
different PID than its creator - exactly the pattern
GatewayService._file_lock hits under gunicorn --preload, which was the
trigger for the busy-loop fixed in the previous commit.

The unbounded floor (filelock>=3.29.4) combined with `make install`
resolving live against PyPI (uv pip install ., not uv sync) meant every
fresh install silently picked up 3.30.0 the moment it was published,
regardless of what's pinned in uv.lock. Capping the floor keeps installs
on a known-good release; the previous commit's fork-safe rebuild is the
real fix and keeps this code correct even past this ceiling once
verified against future filelock releases.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore(deps): raise filelock ceiling to allow the verified-safe 3.30.0

Reproduced the 3.30.0 fork-reuse RuntimeError directly against the real
package (os.fork() + FileLock.acquire() on the inherited object) and
confirmed the PID-rebuild fix from the first commit neutralizes it -
the rebuilt lock never touches the poisoned pre-fork instance, so the
new guard never fires. Full gateway_service test suite passes under
3.30.0 as well.

<3.30.0 was overly conservative once the actual fork-safety bug is
fixed; <3.31.0 keeps a ceiling against an unverified future release
without blocking a version we've now tested against directly.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
…rom change-password-required.html (IBM#5646)

* fix(templates): remove CSP-violating airgapped Tailwind JS branch from change-password-required.html

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* added .secrets.baseline

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

---------

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
…admin (IBM#5647)

The pre-seeded Keycloak test user was configured with email
admin@example.com, identical to the default PLATFORM_ADMIN_EMAIL value
in .env.example. On SSO login, resolve_session_teams() looks up the
local user record by email and would find the built-in admin entry,
silently granting is_admin:true to the Keycloak test account.

Rename username and email: admin@example.com -> admincf@example.com in
infra/keycloak/realm-export.json and update the matching reference in
the Keycloak SSO tutorial doc.

Closes IBM#5542

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
… test endpoint (IBM#5023)

* fix: respect global ssrf_protection_enabled flag in gateway test endpoint

The /admin/gateways/test endpoint was unconditionally blocking private IPs
regardless of the global ssrf_protection_enabled configuration flag. This
created inconsistency with other endpoints and prevented testing internal
services even when SSRF protection was intentionally disabled.

Changes:
- Modified validate_gateway_test_url() to check ssrf_protection_enabled
  before applying private IP blocking (both direct and DNS-resolved IPs)
- Allowlist enforcement remains active regardless of SSRF flag (defense in depth)
- Added comprehensive test suite with 18 tests covering all scenarios
- Created bug report documenting the issue and fix

This ensures consistent behavior across all endpoints while maintaining
security through layered protections (SSRF + allowlist enforcement).

Fixes: Gateway test endpoint SSRF flag bypass
Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: address code review findings for gateway test SSRF protection

Addresses all code review findings from PR feedback:

1. Updated docstring in validate_gateway_test_url() to accurately describe
   conditional SSRF behavior based on ssrf_protection_enabled flag. Previous
   docstring claimed "unconditional blocking" which was no longer true.

2. Updated doctest examples to reflect conditional blocking behavior, adding
   examples showing private IPs are blocked when ssrf_protection_enabled=true
   and allowed when false.

3. Renamed three existing tests in test_validators_advanced.py from
   test_*_blocked_unconditionally to test_*_blocked_when_ssrf_enabled to
   accurately reflect their behavior. Added explicit ssrf_protection_enabled=True
   flag pinning so tests verify what they claim.

4. Added WARNING-level audit logging when SSRF protection is bypassed for
   forensic visibility. Logs are emitted when private/internal IPs are allowed
   through due to ssrf_protection_enabled=false, both for direct IPs and
   DNS-resolved IPs. Includes hostname, resolved IP, and IP type classification.

5. Mocked DNS resolution in three integration tests (test_wildcard_subdomain_matching,
   test_dns_rebinding_protection, test_fqdn_normalization_prevents_bypass) to
   eliminate network dependencies and improve CI stability. Removed @pytest.mark.integration
   markers as tests no longer hit real DNS.

All tests pass. No functionality changes - only documentation, test accuracy,
and observability improvements.

Related to: IBM#5022

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: resolve pylint and ruff formatting issues

- Remove redundant logger redefinition (use module-level logger)
- Fix string concatenation formatting in log messages
- Resolves pylint W0621 (redefined-outer-name)
- Resolves ruff formatting violations

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix(security): address code review findings for SSRF protection PR

Resolved three code quality issues from review:

1. DNS Mock Type Mismatch:
   - Fixed test_dns_timeout_configuration to mock socket.getaddrinfo
     as synchronous function (with time.sleep) instead of async
   - Added clarifying comment explaining getaddrinfo is sync but run
     in executor for async safety

2. Duplicate Exception Handling:
   - Consolidated exception handling comments at lines 1680-1682
     and 1761-1764 for better readability
   - Clarified when each ValueError case applies

3. Duplicate SSRF Check Logic:
   - Extracted complex boolean condition into is_dangerous variable
     at line 1734-1747 for improved readability
   - No functional changes

All tests pass.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* refactor(validators): apply DRY principle to SSRF protection checks

Move ssrf_protection_enabled condition into nested if blocks within the
main IP validation logic, eliminating duplicate code paths for IP parsing
and network checks. This reduces redundancy while maintaining the same
security behavior and logging.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* style: apply ruff formatting to is_dangerous check

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

---------

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Gabriel Costa <gabrielcg@proton.me>
gandhipratik203 and others added 27 commits September 15, 2026 15:35
Add TestUserLifecycle. The class covers the user admin CRUD surface as an
observable contract: creation fields, listing, RBAC role assignment,
duplicate rejection, and deletion. User creation and deletion previously
existed only as fixture helpers, so a regression in those endpoints failed
unrelated RBAC fixtures with an opaque error.

Add an owned-user factory rather than reusing _create_user_with_token. The
existing helper discards the creation response and tolerates outcomes these
tests assert, and it creates API tokens, which belongs to issue IBM#6523. The
factory generates the address, returns the request inputs alongside the
response, and records ownership on a successful status before reading the
body, so a failed creation never marks an existing account for deletion.

Assert role assignment on the full tuple of user, role, scope and scope
identifier, and check a control user does not hold it. Every returned
assignment must belong to the user queried, without which the control check
passes for any user.

Verify cleanup instead of assuming it. Teardown attempts every deletion,
then confirms the accounts, the role assignment and the team membership are
gone, and reports all failures together.

Read the user list with limit=0. The endpoint documents 0 as the whole set.
Its default caps at the configured page size and it never emits a cursor,
so a default read drops users with no signal. Tracked in issue IBM#6834.

Closes IBM#6520

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
* test: add cross-replica E2E consistency checks

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: restore displaced-gateway teardown scope and stale sync comparison

Move the displaced-gateway restore loop back into
streamable_http_gateway's finally block. The loop referenced
displaced_gateways from outside its defining scope, raising NameError
on every teardown and skipping the restore of the compose-seeded
gateway it displaced.

Reset previous_tool_ids on a failed sync probe so a later successful
read cannot compare against a stale, pre-gap snapshot and report
false stability.

Drop the flaky rerun marker from TestCrossReplicaConsistency. Its
fixtures are module-scoped and do not re-run, so the marker hides the
exact divergence the suite exists to catch instead of retrying
independently.

Addresses review feedback on IBM#6804.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore: regenerate secrets baseline

Refresh generated_at timestamp after rebase onto origin/main.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: catch Playwright transport errors in gateway sync loop, document replica deadline

Add playwright.sync_api.Error to the sync-probe except tuple in
streamable_http_gateway. A transient network error from Playwright's
API client previously escaped the tuple and crashed the fixture
instead of retrying, since the class descends directly from Exception
and matched none of AssertionError, KeyError, TypeError, ValueError.

Document MCP_E2E_PUBLISHER_SYNC_DEADLINE and MCP_E2E_REPLICA_SYNC_DEADLINE
in tests/live_gateway/README.md so the replica deadline's default and
rationale are discoverable outside the inline code comment.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* ci: add command-triggered conformance

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: add conformance baselines

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: complete both conformance era pairs

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: simplify conformance command workflow

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: use cf-integration 0.3.2 for conformance

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: provide strong conformance bootstrap passwords

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: run conformance with a dual-era fixture

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: baseline current gateway conformance results

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: run conformance on ready SDK v2 pull requests

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: install conformance tools without restricted actions

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: limit SDK v2 conformance to selected protocol revisions

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: run conformance on the capability discovery stack

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: refresh conformance baselines for the SDK v2 gateway

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: restrict conformance to legacy clients and fixtures

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: record fresh 2025-11-25 legacy conformance baselines

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: install the published conformance binary directly

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* ci: use cf-integration 0.4.0 for conformance

Signed-off-by: Luca <luca@lucas-mbp.home>

* fix: avoid cache export on comment conformance runs

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: run conformance automatically on main

Signed-off-by: lucarlig <luca.carlig@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: Luca <luca@lucas-mbp.home>
Co-authored-by: Luca <luca@lucas-mbp.home>
* refactor(test): extract _mint_token from _create_user_with_token

Token minting moves to its own helper so tests can mint extra tokens for an
existing user. _create_user_with_token now returns token_name as well.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): add token lifecycle fixture and revocation delay knob

MCP_E2E_REVOCATION_DELAY tunes the wait after a revoke without a code change.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): assert token creation and catalog listing

POST /tokens returns access_token and id. GET /tokens lists the token by id
and name.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): assert live auth and expiry for a minted token

The token authenticates GET /tools and an MCP session. expires_in_days=1
lands within 120 seconds of now plus 24 hours.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): assert revocation denies REST and MCP

DELETE /tokens/admin/{id} returns 204. After the propagation wait, GET /tools
returns 401 and MCP initialize raises.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): assert a tools.read token cannot execute a tool

TestMcpScopedTokenPermissions only proves such a token can initialize. This
closes the execute-denial gap.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* docs(test): document TestTokenLifecycle in the e2e README

Records the new class, its coverage, and MCP_E2E_REVOCATION_DELAY.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): harden token-lifecycle assertions from final review

Assert the target tool is registered before relying on isError to mean
"denied" rather than "not found". Capture token_id immediately after
parsing the create response so an early assertion failure still lets
cleanup revoke the token.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test(e2e): verify denial reason, not just failure, in scope/revoke tests

test_scoped_token_denied_tool_execute now checks the caught error is an
access-denial (mcpgateway.middleware.rbac._ACCESS_DENIED_MSG), not any
transport failure. test_revoke_token_denies_mcp narrows to
httpx.HTTPStatusError / ExceptionGroup and checks status 401. Both use a
new _unwrap_exception_group helper: the MCP SDK wraps the underlying error
in a variable number of nested ExceptionGroup layers depending on how many
anyio TaskGroups are open on the call stack, so a fixed-depth unwrap misses
the real error on some call shapes.

Refs IBM#6523

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* test: add team lifecycle coverage to the live E2E suite

Add TestTeamLifecycle. The class covers the team admin surface as an
observable contract: creation fields, creator ownership, listing and
detail, member addition and removal, pagination, deletion, and the
missing-team response. Team creation previously existed only as a fixture
helper, so a regression in these endpoints failed unrelated RBAC fixtures
with an opaque error.

Assert the creator holds an active owner membership. An owner is what lets
the caller manage the team afterwards, and a team created without one is
unusable through the API.

Read the member list back after adding a member. The POST response is
checked first, then the membership is confirmed independently on
user_email, team_id, role and active status, so the test does not trust
the echo alone.

Assert the creator's ownership survives a member removal. Losing it would
leave the team unmanageable and would also block teardown.

Isolate pagination with a per-run search prefix. The assertion compares the
paged union against the created set, rejects an id returned on two pages,
rejects a repeated cursor, and requires more than one page, so a gateway
that stopped paging cannot pass. The listing and cursor shapes are
disjoint: a cursor page carries no total, so no test reads one from it.

Assert deletion through the REST contract. The gateway deletes a team
softly and keeps the row, so the checks cover the listing and a 404 from
the detail read rather than any database row count.

Verify cleanup instead of assuming it. Teardown deletes in reverse creation
order, treats 404 as already deleted, collects transport errors and
unexpected statuses rather than stopping at the first, confirms the teams
are absent from active listings, and reports all failures together. A 403
is unexpected: no test drops the creator's own membership, so one signals
an authorization change and fails.

Order the two teardowns through the fixture graph. owned_teams requests
owned_users, so every team is deleted before the accounts that belong to
it.

Match team members on user_email in the user teardown. The records carry
user_email and never email, so the membership check could not fire.

Closes IBM#6522

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* test: drop a membership cleanup check the team tests cannot reach

test_add_team_member and test_remove_team_member registered their
membership with owned_users, which verifies at teardown that the
membership did not survive. That verification cannot run for these tests.

owned_teams requests owned_users to order the two teardowns, so the team
is deleted first. Deleting a team deactivates every membership on it, and
the member list answers 403 once the caller's own membership is inactive,
so the check's "status == 200" guard is never true and the assertion
silently passes.

Both tests already assert the membership directly: one confirms the
member appears with the expected user_email, team_id, role and active
status, the other confirms the member is gone and the creator still owns
the team.

The check stays valid for TestUserLifecycle, which registers against the
module-scoped RBAC team. A module-scoped team outlives every
function-scoped teardown, so the member list is still readable when the
check runs.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

---------

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Two catalog entries used non-canonical auth_type values that do not match
the documented vocabulary (Open, OAuth2.1, API Key):

  - customgpt:  "API"   → "API Key"   (line 945)
  - instant:    "OAuth" → "OAuth2.1"  (line 209)

catalog_service.py already treats "API" and "API Key" identically at
registration (line 519), confirming these are the same type. The raw
values are surfaced directly in the /v1/catalog auth_types facet, the
legacy admin dropdown, and the new UI catalog filter, causing duplicate
filter options to appear.

This change has one intentional user-visible effect: requests filtering
on ?auth_type=API will no longer match the customgpt entry; callers
should use ?auth_type=API Key instead (matching the documented spec).

Fixes IBM#6793

Signed-off-by: Ommprakash Mohanty <ommmohanty419@gmail.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
IBM#6846)

* feat(plugins): migrate output length guard to Rust-backed CPEX package

Replace the Python output_length_guard plugin implementation with the
published cpex-output-length-guard package (>=0.1.0).

- Remove the local plugins/output_length_guard/ directory.
- Update plugin kind references in charts and YAML configuration files.
- Add cpex-output-length-guard to pyproject.toml and uv.lock.
- Allowlist output_length_guard metric field names in mcpgateway/plugins/utils.py.
- Add integration and end-to-end test suites.
- Update plugin documentation tables.

Closes IBM#6790

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* move imports to third party

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* added live gateway tests

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* fix tests

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* Fix tests

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* add tests

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

* add more plugin tests

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>

---------

Signed-off-by: prakhar-singh1928 <prakhar.singh1928@ibm.com>
…unds (IBM#6852)

* fix(catalog): normalize icon visual weight and strip pale badge surrounds

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix: address comments

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* chore: regenerate .secrets.baseline for 6506-fix-icons-size

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix(catalog): close off-domain redirect gap and pale-backdrop floor bug

Guard the favicon.ico/DuckDuckGo candidate loop in _fetch_icon against
off-domain redirects, matching the existing origin-page guard.

Compare the raw perimeter color against PALE_BACKDROP_FLOOR instead of
the quantized bucket in _strip_pale_backdrop. Quantizing first shifted
the effective floor from 225 to 232.

Collapse _has_backdrop_stripped_marker, _has_scale_boost_marker, and
_has_scale_shrunk_marker into one _has_marker() helper.

Add regression tests for both fixes.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* chore: regenerate .secrets.baseline for 6506-fix-icons-size

Line numbers for two pre-existing, audited findings in
scripts/fetch_catalog_icons.py shifted after the off-domain redirect
guard and pale-backdrop floor fix. No new or live secrets found.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix(catalog): skip-list metro-mcp and dedupe marker/redirect helpers

  Add metro-mcp to the skip list in catalog_icon_overrides.json so
  --force cannot silently replace it from its unstable personal-site
  favicon, and document the exception in catalog.md.

  Collapse the three copy-pasted PNG marker writes in _image_to_png and
  the three needs_* checks in generate_icons into loops over (flag,
  marker) pairs.

  Extract _landed_off_domain() to share the off-domain redirect check
  between the origin-page fetch and the candidate loop in _fetch_icon.

  Add regression tests for the PALE_BACKDROP_MIN_PERIMETER_SHARE guard
  against a fragmented perimeter and for the UnicodeDecodeError branch
  in _rasterize_svg.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* chore: regenerate .secrets.baseline for 6506-fix-icons-size

Line numbers for two pre-existing, audited findings in
scripts/fetch_catalog_icons.py shifted after collapsing the marker
write/check duplication and extracting _landed_off_domain(). No new
or live secrets found.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

---------

Signed-off-by: Marek Dano <mk.dano@gmail.com>
* test: add gateway registration and tool sync E2E coverage

Add TestGatewayLifecycle to tests/live_gateway/e2e/test_e2e.py: register
a gateway against fast_time_server, verify tool sync and gatewayId
linkage, then verify deletion removes synced tools from the catalog.

The fixture registers with visibility=private instead of the issue's
public example. The module already holds fast_time_server's URL
publicly for the whole suite via streamable_http_gateway, and gateway
uniqueness is visibility-scoped, so a second public registration at
that URL returns 409. Private registration avoids the collision while
exercising the same sync and delete paths.

GET /tools calls pass gateway_id and limit=0 so the server filters
tool results; an unpaginated GET /tools truncates at
pagination_default_page_size and can hide synced tools on a busy
catalog.

Closes IBM#6521

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore: regenerate secrets baseline

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: assert ephemeral_gateway registration status, document ordering risk

ephemeral_gateway posted and yielded the raw response without checking
status. A non-2xx registration surfaced downstream as KeyError: 'id'
instead of a clear failure. Assert the status in the fixture.

Document the residual risk from streamable_http_gateway's displacement
scan: it matches by name or url regardless of visibility and restores
without a visibility field, so it comes back public. That scan runs
once, before TestGatewayLifecycle in file order, so a private row from
this fixture is only at risk if the class order changes or a delete
below fails silently.

Addresses review feedback on PR IBM#6848.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* chore: regenerate secrets baseline

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: address gateway lifecycle E2E review feedback

Fixes ephemeral_gateway's declared return type to match the
APIResponse it actually yields, collapses _gateway_tool_names into
a thin wrapper over _gateway_tools instead of duplicating the fetch,
documents MCP_E2E_GATEWAY_SYNC_DEADLINE in the module docstring, and
reuses _GATEWAY_UPSTREAM_URL in streamable_http_gateway instead of
a second hard-coded copy of the URL.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
…BM#6004)

* Fixed issue IBM#5980

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Fixed non-dict payload handling

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* ci: rerun failed lint workflow

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* fixed edge cases

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Address review feedback

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Enhanced docstring

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Address review feedback

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Fixed malformed cached JWT teams

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* run pre-commit

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* Fix broken symlinks

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Sagar-Mondal <sagarandshiva@gmail.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
* fix: persist Rust control denial telemetry

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: retain plugin denial details in OpenTelemetry spans

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

---------

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>
Signed-off-by: nightcityblade <jackchen@haloailabs.com>
* test: assert the denial reason in test_outsider_denied_tools_execute

The assert sat inside a try whose except caught Exception. AssertionError is
an Exception, so the test passed under every gateway behaviour. Move the
assert outside the try. Narrow the except to the transport errors the MCP SDK
raises. Assert the denial reason on both the returned-result path and the
raised path, so an unrelated error cannot read as a confirmed denial.

Refs IBM#6839

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: assert the error reason in test_outsider_calls_nonexistent_tool_error

Both branches passed unconditionally. The assert sat inside a try whose
except caught Exception, and AssertionError is an Exception. Move the assert
outside the try. Narrow the except to the transport errors the MCP SDK
raises. Assert the denial reason, because isError alone stays true for a
plain name-resolution failure and cannot detect an RBAC regression.

Closes IBM#6839

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: narrow swallowing except in translate dynamic-env E2E tests

Three tests put a real assert inside a try whose except caught Exception
broadly. AssertionError is an Exception, so a real assertion failure
became a skip instead of a failure. Narrow the except to httpx.HTTPError,
the base of httpx's transport-error hierarchy. A real connection or
stream failure still produces a skip. An assertion failure now propagates
as a test failure.

Closes IBM#6873

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: narrow RBAC denial status check to 403 only

_DENIED_STATUSES accepted 401 alongside 403 as evidence of a confirmed
RBAC denial. The outsider fixture holds a valid token, so a 401 means
authentication failed before RBAC ran, not that RBAC denied it. Every
RBAC denial in mcpgateway/middleware/rbac.py raises 403 with
_ACCESS_DENIED_MSG; none raise 401. Accepting 401 let a broken-auth
regression masquerade as a confirmed RBAC denial.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: extract the RBAC denial assertion into a shared helper

Both outsider tests repeated the same transport and result assertions. Move
them into `_assert_denied_for_rbac()`, which takes the call and a label.

Replace the `if result is not None` guard with `else`. `_mcp_tool_call()`
either returns a result or raises, so the guard was misleading. The `else`
body is not covered by the `try`, so it still cannot swallow an
`AssertionError`.

Hoist `_TRANSPORT_ERRORS` and `_DENIED_STATUSES` to module level.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(security): require explicit allowlist for CORS origin reflection

SecurityHeadersMiddleware reflected any request Origin when
ALLOWED_ORIGINS was empty and ENVIRONMENT was not production. It also
sent Access-Control-Allow-Credentials: true. A malicious site could
read credentialed responses from an exposed development or staging
gateway.

Reflect an Origin only when it appears in ALLOWED_ORIGINS, in every
environment. An empty allowlist now blocks all origins.

Replace the development-mode test that asserted the old behavior. Add
deny-path tests for an empty allowlist in each environment and for an
unlisted origin. Add allow-path tests for listed origins in
development and staging.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* chore: update secrets baseline for CHANGELOG CORS entry

Audit the 'Access-Control-Allow-Credentials: true' string in the new
CHANGELOG.md entry as a false positive (Secret Keyword) and refresh
line numbers shifted by the entry.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* test: cover CORS with no Origin header

  Add test_cors_no_origin_header_no_cors_headers, parametrized across
  development, staging, and production. Confirm a request without an
  Origin header gets no Access-Control-Allow-Origin,
  Access-Control-Allow-Credentials, or Access-Control-Expose-Headers,
  even when ALLOWED_ORIGINS is populated.

  Addresses review feedback on PR IBM#6941

Signed-off-by: Marek Dano <mk.dano@gmail.com>

---------

Signed-off-by: Marek Dano <mk.dano@gmail.com>
…P server/virtual server) (IBM#6770)

* docs: document API-to-product vocabulary mapping (gateway/server → MCP server/virtual server)

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* docs: address PR IBM#6770 review comments on terminology.md

Signed-off-by: Marek Dano <mk.dano@gmail.com>

---------

Signed-off-by: Marek Dano <mk.dano@gmail.com>
* Dual-era gateway: MCP SDK 2.0 migration + 1.x/2026-07-28 interop (IBM#6485)

* chore(deps): drop fastmcp dev dependency and protocol-compliance make targets

Signed-off-by: Jonathan Springer <jps@s390x.com>

* test(live-gateway): rewrite MCP protocol E2E on the official mcp SDK

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(tests): remove the compliance reference server and matrix tooling

The fastmcp-based protocol-compliance harness was the only consumer of the
compliance reference server and the compliance matrix generator, so both go
with it.

- delete mcp-servers/python/compliance_reference_server/ and
  scripts/compliance_matrix.py
- drop the compliance-reference-server dev dependency and its
  [tool.uv.sources] editable entry; regenerate uv.lock (also sheds fastmcp,
  fastmcp-slim, authlib and their transitives)
- drop the compliance-only pytest markers from pyproject.toml
- refresh the pip-audit skip-list note and the PYTEST_IGNORE comment

Signed-off-by: Jonathan Springer <jps@s390x.com>

* docs: drop protocol-compliance references and name the mcp SDK client

Follow-up to the harness removal: the PR-review validation gate, the
live-gateway README/target list, and the test docs still pointed at
`make test-protocol-compliance` and described the protocol E2E suites as
FastMCP-driven.

- AGENTS.md, tests/AGENTS.md, llms/NOTES.template.md,
  docs/docs/development/release-management.md,
  tests/live_gateway/README.md: retire the removed targets, name the
  official mcp SDK client
- tests/live_gateway/helpers/mcp_test_helpers.py: correct the module
  docstring's consumer list and point new tests at the SDK ClientSession
  pattern

Signed-off-by: Jonathan Springer <jps@s390x.com>

* test(sso): move Entra ID integration test to tests/integration

Ports the relocation from mcp-2-0-0-prerelease (d7c1f9a): the Entra ID
suite is fully in-process (httpx.ASGITransport) and self-skips when the
AZURE_* credentials are absent, so it belongs with the integration tests
that run under the default 'make test' rather than the live-gateway tree.

- tests/live_gateway/sso/test_entra_id_integration.py ->
  tests/integration/test_entra_id_integration.py (header updated)
- tests/live_gateway/sso/__init__.py: docstring now covers only the
  Keycloak suite and points at the new Entra ID location
- docs/docs/testing/entra-id-e2e.md: all pytest invocation paths updated
- Makefile: test-e2e-sso targets Keycloak only, notes the Entra ID move
  (help comment aligned as well)

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(deps): remove fastmcp and rebuild live-gateway E2E on the official mcp SDK

Drops the fastmcp dev dependency and everything that depended on it,
replacing the live-gateway protocol and RBAC E2E suites with a client
built on the official mcp Python SDK.

- drop fastmcp dev dependency and protocol-compliance make targets
- remove the fastmcp-based protocol compliance harness and the
  fast-server go/rust parity test
- rewrite the MCP protocol and RBAC transport E2E suites on the
  official mcp SDK ClientSession
- remove the compliance reference server and compliance matrix tooling
  (the harness was their only consumer); regenerate uv.lock, which also
  sheds fastmcp-slim, authlib and their transitives
- update AGENTS.md, test docs, and the live-gateway README to retire
  the removed targets and name the mcp SDK client

Closes IBM#5910

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(deps): drop fastmcp dev dependency and protocol-compliance make targets

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(tests): remove the compliance reference server and matrix tooling

The fastmcp-based protocol-compliance harness was the only consumer of the
compliance reference server and the compliance matrix generator, so both go
with it.

- delete mcp-servers/python/compliance_reference_server/ and
  scripts/compliance_matrix.py
- drop the compliance-reference-server dev dependency and its
  [tool.uv.sources] editable entry; regenerate uv.lock (also sheds fastmcp,
  fastmcp-slim, authlib and their transitives)
- drop the compliance-only pytest markers from pyproject.toml
- refresh the pip-audit skip-list note and the PYTEST_IGNORE comment

Signed-off-by: Jonathan Springer <jps@s390x.com>

* docs: drop protocol-compliance references and name the mcp SDK client

Follow-up to the harness removal: the PR-review validation gate, the
live-gateway README/target list, and the test docs still pointed at
`make test-protocol-compliance` and described the protocol E2E suites as
FastMCP-driven.

- AGENTS.md, tests/AGENTS.md, llms/NOTES.template.md,
  docs/docs/development/release-management.md,
  tests/live_gateway/README.md: retire the removed targets, name the
  official mcp SDK client
- tests/live_gateway/helpers/mcp_test_helpers.py: correct the module
  docstring's consumer list and point new tests at the SDK ClientSession
  pattern

Signed-off-by: Jonathan Springer <jps@s390x.com>

* feat: migrate to MCP SDK 2.0.0b2 with modern 2026-07-28 upstream negotiation

Squashed arc:

- MCP SDK migration 2.0.0a2 -> 2.0.0b1 -> 2.0.0b2 (Python 3.14 support added; floor stays >=3.11 — mcp 2.0.0b2 needs only >=3.10, cpex >=3.11, CI is py3.12, MCPError rename, transport/Client compatibility fixes)
- Upstream federation path migrated from raw mcp.ClientSession to mcp.client.Client: mode=auto probes server/discover and adopts modern 2026-07-28, with transparent legacy fallback; MCP_CLIENT_CONNECT_MODE=legacy rollback flag
- Pooled upstream session registry on the high-level Client with typed SessionLifecycle handle (replaces _cf_* attribute smuggling), cache=False to preserve ADR-052 notification wiring, dispatcher-aware transport health check
- mcp_proxy_client gained transport= (streamablehttp/sse) and mode= options; all five ad-hoc SSE fallback sites routed through it
- Compose test stack: single cfex-mcp-fast-time-server image (stale Go-era flags fixed), strict modern-only 2026-07-28 fast_time_2026 variant added, five register_* one-shots consolidated into a single 'register' container running scripts/register_mcp_servers.py 'stack'
- Tests: SDK invariant spike suite, config validators, reworked registry/service suites, E2E negotiation suite (11 tests) proving auto/legacy modes against strict and legacy upstreams
- Docs: MCP_CLIENT_CONNECT_MODE and legacy rollback documented
- Fixes: four-parameter httpx2.Timeout in proxy-client default path, token-exchange integration test isolation from shared Redis, pre-existing get_httpx_client_factory dedent repair in gateway_service

Packaging — the SDK pin lands in core dependencies, not in extras:

- An earlier revision of this branch split the `mcp` pin into mutually
  exclusive `runtime` / `live-tests` extras backed by a `[tool.uv] conflicts`
  block, so the gateway could stay on `mcp` 2.x while the fastmcp-based
  protocol-compliance harness stayed on `mcp` < 2.0. That harness is gone, so
  the partition is removed: `mcp==2.0.0b2` and `mcp-types==2.0.0b2` sit in core
  `dependencies` alongside `cpex`, and `pip install mcp-contextforge-gateway`
  once again resolves a working install.
- Every `--extra runtime` / `--extra live-tests` flag is dropped from the Make
  targets, the pytest/pytest-rust/wrapper workflows and both Containerfiles,
  restoring the install targets to their pre-extras form. The `granian` extra
  is dropped from the Containerfile install lists as well; no such extra has
  ever been defined.
- All install documentation, the four affected ADRs and the agent-facing
  dependency-layout guidance revert to their extras-free wording.
- Fixes a duplicate `cpex-secrets-detection` key in
  `[tool.uv.exclude-newer-package]` that left pyproject.toml invalid TOML.

Also realigns two upstream session-factory tests that were left on the
pre-migration SDK shape: they patched `streamablehttp_client` / `ClientSession`
and reached for the retired `_cf_shutdown_event` / `_cf_owner_task` attributes,
so they errored during setup instead of exercising the post-ready WARNING path
and the timeout structured-logging fallback.

Signed-off-by: Jonathan Springer <jps@s390x.com>

* feat: upgrade MCP SDK from 2.0.0b2 to 2.0.0 final

The 2.0.0 final release removes mcp.shared.session entirely (the b2
typing-only RequestResponder stub is gone) and changes Client(cache=)
from a bool flag to CacheConfig | None, where None disables the response
cache so message_handler passes through unwrapped.

- pyproject/uv.lock: mcp/mcp-types >=2.0.0, cpex tracked from the
  0.1.x-mcp-2.0.0 branch, drop stale exclude-newer pins
- mcpgateway/utils/session_compat.py: new shim re-creating the
  RequestResponder typing stub so existing annotations and isinstance
  guards keep working (never instantiated, matching b2 semantics)
- notification_service/upstream_session_registry: import from the shim;
  pass cache=None (was cache=False) in the pooled upstream session
  factory so the response cache does not wrap message_handler
- tests: repoint RequestResponder imports to the shim, assert
  cache is None, update gateway refresh test to the mcp_proxy_client
  seam, refresh upstream session error category expectations, regenerate
  .secrets.baseline

Signed-off-by: Jonathan Springer <jps@s390x.com>

* fix(services): align MCP httpx client factories on httpx2 types

Signed-off-by: Jonathan Springer <jps@s390x.com>

* test(live-gateway): migrate MCP e2e suites to SDK 2.0 final client API

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(compose): restore per-service REST auto-registration, drop strict 2026-07-28 probe

Signed-off-by: Jonathan Springer <jps@s390x.com>

* chore(testing): drop stale Fast Time 2026 strict-probe references

The fast_time_2026_server service was removed from docker-compose.yml in
5e54707, but its advertising and test wiring lingered:

- scripts/register_mcp_servers.py: delete (unused anywhere in the repo)
- Makefile: drop the :8887 banner and auto-registration echo from testing-up
- tests/e2e/test_upstream_connect_mode_e2e.py: strip the strict-upstream
  behavior matrix, wire-capture proofs, and strict registration tests;
  keep the legacy connect-mode proofs against fast_time_server (:8888)

Signed-off-by: Jonathan Springer <jps@s390x.com>

* Restore Containerfile and Makefile to main state

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix(mcp-2.0): address SDK 2.0 review findings and live-stack build issues

Review findings (PR IBM#5493):
- transports/streamablehttp: ctx.meta is a TypedDict (RequestParamsMeta) in
  MCP 2.0 with no model_dump(); extract via dict() with a legacy-model
  fallback at the call_tool/get_prompt/read_resource sites
- transports/streamablehttp: complete() now returns types.CompleteResult
  wrapping types.Completion so the wire shape carries the required
  'completion' nesting; update test assertions to the corrected contract
- utils/streamable_http_compat: build httpx2.AsyncClient (not v0 httpx) in
  the no-factory fallback, matching what the MCP 2.0 SDK transports expect;
  same-class fixes in resource_service factory and the upstream session
  registry (HttpxClientFactory alias + httpx/httpx2 exception matching)
- services/http_client_service: add get_httpx2_timeout(); the v0
  httpx.Timeout from get_http_timeout() is rejected by httpx2.AsyncClient,
  so point the gateway/tool/resource factory fallbacks at the new helper
- transports/streamablehttp: capture server-emitted session IDs whenever
  stateful sessions are enabled, not only when multi-worker affinity is on,
  so the logical-owner claim fires on single-node deployments; unskip the
  affinity-disabled regression test

Live stack:
- Containerfile: add git to the builder stage (uv needs it to fetch the
  git-pinned cpex dependency) and split the monolithic venv RUN into
  discrete steps so build failures name the exact stage
- docker-compose: repin cfex-mcp-fast-time-server to the current digest;
  the previous pin predates the 2026-07-28 caching directives and its
  tools/list responses failed ListToolsResult validation during federation

Also rewrite test_invoke_tool_mcp_refuses_stored_env_filter to the v2
mcp_proxy_client mock pattern (pre-existing stale migration leftover).

Signed-off-by: Jonathan Springer <jps@s390x.com>

* fix: direct_proxy resource reads — SSE transport, final content shape, blob decoding

Three defects left direct_proxy federated resource reads broken or
silently corrupting content:

1. mcp_proxy_client was called without a transport argument, so SSE
   gateways were dialed with streamable-HTTP framing and the connect
   failed. Select the transport from the gateway's registered type.

2. The proxied read already returns final content, but execution fell
   through into the cache-mode pointer-resolution machinery, which
   assumes a DB-shaped object and crashed on content.id (AttributeError,
   masked to "" by the ingress). Build the final ResourceContent (with
   id and mime_type) in the branch and skip the machinery via a flag
   initialized at function scope alongside the other locals.

3. MCP transports carry blobs as base64 strings while
   ResourceContent.blob holds raw bytes; passing the string through
   meant the ingress re-encoded it, double-encoding every binary
   resource. Decode to raw bytes at the seam.

Tests: TestReadResourceDirectProxy updated to assert the final content
shape (ResourceContent) — the previously asserted intermediate shape is
exactly what crashed the pointer-resolution machinery in production —
and the blob test now uses valid base64 and asserts decoded raw bytes.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: let 2026-era requests through the session-less peeks so the ingress can serve modern sessions

The streamable ingress wrapper peeks at session-less POSTs to give
unknown-method probes a truthful -32601 (instead of the SDK's
'Missing session ID'). But 2026-era requests are self-describing and
session-less BY DESIGN, and server/discover is not in the gateway's
known-method list — so every modern client's discover probe was
answered -32601 here, before the SDK's era routing ever ran, and the
client silently fell back to the legacy handshake. The gateway could
consume modern upstream sessions but never serve one.

Mirror the SDK's own era-dispatch rule (MCP-Protocol-Version header
present and not a handshake version => modern) and skip both
session-less peeks for such requests: the SDK's modern classifier owns
their validation and rejections. Also add server/discover to the
known-method list.

With this, a v2 client negotiates 2026-07-28 through the gateway
(tools, prompts, and resources verified against both a 1.x-era SSE
downstream and a 2026-era streamable downstream); legacy clients are
unaffected (no header / handshake header keeps the old peek behavior,
pinned by test).

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* refactor(tests): group era-aware peek-bypass tests under TestModernEraPeekBypass

Groups the three behaviors pinning the modern-era peek bypass (modern
header passes through, handshake header still peeked, server/discover
is a known method) under one class with a shared docstring telling the
era-boundary story — easier to locate and read as a unit in this large
test module. No behavior change.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* rename era peek-bypass group to TestDualEra

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* feat: standing listen streams for modern servers feeding debounced refresh

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* refactor: module-level listener imports, tighter notify_list_changed docstring

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: prune stale catalog rows created via notification-driven refresh

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: answer unknown tool/prompt/resource requests in proper error shapes for both eras

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* feat: relay upstream tool progress to downstream callers

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: surface tool invocation failures as isError results for both eras

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* feat: pass through 2026 MRTR elicitation for modern clients on modern servers

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* port: move IBM#5934 MCP handshake probe onto the SDK 2.0 client

The Test Connection endpoint's initialize-path probe was written against
the SDK 1.x client API. Port it to SDK 2.0: _SniPinningTransport is now
built on httpx2 (the stack the v2 client transports run on) with
identical pinning logic, the local pinned-client factory returns
httpx2.AsyncClient, _probe_session reads the v2 result field names, and
the probe connects via the v2 sse_client / streamable_http_client. The
raw server/discover probe path is unchanged. Tests updated to the v2
shapes; every pinning assertion (sni_hostname, pinned_host, TLS verify
context) still asserts.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

---------

Signed-off-by: Jonathan Springer <jps@s390x.com>
Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>
Co-authored-by: Jonathan Springer <jps@s390x.com>

* ci: run PR checks on pull requests targeting chore/mcp-sdk-v2 (IBM#6715)

* ci: run PR checks on pull requests targeting chore/mcp-sdk-v2

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: relay plugin-blocked tool calls as isError results on the /mcp path

A PluginViolationError raised on the SDK-served /servers/{id}/mcp path fell
through to the generic handler and was wrapped as a bare code-0 JSON-RPC
error. Catch it in call_tool and return an isError CallToolResult, matching
the contract the plugin e2e suites assert.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: port inherited unit tests to SDK v2 call shapes

The tests still targeted the SDK 1.x API: the streamablehttp_client patch
target and 3-tuple transport, camelCase result fields, session.call_tool
mocks where v2 calls client.session.call_tool, and call_tool assertions
without the v2 keyword arguments. Header assertions now inspect the
http_client passed to the SDK instead of a headers kwarg.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: satisfy pylint and pre-commit on migration files

Disable no-member on the mcp_app.request_context reads (the property is
injected at runtime for the v1 handler adapters) and try-except-raise on
the listener's CancelledError guard; strip a trailing space flagged by
pre-commit.

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: satisfy pre-commit and diff coverage for the plugin-block change

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: remove unwanted commentsn from tests

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

---------

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* feat(compose): add 'ui' profile for contextforge-web-ui and document release verification (IBM#6716)

* feat(compose): add 'ui' profile for contextforge-web-ui and document release verification

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix: lint issue

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* docs: fix forbid-source-placeholders hook failure in ui.md

Replace '(source: ...)' pattern on line 130 with an inline link so the
pre-commit forbid-source-placeholders hook no longer rejects the file.
Also refresh .secrets.baseline line numbers after the one-line change.

Closes IBM#6716

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* chore: refresh .secrets.baseline line numbers after rebase onto chore/mcp-sdk-v2

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix(compose): pin web_ui image to a released version and clarify BFF UI docs

Address PR review feedback: pin WEB_UI_IMAGE to a specific released
tag instead of `latest`, add a release-checklist step to bump and
verify that version, and clarify that the built-in UI and the BFF-style
contextforge-web-ui are both supported side by side.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix(compose): pin web_ui image by digest and add ui profile config check

Pin WEB_UI_IMAGE to tag@digest (docker-compose.yml, .env.example) so the
supported ui profile's session/auth-handling BFF service can't be silently
swapped by a retargeted registry tag. Document the resolve-and-bump-digest
step in release-management.md IBM#6.4, add `make compose-ui-config-check`
(docker compose --profile ui config --quiet) plus a matching CI workflow
to catch profile/schema regressions on PRs, and drop the duplicated
WEB_UI_IMAGE version literal from AGENTS.md so there's one source of truth.

Addresses review feedback from vishu-bh and gcgoncalves on IBM#6716.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix: refresh .secrets.baseline line numbers for docker-compose.yml

The baseline committed in ef347b9 was generated before the last edit
to docker-compose.yml's web_ui image comment block, leaving it 5 lines
stale and failing detect-secrets in CI (Pre-commit Checks, run
34468285949) even though it passed locally against the pre-edit state.

Signed-off-by: Marek Dano <mk.dano@gmail.com>

---------

Signed-off-by: Marek Dano <mk.dano@gmail.com>

* fix: read the remote span context through get_current_span in the request middleware

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* feat: enforce legacy-only MCP protocol negotiation for inbound and outbound paths

Add MCP_INBOUND_PROTOCOL_MODE config to gate inbound client protocol
versions at the middleware layer. When set to 'legacy', the gateway
rejects 2026-07-28 with 400 and supported handshake versions, steering
dual-era clients to retry with initialize.

Change MCP_CLIENT_CONNECT_MODE default from 'auto' to 'legacy' so
outbound connections use initialize-only by default. Gate the
server/discover probe in test_gateway_handshake() behind the connect
mode, and add a startup guard that force-disables modern_listener_service
when legacy mode is active.

Closes IBM#6767

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* docs: clarify inbound/outbound MCP client vs server roles in configuration reference

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fixup: make pre-commit

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fixup: .env.example legacy default

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: update stale tests and add coverage for legacy-mode guards

- Fix test_default_protocol_version_applied to pin mode to auto (legacy
  case already covered by dedicated test)
- Fix doctest in protocol_version.py to compare against
  LATEST_HANDSHAKE_VERSION matching the new legacy default
- Add TestGatewayHandshakeLegacyDiscoverSkip verifying discover probe
  is skipped in legacy mode and attempted in auto mode
- Add test_lifespan_legacy_mode_skips_modern_listeners verifying
  init_modern_listener_service is not called in legacy connect mode

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix(tests): add configure_handshake_env fixture for discover-path handshake tests

mcp_client_connect_mode defaults to "legacy", which skips the server/discover
probe. Tests mocking ResilientHttpClient for discover-path behaviour were
timing out (30 s health_check_timeout) while the SDK attempted initialize
against a real host.

- Add configure_handshake_env fixture (configure_allowlist + auto mode) so all
  discover-path handshake tests exercise the probe without repeating the pair
- Revert 9 connectivity tests to configure_allowlist-only; mcp_client_connect_mode
  has no effect on check_mcp_server_connectivity
- Add test_handshake_legacy_mode_skips_discover_uses_initialize: acceptance
  criterion for IBM#6767 — legacy mode bypasses the probe and reports
  negotiation_path='initialize' with no ResilientHttpClient calls

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fixup: make pre-commit and tests

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: pin the conformance suite to cf-integration v0.5.1

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* ci: align the pytest-rust diff-cover gate with pytest.yml at 90%

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: unpin mcp and mcp-types

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: surface upstream HTTP status when the mcp 2.x handshake fails

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* fix: finish the SDK 2 port of the live E2E schema

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

* ci: stop running PR checks on pull requests targeting chore/mcp-sdk-v2

Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>

---------

Signed-off-by: Jonathan Springer <jps@s390x.com>
Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com>
Signed-off-by: Marek Dano <mk.dano@gmail.com>
Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>
Co-authored-by: Jonathan Springer <jps@s390x.com>
Co-authored-by: Marek Dano <marekdano@users.noreply.github.com>
Co-authored-by: Lang-Akshay <akshay.shinde26@ibm.com>
…ests (IBM#6797)

* fix(tests): allow real DNS passthrough for external IdP integration tests

The session-scoped _deterministic_dns stub in tests/conftest.py resolves
every non-localhost hostname to a hardcoded example.com address, which
blackholes all Microsoft Graph / Entra ID egress from the in-process
integration suite (tests/integration/test_entra_id_integration.py):
every network-dependent test fails with httpx.ConnectTimeout, so the
suite cannot pass under tests/ as shipped.

- Add opt-in TESTS_DNS_PASSTHROUGH_HOSTS: comma-separated hosts resolved
  via real DNS while the stub stays active. Hermetic default unchanged.
- Normalize IDNA-encoded bytes hostnames before matching: httpx/anyio
  resolve through anyio.getaddrinfo, which encodes hostnames to bytes;
  the previous string-only comparison never matched passthrough (or
  localhost) entries on that path.
- Document TESTS_DNS_PASSTHROUGH_HOSTS and the required
  --with-integration flag in docs/docs/testing/entra-id-e2e.md.

Verified with a live tenant: the Entra ID E2E suite passes 15/15 with
the variables set; without them, DNS remains stubbed and the suite
self-skips as before.

Signed-off-by: Jonathan Springer <jps@s390x.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Jonathan Springer <jps@s390x.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
* fix(ui): handle object types in anyOf/oneOf schemas

Admin UI form generator now correctly detects object types wrapped in
anyOf/oneOf JSON Schema constructs. Previously, these were treated as
plain text fields, causing dict parameters to be sent as strings and
triggering validation errors.

Updated both form generation and submission logic in tools.js to check
for object types nested within anyOf/oneOf arrays.

Fixes issue where MCP tools with nullable object parameters (using
anyOf with object and null types) failed when invoked from Admin UI.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix(admin-ui): fix anyOf/oneOf object detection in validate modal and submit paths

Closes IBM#6420

- Extract isObjectSchemaType(prop) and allowsNullSchemaType(prop) helpers
  to eliminate duplicated 3-line detection expressions across call sites.

- Fix validate-modal form generation (was only checking prop.type === 'text',
  rendering a single-line input for object/anyOf/oneOf params instead of a
  4-row textarea).

- Fix runToolValidation submit path (was still using prop.type === 'object',
  missing anyOf/oneOf unions entirely).

- Fix null-arm regression: parsed === null is only rejected when the schema
  has no null arm (allowsNullSchemaType). Submitting 'null' for a schema with
  anyOf: [{type:'object'},{type:'null'}] no longer throws 'Value must be an
  object'.

- Fix mixed-union regression: for anyOf/oneOf unions that include non-object
  arms (e.g. anyOf: [{type:'object'},{type:'string'}]), a SyntaxError from
  JSON.parse now falls back to passing the value as a plain scalar instead of
  aborting the submission.

- Add unit tests covering all four paths in both runToolTest and
  runToolValidation: valid JSON object, null-arm allowed, mixed-union
  plain-string fallback, and invalid-JSON error on pure object schema.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix(admin-ui): correct anyOf/oneOf object field parsing in runToolValidation and runToolTest

Extract shared parseObjectFieldValue() helper to replace duplicated ~20-line
try/catch blocks at the two call sites.

Fix 1 (blocking): For anyOf:[{type:"object"},{type:"string"}], valid JSON
scalars like "42" or "true" parse without SyntaxError, then hit the
typeof !== "object" guard and throw "Value must be an object", blocking
a previously valid submission. The helper now checks hasScalarUnionArm()
before rejecting a successfully parsed scalar, and returns the raw string
when the union has a matching scalar arm.

Fix 2 (functionally-impacting): For anyOf:[{type:"object"},{type:"null"}],
invalid JSON (e.g. {bad) throws SyntaxError; the old guard
prop.type !== "object" is always true for union props (prop.type is
undefined), so the raw invalid string passed through silently. The helper
now gates the SyntaxError fallback on hasScalarUnionArm(), which returns
false for object/null-only unions, so invalid JSON is correctly rejected
with a user-visible error.

Closes IBM#6420

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* tests: update assertions in test/unit/js/tools.test.js to align with the centralized error message formating from parseObjectFieldValue()

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

---------

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
…and single-flight (IBM#6933)

* fix: use shared HTTP client and add TTL cache for OpenAPI spec fetches

Replace per-request get_isolated_http_client with the shared singleton
get_http_client() in fetch_openapi_spec. The shared client already sets
follow_redirects=False, so SSRF safety is preserved. Per-request timeout
is passed via client.stream(..., timeout=timeout).

Add a 60s in-process TTL cache keyed on spec_url to collapse concurrent
fetches of the same OpenAPI spec to one upstream call per TTL window.

Closes IBM#6835

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: harden OpenAPI spec fetch with DNS-pinning, bounded cache, cookie isolation, and single-flight

- Replace validate_url with validate_url_for_connection_pinning + pin_url_to_resolved_ip
  to close DNS-rebinding SSRF window (CWE-918)
- Bound spec cache to 64 entries with LRU eviction and active TTL expiry (CWE-400)
- Use ephemeral httpx.AsyncClient per fetch instead of shared singleton to prevent
  cross-host cookie leakage (CWE-565/CWE-359)
- Add per-URL asyncio.Lock for single-flight coordination so concurrent cache misses
  share one upstream fetch (CWE-400)
- Update tests to match new isolated-client and connection-pinning patterns

Closes IBM#6835

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: return deepcopy from spec cache and add TTL/concurrency tests

- Return copy.deepcopy() from _spec_cache hits to prevent callers from
  mutating the shared cached dict.
- Add test_cache_ttl_expiry_triggers_refetch covering the expiry branch.
- Add test_concurrent_fetches_coalesce_to_single_request proving the
  single-flight lock produces exactly one upstream fetch.

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: release single-flight lock on failed OpenAPI spec fetch

A failed fetch writes no cache entry, so neither the TTL sweep nor the LRU
eviction could drop its lock. A caller-controlled stream of failing spec URLs
grew _spec_locks without bound. Pop the lock on the failure path.

Return a deep copy on the cache-miss path so no caller can mutate the cached
spec. Extract the duplicated TTL freshness check into _fresh_cached_copy().
Add debug logging for cache hit, cache miss, and single-flight coalescing.

Add unit regressions for the lock leak, copy isolation, and LRU bound. Add
black-box coverage for POST /v1/tools/generate-schemas-from-openapi against a
running gateway.

Closes IBM#6835

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

* fix: cache failed OpenAPI spec fetches for a short TTL

A failed fetch wrote no cache entry, so every waiter queued behind the single-flight lock ran the failing fetch again. N waiters paid N times the timeout.

Store the raised exception under a short negative TTL of 5 seconds. Waiters re-raise it from the cache instead of refetching. Cache entries now hold an expiry, so specs and failures share one bounded map with different lifetimes.

Drop the manual lock cleanup on the failure path. A failure now writes a cache entry, so the TTL sweep and the LRU eviction reclaim its lock.

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>

---------

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>
* fix: preserve empty dict in structuredContent handling

Empty dicts are valid MCP responses but were being discarded by the 'or'
operator (which treats {} as falsy). This caused RuntimeError for tools
with outputSchema that return structuredContent: {}.

Changed to explicit None check to preserve empty dicts while still
falling back to snake_case variant when needed.

Fixes issue where Hubbl and other MCP servers fail with 'Tool has an
output schema but did not return structured content' error.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* test: add regression test for empty dict in structuredContent

Add unit test to verify that structuredContent: {} is preserved through
the non-direct-proxy MCP tool invocation path. This prevents regression
of the bug where Python's 'or' operator treated empty dicts as falsy.

Also harmonize the same pattern in streamablehttp_transport.py to use
explicit None checks for consistency and future safety.

Addresses review findings for PR IBM#6182.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: preserve empty-dict structuredContent on call_tool success paths

Both success-path return sites in call_tool() gated on plain truthiness
(if structured:), which silently dropped an empty-dict structuredContent
and returned a bare list. The MCP SDK then raised:

  RuntimeError: Tool has an output schema but did not return structured content

Fix both sites with an explicit is-None check so that {} is preserved:
- streamablehttp_transport.py primary path (~line 2126)
- streamablehttp_transport.py forwarded/session-affinity path (~line 1965)

The error-path branches (isError=True) already pass structured through
directly and are unaffected.

Add two regression tests covering call_tool() directly (not just
ToolService.invoke_tool) with structured_content={} for both the primary
and forwarded paths.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fixed the tests

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed tests

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
…le (IBM#6654)

* feat: make MAX_HEADER_VALUE_LENGTH configurable via environment variable

- Add max_header_value_length config field with 16KB default (up from hardcoded 4KB)
- Remove hardcoded MAX_HEADER_VALUE_LENGTH constants from passthrough_headers.py and translate_header_utils.py
- Update sanitize_header_value() functions to use configurable value with Mock-safe fallback
- Add tests for Mock-safe fallback behavior in mocked test environments
- Update .env.example with MAX_HEADER_VALUE_LENGTH documentation
- Adjust test expectations for increased default limit

Closes IBM#6653

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* test: add comprehensive coverage for Mock-safe fallback in header sanitization

- Add unit tests for Mock object detection in sanitize_header_value()
- Add tests for AttributeError and TypeError exception handling
- Cover previously uncovered lines in translate_header_utils.py (lines 101-103)
- Cover previously uncovered lines in passthrough_headers.py (lines 122-123)
- Tests verify fallback to 16384 default when settings access fails

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* refactor: move max_header_value_length validation to config.py

Move validation and default value handling for MAX_HEADER_VALUE_LENGTH
from runtime fallback logic to Pydantic field validation in Settings.

Changes:
- Add field_validator in config.py to validate max_header_value_length
  is an integer, positive, and <= max_header_field_size_bytes
- Change default from 16KB to 4KB to match original MAX_HEADER_VALUE_LENGTH
  constant (prevents regression)
- Remove Mock-safe isinstance() checks from sanitize_header_value() in
  passthrough_headers.py and translate_header_utils.py
- Update .env.example to reflect 4KB default
- Remove obsolete Mock-safe fallback tests that tested the removed behavior
- Update all test files to properly mock max_header_value_length = 4096
  when patching settings objects
- Fix unrelated ruff E712 error in gateway_service.py (use is_active
  instead of is_active == True)

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fix: address review feedback on max_header_value_length validator

- Remove cross-field validator that rejected MAX_HEADER_VALUE_LENGTH=16384
  and broke deployments with MAX_HEADER_FIELD_SIZE_BYTES < 4096; the two
  fields are independent — validator now only enforces positivity
- Drop unreachable isinstance(v, int) dead code in the same validator
- Restore test_large_header_values and test_multiple_concurrent_requests
  that were removed from tests/integration/test_translate_dynamic_env.py
- Update test_config_validation to assert 16384 is valid and that
  max_header_field_size_bytes does not constrain max_header_value_length
- Remove stray mock assignment line inserted mid-docstring in
  test_passthrough_headers.py
- Restore is_active == True with noqa/pylint disable suppression in
  gateway_service.py transfer_gateway_ownership, consistent with all
  other is_active == True sites in the codebase

Closes IBM#6653

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: address second-round review findings for max_header_value_length

- Join config.py description string onto one line to satisfy ruff-format
  (CI 'Run pre-commit hooks' job was failing on this field).
- Reword .env.example comment: drop false 'Must be <= MAX_HEADER_FIELD_SIZE_BYTES'
  and note that MAX_HEADER_FIELD_SIZE_BYTES and MAX_HEADER_TOTAL_SIZE_BYTES must
  also be raised for tokens near or above 8KB (HeaderSizeMiddleware enforces them
  first and returns 431 before sanitization runs).
- Add TestVaultPluginLargeToken in test_vault_plugin.py: three tests that
  reproduce the issue IBM#6653 failure (4KB truncation → invalid JSON), confirm
  intact parse with a raised limit, and exercise the full VaultPlugin end-to-end
  path with an ~8KB Atlassian-style token.
- Drop unreachable try/except (AttributeError, TypeError) fallback in three test
  files; read settings.max_header_value_length directly.

Closes IBM#6653

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* update .secrets.baseline

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fixed tests and improved Changelog details

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

* fixed pre-commit

Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>

---------

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Signed-off-by: Jitesh Nair <jiteshnair@ibm.com>
Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Co-authored-by: Jitesh Nair <jiteshnair@ibm.com>
…ailures (IBM#6181)

* fix: return MCP-compliant error responses for tool execution failures

MCP protocol requires all tool results to include a 'content' field with
an 'isError' flag for failures. Previously, runtime errors (server crashes,
timeouts, network failures) raised exceptions that were converted to
JSON-RPC errors, causing clients to fail with 'content: Field required'
validation errors.

Changes:
- Modified invoke_tool_direct to return CallToolResult with isError=true
  instead of raising ToolInvocationError for connection/execution failures
- Updated SSE transport timeout handler to return structured error response
- Updated SSE transport exception handler to return structured error response
- Updated StreamableHTTP transport timeout handler to return structured error
- Updated StreamableHTTP transport exception handler to return structured error
- Updated affected tests to expect structured error responses instead of
  exceptions

All MCP execution paths now return properly structured CallToolResult objects
even when the upstream server fails, ensuring protocol compliance and better
error handling for MCP clients like langchain-mcp-adapters.

Pre-execution errors (auth failures, validation errors, configuration issues)
continue to raise exceptions as appropriate since they prevent execution from
starting.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix(mcp): address PR review feedback for protocol compliance

- Update invoke_tool_direct docstring to reflect runtime error handling
- Clarify inline comment about previous exception behavior
- Simplify test assertions to use result.isError directly

Addresses review feedback on PR fixing MCP protocol violation (IBM#6275)

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: address review findings for MCP tool error response structure

- Fix IBM#1 (blocking/CWE-209): route invoke_tool_direct exception text
  through sanitize_exception_message before returning in CallToolResult;
  snapshot gateway.auth_query_params inside the ORM session so secrets
  embedded in gateway URLs are redacted before reaching the client.

- Fix IBM#2 (blocking/retry_on_status): attach structured_content with
  status_code to CallToolResult in the SSE and StreamableHTTP inner
  BaseException handlers so the retry plugin can honour retry_on_status
  (e.g. 429, 503) without reaching the outer handler.

- Fix IBM#3 (functionally-impacting): re-raise SystemExit, GeneratorExit,
  and KeyboardInterrupt before the BaseException catch-all in both SSE
  and StreamableHTTP handlers; mirrors the existing CancelledError guard
  (PR IBM#3202 review B7).

- Should-fix: update connect_to_sse_server and
  connect_to_streamablehttp_server docstrings to reflect that these
  functions now return instead of raising for runtime failures.

- Should-fix: extract _make_mcp_tool_error() static helper to eliminate
  five duplicate CallToolResult(isError=True) constructions; structured_
  content is now threaded through a single site.

- Tests: add test_invoke_tool_direct_error_is_sanitized to cover finding
  IBM#1; add test_sse/streamablehttp_http_status_error_sets_structured_
  content_status_code to cover finding IBM#2.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: restore ToolTimeoutError raise on SSE/StreamableHTTP timeout paths and fix structuredContent kwarg

- SSE and StreamableHTTP timeout handlers were returning
  self._make_mcp_tool_error() instead of raising ToolTimeoutError.
  The return value fell through into the post-process block, firing
  TOOL_POST_INVOKE a second time after _run_timeout_post_invoke() had
  already fired it once.  Restore raise ToolTimeoutError() so the
  existing outer handler (line 7174) is the sole post-invoke callsite
  for the timeout path.

- _make_mcp_tool_error() passed structured_content= (snake_case) to
  types.CallToolResult, whose field is structuredContent (camelCase)
  with no alias and extra='allow'.  The value was stored as an
  unrecognised extra attribute, so _coerce_to_tool_result() dropped
  it and structured_content was always None on the direct-proxy path.
  Fix: use structuredContent= kwarg.

- Update _run_timeout_post_invoke docstring: 'control returns to the
  caller which raises' was stale; the caller now raises itself.

- Add tests:
  TestMcpSseTimeoutPostInvokeFiresOnce: asserts invoke_hook fires
  exactly once on an unretried SSE timeout.
  TestMcpStreamableHttpTimeoutPostInvokeFiresOnce: same for
  StreamableHTTP.
  TestMakeToolErrorStructuredContentSurvivesCoercion: asserts
  structuredContent from _make_mcp_tool_error survives
  _coerce_to_tool_result (direct-proxy regression guard).

- Update two existing SSE/StreamableHTTP timeout tests that asserted
  the old broken return-instead-of-raise behaviour; they now expect
  ToolTimeoutError to be raised.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* tests: improve coverage

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>

* fix: update tool_service tests for MCP SDK v2 client API

Tests patched the removed tool_service.sse_client /
tool_service.streamablehttp_client / tool_service.ClientSession symbols and
asserted the camelCase structuredContent/isError attributes. Main's MCP SDK
v2 migration (IBM#6868) replaced those with a single mcp_proxy_client context
manager and renamed CallToolResult's fields to structured_content/is_error.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
…BM#6967)

The auto-created MCP tool for an A2A-wrapped agent validates its
arguments against a schema that requires `query`, not `message`
(see a2a_protocol.py's request-payload handling). The test sent
{"message": "hi"}, so the gateway rejected the tools/call request
at schema validation before it ever reached the upstream A2A
backend, making the downstream vault-header assertion fail for an
unrelated reason.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix: harden server-scoped tool lookup

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: isolate tool lookup cache by invocation scope

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: update tool cache scope implementation

- Modified tool_lookup_cache.py to handle scope-aware caching
- Updated server_service.py and tool_service.py for cache integration
- Added comprehensive test coverage for cache scoping behavior
- Updated e2e tests to validate cache scope functionality

Closes IBM#6959

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: update cache mocks to support get_negative async method

- Add get_negative AsyncMock to cache mocks in test_mcp_apps_security.py
- Add deprecated field to cache payload in test_tool_service_coverage.py
- Simplify test_server_scoping_denies_unattached_tool to test DB path directly

Fixes 4 failing tests related to tool cache scope implementation.

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* test: cover stale scoped cache authorization

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* feat: implement server-scoped tool lookup cache with negative caching

- Add server_id parameter to tool_lookup_cache get/set/invalidate methods
- Implement caller-scoped negative caching for tool resolution failures
- Update registry_cache to support server-scoped invalidation
- Add comprehensive test coverage for cache scoping behavior
- Update a2a_service and tool_service to use scoped cache operations

This enables per-virtual-server tool caching and prevents cache pollution
across different server contexts.

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: address tool cache review findings

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* chore: add missing file header to test_tool_lookup_cache_performance.py

Pre-commit hook automatically added the required copyright header.

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

* fix: validate negative cache key shape

Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com>
Co-authored-by: lucarlig <luca.carlig@ibm.com>
* Update tool service to return MCP spec compliant list instead of dict

Signed-off-by: Chris PC <chrispc@li-4dc2bf4c-325d-11b2-a85c-b68e8b1fc307.ibm.com>

* Update unit test to reflect changes

Signed-off-by: Chris PC <chrispc@li-4dc2bf4c-325d-11b2-a85c-b68e8b1fc307.ibm.com>

* Fix coverage

Signed-off-by: Chris PC <chrispc@li-4dc2bf4c-325d-11b2-a85c-b68e8b1fc307.ibm.com>

* fix: limit non-JSON 2xx errors to tools with outputSchema

A non-JSON 2xx body is now an error only when the tool declares an
outputSchema, which is the case IBM#6199 describes. Tools without a schema
keep passing the raw text through as a successful result.

On the schema path the parse error is returned directly instead of being
fed to extract_using_jq, where TextContent failed serialization and the
truncation message was replaced by a generic jsonpath error.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: only read error key from dict JSON error bodies

A valid JSON array on a 203/207 response raised AttributeError on
result.get, which the parse-error handler then reported as a JSON parse
error. A non-2xx array containing "error" raised TypeError on indexing.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: correct parse-error advice and bound JSON error echo

The parse-error text no longer tells operators to raise REST_RESPONSE_TEXT_MAX_LENGTH. That limit applies only after parsing fails.

A non-2xx JSON body without an error key now echoes the raw text, truncated to the limit. This removes an orjson TypeError on big integers and an unbounded echo.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: live gateway check for non-JSON body on outputSchema tool

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: run non-JSON outputSchema live check on its own gateway

The previous test called the bare /mcp mount, which scopes lookups to the default_server_id sentinel and cannot find the tool. It also used an upstream that a docker gateway cannot reach.

The test now starts its own gateway, adds the tool to a virtual server, and calls it through /servers/{id}/mcp/.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: assert the live upstream receives the GET

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* test: assert truncation bound and add large-valid-JSON boundary case

The truncated-body test only checked for the parse-error text, so it
would pass even without truncation. It now sets an explicit
REST_RESPONSE_TEXT_MAX_LENGTH and asserts the echoed text stops at that
exact limit.

A new test proves the limit only bounds the echoed error text: a valid
JSON body longer than the limit still parses and returns structured
content.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

* fix: bound non-string error values to the response length limit

A non-2xx or 203/207 JSON body with a non-string 'error' value is
serialized fresh with orjson and was never bounded by
_handle_json_parse_error. It could forward an arbitrarily large value,
bypassing REST_RESPONSE_TEXT_MAX_LENGTH. The serialized text is now cut
to the same limit.

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

---------

Signed-off-by: Chris PC <chrispc@li-4dc2bf4c-325d-11b2-a85c-b68e8b1fc307.ibm.com>
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Chris PC <chrispc@li-4dc2bf4c-325d-11b2-a85c-b68e8b1fc307.ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Failures from the anyio/httpx transport family reached the caller as a
bare "Tool invocation failed: ". The outer handler of invoke_tool built
the caller-facing message as str(root_cause), and ClosedResourceError,
EndOfStream, BrokenResourceError and ReadTimeout all stringify to '',
so a caller could not tell a closed stream from a read timeout. The
exception class was logged server-side but never reached the message.

Fall back to the qualified class name when str() is empty. Exceptions
that already carry a message (for example httpx.HTTPStatusError on a
503) are unaffected.

A read timeout raised inside the MCP SDK's TaskGroup arrives wrapped in
an ExceptionGroup, so it misses the per-transport timeout lanes that
emit the typed "Tool invocation timed out after {timeout}s" message and
lands in the same handler. Restate it in that vocabulary so callers key
on one wording regardless of how the timeout arrived.

Closes IBM#6592

Signed-off-by: Ewerton Silva <ewertoncom297@gmail.com>
@Ewertonslv
Ewertonslv force-pushed the fix/6592-empty-transport-error-message branch from bbb7a88 to 833b94b Compare September 28, 2026 13:26
@Ewertonslv

Copy link
Copy Markdown
Author

Friendly ping — could a maintainer approve the fork workflows on this PR? Only DCO has run so far, so the test suite hasn't executed yet. Happy to rebase or adjust anything. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet