Skip to content

Commit b59bf60

Browse files
fix(service-automation): a dispatch to a held connector instance names its fix, not a retry (#28037)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DUGZNCw4uuwpe5WcHJJSY8f)_ domain:server#3 · os-elon-musk · session_016WV4cw8ZC8W9Jeq9E7gY3X Fixes #28016 Clause-②: no Metadata-protocol: none Skills-docs: none — `git grep -n -i -e "retries automatically" -e "retried with backoff" -- skills` finds nothing, and `git grep -n -i degraded -- skills` finds one line (`skills/objectstack-automation/references/integration-recipes.md:9`, an unset ref leaves its instance degraded until set), which stays true. The docs row this card owes is in this PR. `docs/adr/0097-declarative-connector-instances.md:190` still quotes "the platform retries automatically" for every husk; it is governed and outside this card's surface (see Acceptance notes). ## 维护者速读 - 改了什么:连接器实例「被挂起」(重试时遇到配置错误,或上游拒绝了已存储的凭据)时,`connector_action` 派发的拒绝信息不再说「平台会自动重试」,而是写明怎么恢复:编辑声明,或重新设置凭据 `<ref>`(或重启)。真正在退避重试的实例仍说会自动重试。 - 为什么改:#28014 / #27237 之后这类实例根本不会被重试,原来那句话是假的,作者会一直等。 - 风险与回滚:只改拒绝文案和一条引擎 warn 日志;引擎的降级记录多一个可选的 `degradedHold` 字段,新增只读方法 `getConnectorDegradedHold`,`registerDegradedConnector` 多一个可选第五参数,原有调用不变。回滚即还原本 PR。 - 先看:`builtin/connector-nodes.ts`(拒绝文案)、`plugin.ts` 两处挂起点传入 hold、`connector-held-dispatch-refusal.test.ts`。 - 两分钟验证:`pnpm --filter @objectstack/service-automation exec vitest run src/connector-held-dispatch-refusal.test.ts src/connector-credential-door.test.ts`,预期 30/30 通过。 - 需要你做的:无,席位落地。 ## What changes - **The degraded record carries the hold.** `RegisteredConnector` gains an optional `degradedHold: ConnectorHold`, `{ kind: 'declaration' }` or `{ kind: 'credential', ref }`. `registerDegradedConnector` takes it as an optional fifth argument (engine delegate and the body in `plugin-extension-api.ts`), and `AutomationEngine.getConnectorDegradedHold(name)` reads it. A later degrade of the same name replaces the record, so an instance that goes back on the backoff loses the hold. - **`plugin.ts` passes it at the two hold sites.** `failDegradedConnectorInstance` (a configuration fault met on a retry, #27992) passes `declaration`. `holdConnectorForRejectedCredential` passes `credential` with the row's `ref` for a stored value, and `declaration` for an instance with no `credentialRef` (its credential is written into the declaration, so only an edit changes what it sends). An environment or vault value stays on the backoff and passes nothing. - **The refusal (`builtin/connector-nodes.ts`).** Held: `Dispatch is unavailable until its declaration is edited (or the server restarts); the platform does not retry it.` or `… until credential '<ref>' is set again (or the server restarts); …`. On the backoff: unchanged, `… the platform retries automatically.` A reason that already ends with a full stop no longer gets a second one (`…does not list.. Dispatch` before). - **The engine's `Connector registered DEGRADED` warn** says the materializer does not retry a held instance, and until what. The unheld wording is byte-identical (pinned in `degraded-register-cause.test.ts`). - **Docs.** `content/docs/automation/connectors.mdx`'s failure-modes table gains the row for a configuration fault met on a retry. - **Changeset.** `@objectstack/service-automation` patch. ## Tests - New `src/connector-held-dispatch-refusal.test.ts` (4): a configuration fault met on the backoff retry; a credential written into the declaration refused by the upstream; control: an unreachable upstream keeps "retries automatically"; the engine record carries the hold, its warn says so, and an unheld degrade clears it. - `src/connector-credential-door.test.ts` (+3, real ObjectQL over SQLite): a stored value refused at materialization (mcp shape) and on a call (rest / openapi shape) says to set credential `<ref>` again; control: an environment value refused on a call keeps "retries automatically". ## Serial: open PR #27951 (`engine.ts`) #27951 (seat 4) is still open. The two `engine.ts` regions do not overlap. This PR's hunks: `@@ -667,8 +667,27 @@` (`RegisteredConnector`), `@@ -3982,12 +4001,16 @@` and `@@ -3995,7 +4018,7 @@` (the `registerDegradedConnector` delegate), `@@ -4142,6 +4165,17 @@` (the new reader beside `getConnectorDegradedReason`). #27951's hunks against `main`: `@@ -6373,6 …`, `@@ -6398,8 …`, `@@ -6416,7 …`, `@@ -6594,38 …` (resume paths). Over 2,000 lines apart. ## Acceptance notes - `docs/adr/0097-declarative-connector-instances.md:190` says a dispatch to a husk fails with "the platform retries automatically". After this PR that holds only for an instance on the backoff. The ADR is governed and outside this card's surface, so it is left as is. - `ConnectorDescriptor` (`GET /api/v1/automation/connectors`) does not carry the hold; adding it is a `packages/spec` change, outside this card. The listing's `degradedReason` for a refused credential already says it is not retried. - The additions to the engine's surface (`getConnectorDegradedHold`, the optional `degradedHold` field and fifth argument, the `ConnectorHold` type exported from `engine.ts`, not from the package entry) are additive and optional; no accept set narrows, hence `Clause-②: no` as ordered. ## Verification All readings at head `824e0f5d`. <details><summary>Red on main, green with the fix, ablations, restore</summary> - On `main`'s `engine.ts`, `plugin-extension-api.ts`, `plugin.ts`, `builtin/connector-nodes.ts` with the new tests: `Tests 4 failed | 25 passed (29)`. Each held case received `… Dispatch is unavailable until that is resolved; the platform retries automatically.` (the configuration case as `…does not list.. Dispatch …`). Both controls passed. - With the fix: the five related files (`connector-held-dispatch-refusal`, `connector-credential-door`, `connector-retry-configuration-fault`, `connector-materialization`, `degraded-register-cause`) `Tests 93 passed (93)`. - Ablation through `scripts/ablation-replace.mjs`, fix committed first, each restored with blob == HEAD and `git diff HEAD` empty: 1. `builtin/connector-nodes.ts`, `const hold = engine.getConnectorDegradedHold(cfg.connectorId);` to `undefined`: anchor 1 to 0, blob 2491245 to 98ae59d65b55, `Tests 4 failed | 26 passed (30)` (the four held dispatch cases; controls green). 2. `plugin.ts`, the credential-hold argument to `undefined`: blob 6a0a751 to 11d034a84acc, `Tests 3 failed | 27 passed (30)` (declared credential, stored credential x2). 3. `plugin.ts`, `failDegradedConnectorInstance`'s `{ kind: 'declaration' }` to `undefined`: `Tests 1 failed | 3 passed (4)` (the configuration case). </details> <details><summary>Package suite, typecheck, lint</summary> - `pnpm --filter @objectstack/service-automation exec vitest run`: `Test Files 224 passed (224)`, `Tests 2848 passed | 1 expected fail (2849)`. - `pnpm --filter @objectstack/service-automation run typecheck`: exit 0. - `pnpm lint`: exit 0. </details> <details><summary>dispatch-gates families (92 derived, 92 run, 0 NOT-MEASURED)</summary> `check-plugin-teardown-shape --self-test` after fetching its pinned commit; `check:skill-examples` and `check:dual-build-cjs-loads` exited 3 on the closure build and 0 after a whole-tree `turbo run build` (exit 0). `dispatch-gates --ran`: 92 derived families accounted for, 92 run, 0 NOT-MEASURED. ```text node scripts/check-adr-0087-registration.mjs --base origin/main :: exit 0 node scripts/check-adr-0087-registration.mjs --self-test :: exit 0 node scripts/check-adr-links.mjs :: exit 0 node scripts/check-adr-links.mjs --self-test :: exit 0 node scripts/check-changeset-no-major.mjs --base origin/main :: exit 0 node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-doc-route-spelling.mjs --advisory :: exit 0 node scripts/check-doc-route-spelling.mjs --self-test :: exit 0 node scripts/check-docs-section-name.mjs :: exit 0 node scripts/check-docs-section-name.mjs --self-test :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-empty-changeset.mjs --base origin/main :: exit 0 node scripts/check-empty-changeset.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/check-section-landing-index.mjs :: exit 0 node scripts/check-section-landing-index.mjs --self-test :: exit 0 node scripts/check-tenant-audit-census.mjs :: exit 0 node scripts/check-tenant-audit-census.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 node scripts/pm/release-rehearsal-clone.mjs --self-test :: exit 0 node scripts/release-pending-publish.mjs --self-test :: exit 0 pnpm --filter @objectstack/lint run check:doc-formula-expressions :: exit 0 pnpm --filter @objectstack/lint run check:doc-security-posture :: exit 0 pnpm --filter @objectstack/spec run check:docs :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm --filter @objectstack/spec run check:empty-state :: exit 0 pnpm --filter @objectstack/spec run check:liveness :: exit 0 pnpm --filter @objectstack/spec run check:strictness-ledger :: exit 0 pnpm --filter @objectstack/spec run check:variant-docs :: exit 0 pnpm --filter @objectstack/spec run check:yaml-examples :: exit 0 pnpm check:changeset-gate-self-tests :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:corpus-claim-drift :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:dispatcher-error-vocabulary :: exit 0 pnpm check:doc-anchors :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:doc-frontmatter :: exit 0 pnpm check:docs-audit-scope :: exit 0 pnpm check:docs-redirects :: exit 0 pnpm check:docs-single-h1 :: exit 0 pnpm check:docs-spec-enumerations :: exit 0 pnpm check:docs-transcript-drift :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:durability-log-level :: exit 0 pnpm check:engine-double-contract :: exit 0 pnpm check:error-status-conformance :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:pm-changeset-deadline-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:published-readme-links :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:react-page-adapter-contract :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:role-word :: exit 0 pnpm check:skill-identifier-liveness :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vendor-version-stamps :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 pnpm --filter @objectstack/spec run check:skill-examples :: exit 0 pnpm check:dual-build-cjs-loads :: exit 0 pnpm lint :: exit 0 ``` </details> --- _Generated by [Claude Code](https://claude.ai/code/session_016WV4cw8ZC8W9Jeq9E7gY3X)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016WV4cw8ZC8W9Jeq9E7gY3X Co-authored-by: Claude <noreply@anthropic.com>
1 parent 5aee206 commit b59bf60

8 files changed

Lines changed: 334 additions & 14 deletions

File tree

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
'@objectstack/service-automation': patch
3+
---
4+
5+
fix(service-automation): a `connector_action` dispatch to a held connector instance names what recovers it, instead of saying the platform retries it
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
A degraded connector instance that the platform does not retry (it is **held**) used to answer a dispatch with "Dispatch is unavailable until that is resolved; the platform retries automatically." Nothing retried it. Now the refusal names the fix:
11+
12+
- **A configuration fault met on a retry** (#27992), or **a credential written into the declaration that the upstream refused**: "Dispatch is unavailable until its declaration is edited (or the server restarts); the platform does not retry it."
13+
- **A stored credential the upstream refused** (#27237): "Dispatch is unavailable until credential '<ref>' is set again (or the server restarts); the platform does not retry it."
14+
- **An instance on the backoff** (an unreachable upstream, an environment or vault credential the upstream refused) keeps "the platform retries automatically".
15+
16+
The engine's degraded-connector record carries the hold: `registerDegradedConnector` takes an optional fifth argument, `RegisteredConnector` gains an optional `degradedHold`, and `AutomationEngine.getConnectorDegradedHold(name)` reads it. The engine's `Connector registered DEGRADED` warning no longer says the materializer retries a held instance. A degraded reason that ends with a full stop no longer gets a second one in the refusal.

‎content/docs/automation/connectors.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,7 @@ retry):
454454
| Instance `name` collides with a plugin-registered connector | **Hard boot error** — there is no silent precedence between the two worlds |
455455
| stdio transport without host opt-in | **Hard boot error** with the `declarativeStdio` fix inline — a security rejection is never retried |
456456
| MCP server unreachable / `tools/list` fails; remote spec URL unreachable or transiently failing (408/429/5xx) | **Not fatal**: the instance registers **degraded** (visible on `GET /api/v1/automation/connectors` with the reason), dispatches fail with that reason, and the platform retries on a 5s→5min backoff — recovery is automatic |
457+
| A retry of a degraded instance reaches the upstream and meets a configuration fault (an `include` naming a tool or operation the upstream does not offer) | **Not fatal**: the instance stays **degraded**, now with the configuration reason, and is **not retried** — dispatches fail saying to edit its declaration; an edit of the entry (or a restart) re-attempts it |
457458
| Wrong spec URL (other 4xx) or unparseable document | Configuration fault — **hard boot error** |
458459
| A bad entry arrives via **runtime reload** (Studio publish / dev reload) | Logged and **skipped**, never crashing the live server; a changed instance's old connector keeps serving until its replacement materializes |
459460

‎packages/services/service-automation/src/builtin/connector-nodes.ts‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { PluginContext } from '@objectstack/core';
44
import { defineActionDescriptor } from '@objectstack/spec/automation';
55
import type { AutomationEngine, ConnectorActionContext } from '../engine.js';
66
import { refuseNode } from '../guard-refusal.js';
7+
import { describeHold } from '../plugin-extension-api.js';
78
import { resolveValueSlotMap } from './value-slot-map.js';
89

910
/**
@@ -63,17 +64,24 @@ export function registerConnectorNodes(engine: AutomationEngine, ctx: PluginCont
6364
const handler = engine.resolveConnectorAction(cfg.connectorId, cfg.actionId);
6465
if (!handler) {
6566
// A degraded declarative instance (#3017) is registered but has no
66-
// handlers — say WHY dispatch fails and that recovery is automatic,
67-
// instead of the generic wiring hint below. The reason names the
68-
// cause: an unreachable upstream, or (#27237) a credential that
69-
// is not set.
67+
// handlers — say WHY dispatch fails and what recovers it, instead
68+
// of the generic wiring hint below. The reason names the cause: an
69+
// unreachable upstream, or (#27237) a credential that is not set or
70+
// was refused. [#28016] Recovery is automatic only on the backoff:
71+
// a held instance (a configuration fault met on a retry, a refused
72+
// credential the platform will not re-send) names its fix instead.
7073
const degraded = engine.getConnectorDegradedReason(cfg.connectorId);
7174
if (degraded) {
75+
const hold = engine.getConnectorDegradedHold(cfg.connectorId);
76+
// A reason that is a full sentence keeps one full stop.
77+
const reason = degraded.replace(/\.\s*$/, '');
7278
return {
7379
success: false,
7480
error:
75-
`connector_action '${node.id}': connector '${cfg.connectorId}' is degraded — ${degraded}. `
76-
+ `Dispatch is unavailable until that is resolved; the platform retries automatically.`,
81+
`connector_action '${node.id}': connector '${cfg.connectorId}' is degraded — ${reason}. `
82+
+ (hold
83+
? `Dispatch is unavailable until ${describeHold(hold)} (or the server restarts); the platform does not retry it.`
84+
: `Dispatch is unavailable until that is resolved; the platform retries automatically.`),
7785
};
7886
}
7987
return {

‎packages/services/service-automation/src/connector-credential-door.test.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -722,3 +722,64 @@ describe('[#27237] contract review: an encoded spelling of the value is withheld
722722
}
723723
});
724724
});
725+
726+
/** [#28016] Run a one-node flow dispatching to `billing.request` and return its refusal text. */
727+
async function dispatchRefusal(engine: AutomationEngine): Promise<string> {
728+
engine.registerFlow('call_billing', {
729+
name: 'call_billing',
730+
label: 'Call billing',
731+
type: 'autolaunched',
732+
nodes: [
733+
{ id: 'start', type: 'start', label: 'Start' },
734+
{ id: 'call', type: 'connector_action', label: 'Request', connectorConfig: { connectorId: 'billing', actionId: 'request', input: {} } },
735+
{ id: 'end', type: 'end', label: 'End' },
736+
],
737+
edges: [
738+
{ id: 'e1', source: 'start', target: 'call' },
739+
{ id: 'e2', source: 'call', target: 'end' },
740+
],
741+
} as never);
742+
const result = await engine.execute('call_billing');
743+
expect(result.success).toBe(false);
744+
return String(result.error);
745+
}
746+
747+
describe('[#28016] a dispatch to an instance held on a refused stored credential says to set it again', () => {
748+
it('refused at materialization (the mcp shape): set the credential again; nothing retries it', async () => {
749+
const { engine, provider } = await boot([billing()]);
750+
provider.state.rejectAtMaterialization = true;
751+
await engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE, context: ADMIN });
752+
753+
const error = await dispatchRefusal(engine);
754+
expect(error).toContain("connector 'billing' is degraded — ");
755+
expect(error).toContain(`until credential '${REF}' is set again (or the server restarts); the platform does not retry it.`);
756+
expect(error).not.toMatch(/retries automatically/);
757+
expect(error).not.toContain('.. Dispatch');
758+
expect(error.includes(DUMMY_VALUE)).toBe(false);
759+
});
760+
761+
it('refused on a call (the rest / openapi shape): set the credential again; nothing retries it', async () => {
762+
const { engine, provider, call } = await boot([billing()]);
763+
await engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE, context: ADMIN });
764+
provider.state.callStatus = 401;
765+
await call();
766+
767+
const error = await dispatchRefusal(engine);
768+
expect(error).toContain(`until credential '${REF}' is set again (or the server restarts); the platform does not retry it.`);
769+
expect(error).not.toMatch(/retries automatically/);
770+
expect(error).not.toContain('.. Dispatch');
771+
});
772+
773+
it('control: an environment value refused on a call is on the backoff, and the refusal says the platform retries it', async () => {
774+
process.env[REF] = DUMMY_ENV_VALUE;
775+
envKeys.push(REF);
776+
const { engine, provider, call } = await boot([billing()]);
777+
provider.state.callStatus = 403;
778+
await call();
779+
780+
const error = await dispatchRefusal(engine);
781+
expect(error).toContain('The instance is retried with backoff.');
782+
expect(error).toContain('Dispatch is unavailable until that is resolved; the platform retries automatically.');
783+
expect(error).not.toMatch(/does not retry/);
784+
});
785+
});
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// #28016 — a `connector_action` dispatch to a degraded instance says what
4+
// recovers it. An instance on the #3017 backoff recovers by itself, so its
5+
// refusal says the platform retries it. A HELD instance is not retried:
6+
// a configuration fault met on a backoff retry (#27992), or a credential the
7+
// upstream refused when only an edit of the declaration changes what it sends
8+
// (#27237). Its refusal names the fix and never says it retries. The stored-
9+
// credential hold (set the credential again) is pinned beside its door, in
10+
// connector-credential-door.test.ts.
11+
12+
import { describe, it, expect, afterEach, vi } from 'vitest';
13+
import { LiteKernel, ObjectLogger } from '@objectstack/core';
14+
import type { Connector, ConnectorProviderFactory } from '@objectstack/spec/integration';
15+
import {
16+
ConnectorCredentialRejectedError,
17+
ConnectorUpstreamUnavailableError,
18+
} from '@objectstack/spec/integration';
19+
import { AutomationServicePlugin, DECLARATIVE_RETRY_BASE_MS } from './plugin.js';
20+
import { AutomationEngine } from './engine.js';
21+
22+
type Upstream = 'down' | 'refuses-include' | 'refuses-credential';
23+
24+
/** An mcp-shaped provider: the marker errors for down / refused credential, a plain Error for a bad `include`. */
25+
function makeProvider(upstream: Upstream) {
26+
const state = { upstream, calls: 0 };
27+
const factory: ConnectorProviderFactory = (ctx) => {
28+
state.calls++;
29+
if (state.upstream === 'down') {
30+
throw new ConnectorUpstreamUnavailableError(
31+
`connector-mcp provider: connector '${ctx.name}' could not reach its MCP server: connect ECONNREFUSED`,
32+
);
33+
}
34+
if (state.upstream === 'refuses-credential') {
35+
throw new ConnectorCredentialRejectedError(
36+
`connector-mcp provider: connector '${ctx.name}' was refused by its upstream (HTTP 401).`,
37+
401,
38+
);
39+
}
40+
throw new Error(
41+
`connector-mcp provider: connector '${ctx.name}' providerConfig.include names 'nope', which the MCP server does not list.`,
42+
);
43+
};
44+
return { factory, state };
45+
}
46+
47+
const ENTRY = { name: 'upstream', label: 'upstream', type: 'api', provider: 'mcp', providerConfig: { url: 'https://upstream.example', include: ['nope'] } };
48+
49+
async function boot(factory: ConnectorProviderFactory) {
50+
const kernel = new LiteKernel({ logger: { level: 'silent' } } as never);
51+
kernel.use(new AutomationServicePlugin());
52+
kernel.use({
53+
name: 'test.harness',
54+
type: 'standard' as const,
55+
version: '1.0.0',
56+
dependencies: ['com.objectstack.service-automation'],
57+
async init(ctx: any) {
58+
ctx.registerService('objectql', {
59+
registry: { listItems: (type: string) => (type === 'connector' ? [ENTRY] : []) },
60+
});
61+
ctx.getService('automation').registerConnectorProvider('mcp', factory);
62+
},
63+
async start() {},
64+
} as never);
65+
await kernel.bootstrap();
66+
return { kernel, engine: kernel.getService('automation') as AutomationEngine };
67+
}
68+
69+
/** Run a one-node flow dispatching to `upstream.ping` and return its refusal text. */
70+
async function dispatchRefusal(engine: AutomationEngine): Promise<string> {
71+
engine.registerFlow('call_upstream', {
72+
name: 'call_upstream',
73+
label: 'Call upstream',
74+
type: 'autolaunched',
75+
nodes: [
76+
{ id: 'start', type: 'start', label: 'Start' },
77+
{ id: 'call', type: 'connector_action', label: 'Ping', connectorConfig: { connectorId: 'upstream', actionId: 'ping', input: {} } },
78+
{ id: 'end', type: 'end', label: 'End' },
79+
],
80+
edges: [
81+
{ id: 'e1', source: 'start', target: 'call' },
82+
{ id: 'e2', source: 'call', target: 'end' },
83+
],
84+
} as never);
85+
const result = await engine.execute('call_upstream');
86+
expect(result.success).toBe(false);
87+
return String(result.error);
88+
}
89+
90+
describe('#28016 — a dispatch to a held connector instance names its fix, not a retry', () => {
91+
afterEach(() => {
92+
vi.useRealTimers();
93+
});
94+
95+
it('a configuration fault met on the backoff retry: edit the declaration; nothing retries it', async () => {
96+
vi.useFakeTimers();
97+
const { factory, state } = makeProvider('down');
98+
const { kernel, engine } = await boot(factory);
99+
state.upstream = 'refuses-include';
100+
await vi.advanceTimersByTimeAsync(DECLARATIVE_RETRY_BASE_MS);
101+
expect(state.calls).toBe(2);
102+
103+
const error = await dispatchRefusal(engine);
104+
expect(error).toContain("connector 'upstream' is degraded — ");
105+
expect(error).toContain("include names 'nope'");
106+
expect(error).toContain('until its declaration is edited (or the server restarts); the platform does not retry it.');
107+
expect(error).not.toMatch(/retries automatically/);
108+
expect(error).not.toContain('.. Dispatch');
109+
110+
await kernel.shutdown();
111+
});
112+
113+
it('a credential written into the declaration, refused by the upstream: edit the declaration; nothing retries it', async () => {
114+
const { factory } = makeProvider('refuses-credential');
115+
const { kernel, engine } = await boot(factory);
116+
expect(engine.getConnectorDegradedReason('upstream')).toMatch(/refused by its upstream \(HTTP 401\)/);
117+
118+
const error = await dispatchRefusal(engine);
119+
expect(error).toContain('until its declaration is edited (or the server restarts); the platform does not retry it.');
120+
expect(error).not.toMatch(/retries automatically/);
121+
expect(error).not.toContain('.. Dispatch');
122+
123+
await kernel.shutdown();
124+
});
125+
126+
it('control: an unreachable upstream is on the backoff, and the refusal says the platform retries it', async () => {
127+
vi.useFakeTimers();
128+
const { factory, state } = makeProvider('down');
129+
const { kernel, engine } = await boot(factory);
130+
await vi.advanceTimersByTimeAsync(DECLARATIVE_RETRY_BASE_MS);
131+
expect(state.calls).toBe(2);
132+
133+
const error = await dispatchRefusal(engine);
134+
expect(error).toContain('could not reach its MCP server');
135+
expect(error).toContain('Dispatch is unavailable until that is resolved; the platform retries automatically.');
136+
expect(error).not.toMatch(/does not retry/);
137+
138+
await kernel.shutdown();
139+
});
140+
});
141+
142+
describe('#28016 — the degraded record carries the hold', () => {
143+
const husk = { name: 'billing', label: 'billing', type: 'api', authentication: { type: 'none' }, actions: [] } as unknown as Connector;
144+
145+
it('a held degrade stores what re-attempts it and its warn says it is not retried; an unheld one clears it', () => {
146+
const logger = new ObjectLogger({ level: 'silent' });
147+
const warn = vi.spyOn(logger, 'warn');
148+
const engine = new AutomationEngine(logger);
149+
const lastWarn = () => String(warn.mock.calls[warn.mock.calls.length - 1]?.[0]);
150+
151+
engine.registerDegradedConnector(husk, 'refused', 'declarative', undefined, { kind: 'credential', ref: 'BILLING_TOKEN' });
152+
expect(engine.getConnectorDegradedHold('billing')).toEqual({ kind: 'credential', ref: 'BILLING_TOKEN' });
153+
expect(lastWarn()).toContain(
154+
"the materializer does not retry it until credential 'BILLING_TOKEN' is set again (or a restart).",
155+
);
156+
expect(lastWarn()).not.toContain('retries with backoff');
157+
158+
// The next degrade of the same name replaces the record: on the backoff, no hold.
159+
engine.registerDegradedConnector(husk, 'could not reach it', 'declarative');
160+
expect(engine.getConnectorDegradedHold('billing')).toBeUndefined();
161+
expect(lastWarn()).toContain('the materializer retries with backoff.');
162+
expect(engine.getConnectorDegradedHold('unregistered')).toBeUndefined();
163+
});
164+
});

‎packages/services/service-automation/src/engine.ts‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -667,8 +667,27 @@ export interface RegisteredConnector {
667667
readonly state: ConnectorState;
668668
/** Why the connector is degraded — set only when `state` is `degraded`. */
669669
readonly degradedReason?: string;
670+
/**
671+
* [#28016] Set only on a degraded connector the platform does NOT retry:
672+
* what re-attempts it instead. Absent on an instance on the #3017 backoff.
673+
*/
674+
readonly degradedHold?: ConnectorHold;
670675
}
671676

677+
/**
678+
* [#28016] What re-attempts a **held** degraded instance — one neither the
679+
* retry timer nor an unchanged reload re-attempts (a restart always does):
680+
*
681+
* - `declaration`: an edit of its declaration — a configuration fault met on
682+
* a retry (#27992), or a credential written into the declaration that the
683+
* upstream refused (#27237);
684+
* - `credential`: setting the stored credential `ref` again — a value stored
685+
* through the credential door that the upstream refused (#27237).
686+
*/
687+
export type ConnectorHold =
688+
| { readonly kind: 'declaration' }
689+
| { readonly kind: 'credential'; readonly ref: string };
690+
672691
// The connector **provider** contract (ADR-0097) — ConnectorProviderFactory,
673692
// ConnectorProviderContext, ConnectorMaterialization — lives in
674693
// `@objectstack/spec/integration` so a connector plugin can implement a factory
@@ -3982,20 +4001,24 @@ export class AutomationEngine implements IAutomationService {
39824001
* @param cause the thrown value behind the degrade, for the log record's
39834002
* structured `meta`. Optional — a caller with no thrown value in hand
39844003
* still gets `degradedReason` in the record.
4004+
* @param hold [#28016] set when the platform will NOT retry the instance:
4005+
* what re-attempts it instead, stored as `degradedHold` and named by a
4006+
* `connector_action` refusal. Omitted, the instance is on the backoff.
39854007
*/
39864008
registerDegradedConnector(
39874009
def: Connector,
39884010
reason: string,
39894011
origin: ConnectorOrigin = 'declarative',
39904012
cause?: unknown,
4013+
hold?: ConnectorHold,
39914014
): void {
39924015
// The body lives in ./plugin-extension-api.ts (#25222).
39934016
const host: ThisParameterType<typeof registerDegradedConnector> = {
39944017
assertSameOriginOrFree: (...a) => this.assertSameOriginOrFree(...a),
39954018
connectors: this.connectors,
39964019
logger: this.logger,
39974020
};
3998-
registerDegradedConnector.call(host, def, reason, origin, cause);
4021+
registerDegradedConnector.call(host, def, reason, origin, cause, hold);
39994022
}
40004023

40014024
/** Enforce the ADR-0097 §4 two-sources-of-truth rule; warn on same-origin replace. */
@@ -4142,6 +4165,17 @@ export class AutomationEngine implements IAutomationService {
41424165
return reg?.state === 'degraded' ? (reg.degradedReason ?? 'upstream unavailable') : undefined;
41434166
}
41444167

4168+
/**
4169+
* [#28016] What re-attempts a held degraded connector, or `undefined` when
4170+
* it is not held — `ready`, unregistered, or degraded on the #3017 backoff.
4171+
* Lets the `connector_action` node say "retries automatically" only when
4172+
* that is true.
4173+
*/
4174+
getConnectorDegradedHold(name: string): ConnectorHold | undefined {
4175+
const reg = this.connectors.get(name);
4176+
return reg?.state === 'degraded' ? reg.degradedHold : undefined;
4177+
}
4178+
41454179
/**
41464180
* Register a connector **provider factory** (ADR-0097 §2). A connector
41474181
* plugin (e.g. `@objectstack/connector-openapi`) calls this at `init()` under

0 commit comments

Comments
 (0)