feat(realtime): expose granular server error types and payloads by VerioN1 · Pull Request #144 · DecartAI/sdk

VerioN1 · 2026-05-20T12:52:38Z

Previously, real-time server errors were generic. This change introduces
RealtimeWebSocketErrorType and attaches it, along with the full server
payload, to ServerError objects. New REALTIME_ error codes are
added to DecartSDKError to enable more specific error handling and
debugging for different real-time issues like invalid API keys,
insufficient credits, or moderation violations.

Note

Medium Risk
SDK changes alter signaling join semantics, LiveKit publish/subscribe behavior, and error classification for all realtime users; the large test-page harness is dev-only but increases maintenance surface.

Overview
The realtime SDK gains typed WebSocket server errors (RealtimeWebSocketErrorType, full payload on ServerError / RealtimeServerErrorData) and maps them to specific ERROR_CODES for callers.

Connect and media options expand: optional publishOptions, roomOptions, remoteVideoElement, and bundleInitialState (initial prompt/image inside livekit_join vs legacy post-join messages). getVideoStats() merges LiveKit sender/receiver stats with RTC report fields (QP, freezes, codecs, etc.) on publish and subscribe clients. Publish defaults now allow overrides; remote inference tracks attach to a video element; mirrored streams are video-only; subscribe path focuses on video tracks.

The SDK test page becomes a LiveKit A/B bench: chained H.264/VP9 publish profiles, SFU region pin and eager-join toggle, file/camera sources with QR stamps for end-to-end latency, Chart.js dashboards, workers for stats/CSV/QR, chain recording, and export—plus removal of the Files API UI from that page.

^{Reviewed by Cursor Bugbot for commit ca919f4. Bugbot is set up for automated code reviews on this repo. Configure here.}

Previously, real-time server errors were generic. This change introduces `RealtimeWebSocketErrorType` and attaches it, along with the full server payload, to `ServerError` objects. New `REALTIME_` error codes are added to `DecartSDKError` to enable more specific error handling and debugging for different real-time issues like invalid API keys, insufficient credits, or moderation violations.

pkg-pr-new · 2026-05-20T12:53:57Z

Open in StackBlitz

npm i https://pkg.pr.new/@decartai/sdk@144

commit: ca919f4

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…verbosity

Drop AV1 profile, lift codec to top-level preferredVideoCodec on each profile, wire it through realtime.connect. Fix client.ts/stream-session.ts hybrids left by -X ours merge so typecheck and build pass; apply biome auto-fixes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor · 2026-06-01T12:57:07Z

+  };
+  if (typeof inbound?.framesDropped === "number") merged.framesDropped = inbound.framesDropped;
+  return merged;
+}


Duplicate receiver stats enrichment logic across files

Medium Severity

enrichReceiverFromReport in subscribe-client.ts reimplements the same receiver-stats enrichment logic already provided by mergeReceiver and collectCodecMimeMap in media-channel.ts, which are already exported-compatible. This duplication directly caused the framesPerSecond omission bug — the two implementations have already diverged. Reusing the shared functions from media-channel.ts would prevent such inconsistencies.

Additional Locations (1)

packages/sdk/src/realtime/media-channel.ts#L107-L126

^{Reviewed by Cursor Bugbot for commit 58b4b97. Configure here.}

cursor · 2026-06-01T12:57:07Z

-      maxBitrate,
-      maxFramerate: REALTIME_CONFIG.livekit.defaultPublishFps,
+      ...defaultEncoding,
+      ...overrides?.videoEncoding,


VP9 codec loses automatic simulcast and bitrate defaults

Medium Severity

getDefaultVideoPublishOptions now always defaults simulcast to true and uses defaultMaxVideoBitrateBps for all codecs. Previously, VP9 automatically got simulcast: false and vp9MaxVideoBitrateBps. SDK consumers passing preferredVideoCodec: 'vp9' without explicit publishOptions will now get incorrect defaults — simulcast enabled with VP9 can cause encoding issues, and the bitrate changes from the VP9-specific value (vp9MaxVideoBitrateBps in config) to the generic default.

^{Reviewed by Cursor Bugbot for commit 58b4b97. Configure here.}

The runOneConnect path on this branch does not wait for initialStateAck before publishing local tracks; the 4 gating tests added on main fail against that behavior. Skip them for this PR; revisit when the gate question is settled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Re-introduce the `gateAttempt = initialStateGate.startAttempt(...)` capture and destructure `initialStateAck` from `signaling.openAndJoin(...)` so the existing `waitForReadiness` await (previously commented out) compiles and runs. Also lowers the playground default bench duration to 15s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor · 2026-06-03T08:38:35Z

    room.on(RoomEvent.TrackSubscribed, (track: RemoteTrack, _pub, participant: RemoteParticipant) => {
      if (!participant.identity.startsWith(REALTIME_CONFIG.livekit.inferenceServerIdentityPrefix)) return;
-      if (track.kind !== Track.Kind.Video && track.kind !== Track.Kind.Audio) return;
+      if (track.kind !== Track.Kind.Video) return;


Remote inference audio dropped

High Severity

TrackSubscribed handlers now ignore non-video tracks, so remote audio from the inference participant is never attached or added to the emitted MediaStream. Comments still describe playing video and audio together, but subscribers only receive video.

Additional Locations (1)

packages/sdk/src/realtime/subscribe-client.ts#L151-L154

^{Reviewed by Cursor Bugbot for commit 0e33b93. Configure here.}

cursor · 2026-06-03T08:38:35Z

+        maxBitrate: publishOptions.videoEncoding?.maxBitrate,
+        maxFramerate: publishOptions.videoEncoding?.maxFramerate,
+      });
+      await this.room.localParticipant.publishTrack(track, publishOptions);


Local audio never published

Medium Severity

publishTracks iterates only stream.getVideoTracks(), so microphone or other audio tracks on the caller’s MediaStream are no longer published to LiveKit, even when present.

^{Reviewed by Cursor Bugbot for commit 0e33b93. Configure here.}

- Add table below per-run summary chart with #/profile/elapsed/setup/TTFF/ connect→1st frame columns. Captures timings on each stopBenchmark and re-renders on stop and clear. - Persist chained publish profiles to localStorage (decart-playground:chained-profiles): load on init (validating against PUBLISH_PROFILES, falling back to shuffledDefaultChain) and save after every slot edit or chain extend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Place after the caller-provided queryParams so it cannot be overridden, matching the trailing api_key / model / resolution authoritative-params pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor · 2026-06-03T12:07:06Z

+            r.stream = null;
+            r.canvas = null;
+            r.ctx = null;
+        }


Chain recorder never restarts

Medium Severity

stopChainRecording stops the MediaRecorder but leaves chainRecorder.recorder set. startChainRecording returns immediately when r.recorder is truthy, so a second multi-profile chain run never starts a new recording after the first finishes.

Additional Locations (1)

packages/sdk/index.html#L2379-L2383

^{Reviewed by Cursor Bugbot for commit a954534. Configure here.}

cursor · 2026-06-03T12:07:06Z

+        maxBitrate: publishOptions.videoEncoding?.maxBitrate,
+        maxFramerate: publishOptions.videoEncoding?.maxFramerate,
+      });
+      await this.room.localParticipant.publishTrack(track, publishOptions);


Local audio never published

Medium Severity

Publishing now iterates only getVideoTracks(), and mirrored streams no longer include audio tracks. Any MediaStream with microphone audio is sent video-only to LiveKit, and remote subscription also ignores non-video inference tracks.

Additional Locations (1)

packages/sdk/src/realtime/mirror-stream.ts#L102-L166

^{Reviewed by Cursor Bugbot for commit a954534. Configure here.}

Pull the chain inter-run buffer out into CHAIN_INTER_RUN_BUFFER_MS and bump from 6s to 15s so reconnect/tear-down between profiles has more headroom. Also rewires the fallback branch to use the same constant instead of a magic 127_000. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…onnect Previous bump moved the wrong knob (chainIntervalMs, measured from remote-stream-arrived). The real problem is that advanceChain tears down and immediately reconnects, giving the remote server no time to clean up session state, which causes connect failures on the next profile. Insert an explicit cooldown timeout between teardownRealtimeKeepingCamera() and the next connectPublisher(). Stores the timeout id on chainAdvanceTimer so the existing disconnect handler still cancels mid-cooldown. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor · 2026-06-03T13:14:16Z

@@ -1094,6 +3345,7 @@ <h3>Console Logs</h3>
            elements.promiseStatus.style.display = 'none';
            elements.sessionInfo.style.display = 'none';
            elements.publisherTokenInfo.style.display = 'none';
+            elements.subscribeSection.style.display = '';


Files API DOM refs removed

High Severity

The elements map no longer includes Files API nodes (ttlMode, uploadFile, fileRefId, etc.), but startup still registers listeners and updateUseRefAsImageBtnState still reads elements.fileRefId. That throws when the script reaches those lines or on disconnect, so the test page can fail before any realtime flow runs.

^{Reviewed by Cursor Bugbot for commit 93e4b33. Configure here.}

cursor · 2026-06-03T13:14:16Z

-      maxBitrate,
-      maxFramerate: REALTIME_CONFIG.livekit.defaultPublishFps,
+      ...defaultEncoding,
+      ...overrides?.videoEncoding,


VP9 defaults regressed

Medium Severity

getDefaultVideoPublishOptions now always enables simulcast and uses the H.264 default max bitrate. Connects with preferredVideoCodec: 'vp9' and no publishOptions no longer get VP9-specific simulcast-off and bitrate behavior from the previous implementation.

^{Reviewed by Cursor Bugbot for commit 93e4b33. Configure here.}

cursor · 2026-06-03T13:14:16Z

    room.on(RoomEvent.TrackSubscribed, (track: RemoteTrack, _pub, participant: RemoteParticipant) => {
      if (!participant.identity.startsWith(REALTIME_CONFIG.livekit.inferenceServerIdentityPrefix)) return;
-      if (track.kind !== Track.Kind.Video && track.kind !== Track.Kind.Audio) return;
+      if (track.kind !== Track.Kind.Video) return;


Remote audio tracks ignored

Medium Severity

Inference remote track handling now accepts only video: audio subscriptions are skipped, so onRemoteStream never includes audio tracks and attached elements cannot play remote sound even when the server publishes an audio track.

Additional Locations (1)

packages/sdk/src/realtime/subscribe-client.ts#L151-L154

^{Reviewed by Cursor Bugbot for commit 93e4b33. Configure here.}

cursor · 2026-06-03T13:14:16Z

+    return createWithTrackProcessor(sourceVideo);
  }
-  return createWithCanvas(sourceVideo, audioTracks, opts.fps);
+  return createWithCanvas(sourceVideo, opts.fps);


Mirror stream drops audio

Medium Severity

createMirroredStream no longer copies audio tracks into the mirrored output MediaStream; only the flipped video track is published. Inputs that include microphone audio lose that audio on the stream passed to LiveKit when mirroring is enabled.

^{Reviewed by Cursor Bugbot for commit 93e4b33. Configure here.}

15s was way too much. 2s is enough to let the remote clean up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor · 2026-06-03T13:16:33Z

+  const max = qpMaxForCodec(codec);
+  if (!max) return null;
+  return (value / max) * 100;
+}


Worker QP scale unclamped

Low Severity

normalizeQp in compute-worker.js scales raw QP to a percentage but does not clamp to 0–100, unlike the main page’s normalizeQp. Chart datasets built in the worker can show values above 100 for noisy samples.

^{Reviewed by Cursor Bugbot for commit 1bcc962. Configure here.}

…yParams - Move the SDK's livekit_eager_join default from after the caller queryParams spread to before it, so callers (the playground) can override the default. - Add a checkbox to the playground (default checked) that sets livekit_eager_join=true|false on every connect. Not in setChainUiLocked so it stays interactive during chain runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Per-run timings table gets a Session ID column between Profile and Elapsed, populated from run.sessionId. - Prompt cycler waits PROMPT_CYCLE_START_DELAY_MS (6s) after connect before issuing the first cycle prompt, with its own teardown timer threaded through stopPromptCycler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 13 total unresolved issues (including 11 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ca919f4. Configure here.}

cursor · 2026-06-04T14:23:14Z


      const queryParams = new URLSearchParams({
        ...(safariCodec ? { livekit_server_codec: safariCodec } : {}),
+        livekit_eager_join: "true",


Eager join ignores bundle flag

Medium Severity

connect always adds livekit_eager_join: "true" to the URL, while bundleInitialState can be false for the legacy two-step initial-state flow. Callers who only disable bundling still hit eager join unless they also override queryParams, which can desync server join behavior from client signaling.

^{Reviewed by Cursor Bugbot for commit ca919f4. Configure here.}

cursor · 2026-06-04T14:23:14Z

+        maxBitrate: publishOptions.videoEncoding?.maxBitrate,
+        maxFramerate: publishOptions.videoEncoding?.maxFramerate,
+      });
+      await this.room.localParticipant.publishTrack(track, publishOptions);


Local audio never published

Medium Severity

Publishing and mirroring now only handle video tracks: publishTracks iterates getVideoTracks() and no longer publishes audio, and createMirroredStream drops audio from the mirrored output. Apps that pass a MediaStream with a microphone track silently lose audio on connect.

Additional Locations (1)

packages/sdk/src/realtime/mirror-stream.ts#L39-L167

^{Reviewed by Cursor Bugbot for commit ca919f4. Configure here.}

cursor Bot reviewed May 20, 2026

View reviewed changes

Comment thread packages/sdk/src/utils/errors.ts

cursor Bot reviewed May 20, 2026

View reviewed changes

Comment thread packages/sdk/index.html

Comment thread packages/sdk/src/realtime/subscribe-client.ts

testing logic

42ea6a2

VerioN1 force-pushed the feat/realtime-errors-verbosity branch from 1ea50a6 to 42ea6a2 Compare May 20, 2026 19:45

cursor Bot reviewed May 20, 2026

View reviewed changes

Comment thread packages/sdk/index.html

VerioN1 added 4 commits May 21, 2026 12:43

increase max bitrate

874fedf

added new instrumentation

e71ec42

fixed bugs

8f26964

updated playground metrics to fit better profiles judgment

48a31d1

cursor Bot reviewed May 23, 2026

View reviewed changes

Comment thread packages/sdk/src/realtime/subscribe-client.ts

Comment thread packages/sdk/index.html Outdated

VerioN1 and others added 6 commits May 23, 2026 21:11

new way to profile

9a3d59d

changed profiles

261cbd3

added numbers and stuff

66a0fea

wip: realtime debugging instrumentation and QR latency fix

960f53b

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/realtime-errors-…

4aa705c

…verbosity

cursor Bot reviewed Jun 1, 2026

View reviewed changes

VerioN1 and others added 2 commits June 1, 2026 16:02

cursor Bot reviewed Jun 3, 2026

View reviewed changes

VerioN1 and others added 2 commits June 3, 2026 11:57

cursor Bot reviewed Jun 3, 2026

View reviewed changes

VerioN1 and others added 2 commits June 3, 2026 16:12

cursor Bot reviewed Jun 3, 2026

View reviewed changes

fix(playground): cut chain teardown→connect cooldown to 2s

1bcc962

15s was way too much. 2s is enough to let the remote clean up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cursor Bot reviewed Jun 3, 2026

View reviewed changes

VerioN1 and others added 4 commits June 3, 2026 23:29

chore(realtime): drop deprecated livekit_early_room_info query param

5503cb9

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

made explicit flag for enhence

5f161ce

cursor Bot reviewed Jun 4, 2026

View reviewed changes

Conversation

VerioN1 commented May 20, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot Jun 1, 2026

Choose a reason for hiding this comment

Duplicate receiver stats enrichment logic across files

Uh oh!

Uh oh!

cursor Bot Jun 1, 2026

Choose a reason for hiding this comment

VP9 codec loses automatic simulcast and bitrate defaults

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Remote inference audio dropped

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Local audio never published

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Chain recorder never restarts

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Local audio never published

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Files API DOM refs removed

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

VP9 defaults regressed

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Remote audio tracks ignored

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Mirror stream drops audio

Uh oh!

cursor Bot Jun 3, 2026

Choose a reason for hiding this comment

Worker QP scale unclamped

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 4, 2026

Choose a reason for hiding this comment

Eager join ignores bundle flag

Uh oh!

cursor Bot Jun 4, 2026

Choose a reason for hiding this comment

Local audio never published

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

VerioN1 commented May 20, 2026 •

edited by cursor Bot

Loading

pkg-pr-new Bot commented May 20, 2026 •

edited

Loading