[All] Correct SDK documentation and examples by teodordelibasic-db · Pull Request #729 · databricks/zerobus-sdk

teodordelibasic-db · 2026-08-12T19:17:05Z

What changes are proposed in this pull request?

Correct README snippets, checked-in examples, and public API doc examples across Rust, Python, Java, Go, pure Go, TypeScript, C++, and .NET so they compile and demonstrate the current APIs. The changes fix API names and arguments, stream format selection, dependency setup, recovery and callback semantics, resource cleanup, and queue-then-flush ingestion. They also add the generated-message fixture needed to run the .NET Protobuf example.

The documentation had drifted as the public APIs evolved, leaving examples that either failed to compile or demonstrated incorrect recovery, durability, and throughput patterns. Keeping these examples aligned with the supported APIs prevents users from copying invalid or unnecessarily slow client code.

This PR documents current behavior. The following implementation issues are deferred:

Python _zerobus_core.pyi still has wrong get_unacked_records() return types, create_stream argument order, async signatures, and is missing row-stream get_unacked_batches().
Python stream_id and get_state() still return placeholder values (stream-placeholder-id and constant OPENED).
TypeScript custom header callbacks are invoked once and snapshotted; they do not refresh on recovery.
Java JNI logs AckCallback exceptions but does not clear the pending Java exception, which can poison later callback operations.
Java setRecoveryBackoffMs does not reject negative values.
Java NativeTestHelper remains a public production class despite exposing test-only JNI symbols.
Python ingest_record_nowait / ingest_records_nowait are documented as unsafe but do not emit a deprecation warning.
A flush timeout can leave a Pure-Go or C++ stream active, so unacked retrieval fails until the stream is actually closed. Recovering that case needs an implementation change.
go/version.go remains 1.4.0. The user-agent string bumps with the next Go release, not this docs PR.

How is this tested?

All runnable examples and executable documentation snippets were compiled or type-checked. Representative JSON, Protobuf, Arrow, batch, recovery, and custom-header paths were also exercised end to end.

The affected SDK formatters, linters, unit tests, example builds, and doc tests pass. The .NET suite passes 47 unit and 54 integration tests. TypeScript build, tests, and example type-check pass; its all-features Clippy check still reports three pre-existing warnings unrelated to these documentation changes. The C++ Arrow example was source-reviewed but not linked because Apache Arrow C++ was unavailable; the other C++ examples built and their JSON batch and custom-header paths were exercised.

nikolaobradovic-db · 2026-08-13T10:48:23Z

            # ack = stream.ingest_record(record_dict)  # Deprecated
            # offset = ack.wait_for_ack()  # Extra step needed

-            end_time = time.time()


ingest_record_nowait() and ingest_records_nowait() spawn detached tasks and discard enqueue errors. flush() can complete before those tasks allocate offsets, so this example may report durability while losing submissions.

@teodordelibasic-db @elenagaljak-db I think we should remove nowait from examples and put it on a deprecation path.

Removed nowait from the examples. Docs now rank ingest_records_offset() + one flush() as the bulk path and call nowait unsafe (detached tasks, not synchronized with flush). Putting the APIs on a real deprecation path (DeprecationWarning) is a follow-up, not this docs PR.

nikolaobradovic-db · 2026-08-13T10:50:24Z

-    unacked = stream.get_unacked_records()  # Returns List[bytes]
+    stream.close()
+except ZerobusException as e:
+    unacked = list(stream.get_unacked_records())


This catch also handles immediate enqueue errors that leave the stream active. Calling get_unacked_records() or recreate_stream() then fails and masks the original error.

Please separate enqueue failures from terminal failures, close before recovery, and protect the replacement stream with try/finally.

Updated. The snippet now distinguishes that enqueue failures can leave the stream active, closes before get_unacked_records()/recreate_stream(), re-raises the original ingest error if unacked retrieval fails, and wraps the replacement stream in try/finally.

nikolaobradovic-db · 2026-08-13T10:51:12Z

-when records are acknowledged by the server or encounter errors.
+Subclass this in Python to create custom callbacks that are invoked once per
+logical ingest submission. A batch submission produces one callback, not one
+callback per record in the batch.


Pre-queue validation, size, type, and closed-stream failures do not generate callbacks, so this does not apply to every attempted submission.
Please qualify it as one callback per successfully queued submission that later acknowledges or fails.

Qualified: one callback per successfully queued logical ingest submission that later acknowledges or fails. Pre-queue validation / size / type / closed-stream failures raise immediately and do not generate a callback.

nikolaobradovic-db · 2026-08-13T10:52:02Z

 ```python
-unacked_batches = stream.get_unacked_batches()  # Returns List[List[bytes]]


This ranks per-record ingest_record_nowait() highest but omits the batch APIs. Batch ingestion amortizes the Python→Rust crossing and is the preferred hot path. Additionally, detached nowait submissions are not safely synchronized with flush.

Please include the batch APIs and recommend ingest_records_offset() plus one flush for reliable bulk ingestion.

Table at 385

Table now ranks ingest_records_offset() + one flush() first. nowait is listed as unsafe and is not recommended for bulk ingestion.

nikolaobradovic-db · 2026-08-13T10:53:09Z

 * );
 * ```
 *
- * **How to use custom authentication (PAT, etc.):**


The exported interface requires async getHeaders(): Promise<...>, while this example and createStream() require synchronous getHeadersCallback. A class implementing the public interface cannot be passed to the documented API.

Please expose and document one provider type matching createStream().

The exported HeadersProvider is now the shape createStream() actually accepts: getHeadersCallback returning header tuples synchronously. The docs no longer show an async getHeaders() class.

nikolaobradovic-db · 2026-08-13T10:55:02Z

    ///
-    /// * `stream` - The failed or closed stream to recreate
+    /// * `stream` - The terminally failed stream to recreate. The TypeScript wrapper
+    ///   must not have been closed because `close()` releases its native handle.


LLM find:
TypeScript custom credentials cannot refresh
Affected: typescript/src/lib.rs:1024-1046

The callback runs once during stream creation and its result is stored in StaticHeadersProvider. Long-running streams therefore reuse the original token during recovery, so rotating or expiring custom credentials eventually break reconnection.

Keep the threadsafe callback in a HeadersProvider adapter and invoke it whenever the Rust core requests fresh headers. Support invalidation if required by the core contract.

Agreed, this is a real implementation bug. Out of scope for this docs PR. The headers-provider docs now say the callback is invoked once at stream creation and that token refresh is not wired through. Follow-up will keep the callback and invoke it whenever the core requests headers.

nikolaobradovic-db · 2026-08-13T10:56:02Z

-    // Optional: Inspect what needs recovery (must be called on closed stream)
+    // Optional: Inspect what needs recovery after a terminal stream failure.
    const unackedBatches = await stream.getUnackedBatches();
    console.log(`Batches to recover: ${unackedBatches.length}`);


getUnackedBatches() runs before cleanup protection, and this catch can also handle non-terminal failures. An inspection failure can leak the original stream and mask the ingestion error.

Please use an outer finally, recover only from confirmed terminal failures, and close every replacement stream in a nested finally.

Updated: outer finally always closes the original stream, recovery only runs after a failure, and the replacement stream is closed in a nested finally.

nikolaobradovic-db · 2026-08-13T10:57:00Z

+            AirQuality.create({ deviceName: 'sensor-002', temp: 23, humidity: 67 }),
+            AirQuality.create({ deviceName: 'sensor-003', temp: 24, humidity: 69 })


typescript/examples/proto/batch.ts:113-116

Waiting here before queueing the next batch serializes the example into one server round trip per batch.

Please queue all demonstration batches first, then call flush() once or wait only on the final offset. The JSON batch example has the same issue.

Proto and JSON batch examples now queue every demonstration batch, then flush() once.

nikolaobradovic-db · 2026-08-13T10:58:32Z


 ### Code Highlights

 **Offset-based API (Recommended):**


typescript/examples/json/README.md:65-71

This first “Recommended” pattern immediately waits after one ingest. Although valid for strict low-volume confirmation, it is not the default pattern readers should copy.

Please show loop-then-flush() first and move this into a clearly labeled low-volume section. The Protobuf README has the same ordering issue.

JSON and Protobuf example READMEs now lead with loop-then-flush(). Immediate waitForOffset after a single ingest is labeled as a low-volume confirmation pattern, not the default.

nikolaobradovic-db · 2026-08-13T10:59:15Z

+
+main().catch((error) => {
+  console.error('Fatal error:', error);
+});


This logs a fatal error but leaves the process exit status as zero, so copied CLI or CI code can report success after ingestion fails.

Please set process.exitCode = 1 or rethrow.

main().catch now sets process.exitCode = 1.

nikolaobradovic-db · 2026-08-13T11:00:29Z

I don't see any changes to GO SDK while it was mentioned in PR desc. Maybe you audited it and there were no stale docs/examples?

nikolaobradovic-db · 2026-08-13T11:01:31Z

    >>>
    >>> # New optimized API
-    >>> offset = stream.ingest_record_offset(b"data")
+    >>> offset = stream.ingest_record_offset('{"value": "data"}')


python/zerobus/init.py:13

Callbacks represent logical submissions and can correspond to batches, so “Record acknowledged” is too specific.

Please use “Submission acknowledged” for consistency.

Changed to “Submission acknowledged”.

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>

elenagaljak-db · 2026-08-14T11:16:59Z


 ```bash
-dotnet add package Databricks.Zerobus.Ingest.Sdk
+dotnet add package Databricks.Zerobus --version 0.5.1


Why 0.5.1? Databricks.Zerobus is incorrect

Fixed. The package id is Databricks.Zerobus.Ingest.Sdk. Left it unpinned under "NuGet (when published)" since it is not on NuGet yet.

elenagaljak-db · 2026-08-14T11:21:47Z

+// IMPORTANT: Call this on a failed stream before Close(). Close() nils the
+// handle and frees native resources, so a later GetUnackedRecords() call fails.
 //
 // Use this method to:
 //   - Retrieve unacknowledged records after stream failure for retry logic
 //   - Check which records weren't durably written after Close() fails
 //   - Implement custom retry strategies after stream errors
 //


The important note and the second bullet contradict

Good catch. A flush timeout can leave the stream active, so that bullet was wrong. It now says to inspect queued-but-unacked payloads before Close(), matching the IMPORTANT note.

elenagaljak-db · 2026-08-14T11:49:23Z

+- Documented that `get_unacked_records()` and `recreate_stream()` require a closed
+  stream, and that enqueue failures must be closed before recovery.


This contradicts readme, which says that "An enqueue failure leaves the stream active". The failed record was never queued, so there is nothing to recover?

Agreed. An enqueue failure never queued that payload, so there is nothing to recover for that call. The changelog now matches the README: close first only if you need to inspect records that were already accepted.

zlata-stefanovic-db · 2026-08-14T12:28:58Z


 ## Installation

-### NuGet (when published)


Is it published yet? Maybe better to leave it like this?

Not published yet. Restored "NuGet (when published)" and dropped the version pin.

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>

teodordelibasic-db changed the title ~~Correct SDK documentation and examples~~ [All] Correct SDK documentation and examples Aug 13, 2026

teodordelibasic-db marked this pull request as ready for review August 13, 2026 06:50

teodordelibasic-db force-pushed the codex/audit-sdk-snippets branch from 0e03af6 to 9a004c1 Compare August 13, 2026 07:19

teodordelibasic-db requested a review from a team August 13, 2026 07:23

teodordelibasic-db self-assigned this Aug 13, 2026

teodordelibasic-db requested a review from nikolaobradovic-db August 13, 2026 07:23

nikolaobradovic-db reviewed Aug 13, 2026

View reviewed changes

teodordelibasic-db added 12 commits August 14, 2026 09:16

Initial commit

b7a5594

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>

Address Python review comments

ded6311

Address TypeScript review comments

800b423

Fix Go documentation

e027f5e

Fix Java documentation

bdc21c2

Fix remaining SDK docs

c425941

Fix leftover Java docs

2fb6296

Fix leftover Rust docs

d8dbfc3

Fix TypeScript inflight default

7da7400

Fix leftover Go docs

d4641f7

Fix C++ Arrow schema docs

2875180

Fix Python recovery snippet

f621526

zlata-stefanovic-db self-requested a review August 14, 2026 10:59

teodordelibasic-db requested a review from a team August 14, 2026 11:04

Fix batch example wording

dfbbbb9

elenagaljak-db reviewed Aug 14, 2026

View reviewed changes

teodordelibasic-db added 5 commits August 14, 2026 12:21

Fix Go durability snippets

2f07b45

Fix Pure-Go example recovery

f7ba22c

Fix Java example docs

e9b96da

Fix Rust example wording

2f46a40

Fix TypeScript example docs

94d0898

zlata-stefanovic-db reviewed Aug 14, 2026

View reviewed changes

Address comments

e1f0e61

teodordelibasic-db requested review from elenagaljak-db, nikolaobradovic-db and zlata-stefanovic-db August 14, 2026 12:48

zlata-stefanovic-db previously approved these changes Aug 14, 2026

View reviewed changes

elenagaljak-db previously approved these changes Aug 14, 2026

View reviewed changes

nikolaobradovic-db previously approved these changes Aug 14, 2026

View reviewed changes

Final pass

786638c

Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>

teodordelibasic-db dismissed stale reviews from nikolaobradovic-db, elenagaljak-db, and zlata-stefanovic-db via 786638c August 14, 2026 14:19

nikolaobradovic-db approved these changes Aug 14, 2026

View reviewed changes

teodordelibasic-db enabled auto-merge August 14, 2026 14:24

teodordelibasic-db disabled auto-merge August 14, 2026 14:54

teodordelibasic-db added this pull request to the merge queue Aug 14, 2026

Merged via the queue into main with commit 1444541 Aug 14, 2026
169 of 170 checks passed

teodordelibasic-db deleted the codex/audit-sdk-snippets branch August 14, 2026 15:31

		```python
		unacked_batches = stream.get_unacked_batches() # Returns List[List[bytes]]

		AirQuality.create({ deviceName: 'sensor-002', temp: 23, humidity: 67 }),
		AirQuality.create({ deviceName: 'sensor-003', temp: 24, humidity: 69 })

		- Documented that `get_unacked_records()` and `recreate_stream()` require a closed
		stream, and that enqueue failures must be closed before recovery.

Uh oh!

Conversation

teodordelibasic-db commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How is this tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikolaobradovic-db Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikolaobradovic-db Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikolaobradovic-db commented Aug 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elenagaljak-db Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

teodordelibasic-db commented Aug 12, 2026 •

edited

Loading

nikolaobradovic-db Aug 13, 2026 •

edited

Loading

nikolaobradovic-db Aug 13, 2026 •

edited

Loading

elenagaljak-db Aug 14, 2026 •

edited

Loading