Add Dekaf Kafka client integration and Schema Registry hosting support

[thomhurst]

Related to an existing integration?

No

Existing integration

New Dekaf client integration, composing with the existing Aspire.Hosting.Kafka broker resource.

Overview

Add CommunityToolkit.Aspire.Kafka.Dekaf, a .NET 10 client integration for the managed Dekaf Kafka client. The Aspire maintainers recommended moving this work here: microsoft/aspire#18828 (comment). The original implementation is in microsoft/aspire#20290.

The client integration supports:

• Ordinary and keyed producer, consumer, and admin client registrations.
• Named configuration, native builder callbacks with DI access, host startup initialization, and disposal.
• Native Dekaf health checks, logging, OpenTelemetry tracing, and metrics.
• Ordinary and keyed schema registry clients, authentication and TLS configuration, failover, caching, and read-only health checks.
• JSON Schema serialization, with optional Avro and Protobuf packages through Dekaf's existing builder helpers.

Also propose a hosting helper for a local Schema Registry container connected to an existing Aspire Kafka resource, with connection references, health checks, startup dependencies, and an example AppHost. The existing Kafka hosting resource remains the broker implementation. The hosting API and package name can follow the Toolkit maintainers' preferred conventions.

Usage example

Client application:

builder.AddDekafKafkaProducer<string, string>("messaging");
builder.AddDekafKafkaConsumer<string, string>("messaging", consumer => consumer
    .WithGroupId("orders-service")
    .SubscribeTo("orders"));
builder.AddDekafKafkaAdminClient("messaging");
builder.AddDekafSchemaRegistryClient("schema-registry");

Proposed AppHost usage, subject to API review:

var kafka = builder.AddKafka("messaging");
var registry = builder.AddKafkaSchemaRegistry("schema-registry", kafka);

builder.AddProject<Projects.KafkaWorker>("worker")
    .WithReference(kafka)
    .WithReference(registry)
    .WaitFor(kafka)
    .WaitFor(registry);

Breaking change?

No

Alternatives

Applications can manually register Dekaf and its schema registry clients and configure their lifecycle, health checks, and telemetry. They can also configure a Schema Registry container with AddContainer. The existing Confluent.Kafka integration is another option, but it uses librdkafka rather than a managed Kafka implementation.

Additional context

The original client implementation has 154 passing tests, including real Kafka container round trips, health checks, telemetry, and JSON/Avro/Protobuf serialization. Schema registry scenarios use the real Dekaf client with an in-memory HTTP transport. I am adapting the implementation and tests to the Toolkit conventions and adding hosting coverage and examples.

The client package requires .NET 10 because Dekaf's DI and health-check packages require it. Configuration binding uses reflection; this integration does not claim Native AOT support.

Would the Toolkit maintainers accept this scope? I can submit the port as a pull request after this proposal is approved.

Help us help you

Yes, I'd like to be assigned to work on this item.