debezium/dbz#2101 Add pipeline deployment lifecycle with Ansible-based container management by d1vyanshu-kumar · Pull Request #493 · debezium/debezium-platform

d1vyanshu-kumar · 2026-08-13T17:52:15Z

Description

Implements the full pipeline deployment lifecycle for host-mode environments using Ansible ad-hoc commands over SSH instead of the future Host Agent. This allows pipelines to be deployed, monitored, stopped, started, and undeployed on remote servers without blocking on the Host Agent implementation.

Changes

Pipeline Configuration Mapping

Add HostPipelineMapper to convert pipeline objects into flat application.properties format with source, sink, offset storage, schema history, transforms, and signal/notification defaults
Apply platform-required defaults (signal channels, notification channels) after user config to prevent silent overrides, consistent with the operator PipelineMapper
Map debezium.format.key and debezium.format.value from JsonConverter to Json to match Debezium Server's expected format enum

Deployment Service & Host Selection

Add HostDeploymentService with transactional deployment creation, pessimistic-locked host selection, and port allocation
Add DeployStrategy interface with RoundRobinStrategy (least-loaded, deterministic tie-breaking by host ID)
Add deployedAt timestamp to HostDeploymentEntity for deployment age tracking
Add findReadyHosts() query for container cleanup when deployment records are removed by cascade

Pipeline Controller

Add HostPipelineController orchestrating deploy, stop, start, and undeploy operations via AnsibleCommandRunner
Config files are copied to remote hosts using Ansible copy module; containers are managed using Ansible shell module running Docker commands
Clean up existing deployment records and containers before redeployment to prevent UNIQUE constraint violations on pipeline_id
Handle undeploy when the deployment DB record was already removed by ON DELETE CASCADE by falling back to docker rm -f on all ready hosts
Add null guard on Arc.container() in CDI request context activation to support plain JUnit test execution

Status Monitoring

Add HostDeploymentStatusPoller that checks container health every 30 seconds via docker inspect
Implement a 5-minute grace period after deployment to allow Docker image pulls on fresh servers before marking containers as FAILED
Detect config drift by comparing SHA-256 hash of deployed config against the live file on the remote host
Reset deployedAt timestamp when transitioning back to DEPLOYING (restart scenario) to prevent false FAILED marking
Add ansible_become_timeout (60s) to prevent Ansible become hangs on slow hosts

Ansible Command Runner

Add AnsibleCommandRunner as a reusable wrapper for executing Ansible ad-hoc commands with structured result handling via sealed CommandResult interface

Pipeline Deletion

Add ON DELETE CASCADE to FK_host_deployment_pipeline foreign key so deleting a pipeline from the UI automatically removes the associated host_deployment row without a constraint violation
Add @OnDelete(action = CASCADE) on HostDeploymentEntity.pipeline to keep the JPA model consistent with the database schema

Dev Profile Configuration

Override JDBC offset and schema history connection URLs under the %dev profile so that Debezium Server containers on remote hosts connect to the correct PostgreSQL instance instead of localhost

Testing

HostPipelineMapperTest — 17 unit tests covering source/sink/transform mapping, offset/schema history config, signal defaults ordering, format mapping, and SHA-256 hashing
HostPipelineControllerTest — 17 unit tests for deploy/stop/start/undeploy orchestration, idempotent redeployment cleanup, and cascade undeploy fallback
HostDeploymentStatusPollerTest — 12 unit tests covering all state transitions, grace period logic, config drift detection, and become timeout configuration
HostDeploymentServiceIT — 15 integration tests against real PostgreSQL for deployment CRUD, status transitions, deployedAt lifecycle, pessimistic locking, ready host queries, and cascade deletion
AnsibleCommandRunnerTest — unit tests for command construction and result parsing
RoundRobinStrategyTest — unit tests for least-loaded selection and tie-breaking

PR Checklist

I have read the contribution guidelines and the governance document on PR expectations.
Minimal changes to code not directly related to your change (e.g. no unnecessary formatting changes or refactoring to existing code)
One feature/change per PR unless tightly coupled
Do a rebase on upstream main

…d container management Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

…format mapping, and offset JDBC URLs Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

…ployment Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

d1vyanshu-kumar · 2026-08-13T17:54:35Z

d1vyanshu-kumar · 2026-08-13T18:23:27Z

Hi @mfvitale, @kmos, could you take a look at this when you get a chance?

One thing I wanted to mention, during our call we discussed using file-based offset/schema storage for Host mode, but after working through the implementation I went with the same JDBC-based approach (JdbcOffsetBackingStore / JdbcSchemaHistory) that we already use in Operator mode. The main reason is that with file-based storage, offsets live inside the container filesystem, so if a container gets recreated during redeploy or moved to another host, those offsets are lost and Debezium would have to re-read the WAL from scratch. With JDBC storage the containers stay fully stateless - the offset bookmark lives in a database table, so redeploys just resume from where they left off.

It also kept things simpler on the implementation side since we reuse the existing PipelineConfigGroup config paths without needing extra volume mount logic over SSH. The %dev profile points to a Postgres on the target host for local testing, and in production it resolves through the same OFFSET_JDBC_URL env var as Operator mode.

That said, if you'd prefer the file-based approach instead, I'm happy to rework it, just let me know!

mfvitale · 2026-08-13T19:14:14Z

That said, if you'd prefer the file-based approach instead, I'm happy to rework it, just let me know!

We should go with file based offset mounting as a volume on the target host.

kmos · 2026-08-13T22:17:17Z

That said, if you'd prefer the file-based approach instead, I'm happy to rework it, just let me know!

We should go with file based offset mounting as a volume on the target host.

I agree. The target host shouldn't change like a k8s pod or volume. I think it's fine for now

mfvitale · 2026-08-14T13:04:06Z

    server_port integer not null,
    deployment_status varchar(255) not null,
    config_hash varchar(255) not null,
+    deployed_at timestamp(6) with time zone not null,
    primary key (id)
 );

 alter table if exists host_deployment
    add constraint FK_host_deployment_pipeline
    foreign key (pipeline_id)
-    references pipeline;
+    references pipeline
+    on delete cascade;

 alter table if exists host_deployment
    add constraint FK_host_deployment_host_status


You need to create another change file. You cannot modify an old one.

mfvitale · 2026-08-14T13:08:47Z

+     * @return the selected host entity
+     * @throws io.debezium.DebeziumException if no hosts are available
+     */
+    HostStatusEntity select(List<HostStatusEntity> readyHosts, LoadCounter loadCounter);


LoadCounter is specific for the round-robin load based implementation. Should not be in the interface

mfvitale · 2026-08-14T13:15:17Z

+ * used in config drift detection.
+ */
+@ApplicationScoped
+public class HostPipelineMapper {


I think that you can simply this a lot using the io.debezium.platform.environment.operator.PipelineMapper.map(pipelineFlat) and the call the asConfiguration on the DebeziumServer returned from the map method.

This is effectively what the operator does to create the config map with the application.properties

mfvitale · 2026-08-14T13:21:05Z

+            String sshAlias = allocation.hostStatus().getSshAlias();
+            int port = allocation.allocatedPort();
+
+            String containerName = hostConfig.containerNamePrefix() + pipelineId;


We should use the pipeline the name ad we do for the k8s

mfvitale · 2026-08-14T13:27:31Z

+                    containerName, hostConfig.debeziumServerImage(),
+                    port, mappedConfig.configHash());
+
+            CommandResult mkdirResult = ansibleRunner.createDirectory(sshAlias, configDir);


I suppose that all commands below this point are Ansible related. Since we will replace this part with the host agent, I could see an interface with the start, stop, restart and in this PR the implementation is done via Ansible and then the other one that will call the API?

Naros · 2026-08-15T13:08:59Z

+ * this class will be swapped for HTTP REST calls — all other logic
+ * (host selection, port allocation, status tracking) stays unchanged.
+ */
+@Dependent


Should this be @ApplicationScoped?

In CDI, for a @Dependent scoped bean, the container will create a new instance specifically for each observer method invocation, then destroy it afterward. The instance injected into HostEnvironmentController, the one holding the executor, is never notified when the shutdown event is observed.

Naros · 2026-08-15T13:13:56Z

+     * @return the command result
+     */
+    public CommandResult copyContent(String sshAlias, String content, String destPath) {
+        String copyArgs = "content='" + content + "' dest=" + destPath + " mode=0644";


The content argument is being wrapped here in single quotes; however, if any user-provided config value, such as a password or other connector property, contains a single quote, the Ansible copy module argument parsing will break.

We should either add escaping logic to guard against this, use Ansible's stdin, or write to a temp file and use src= rather than content=.

I've switched from content='...' to writing a local temp file and using src= instead. The config content never appears in the Ansible command string now, so single quotes (or any other characters) in user-provided values can't break argument parsing.

Naros · 2026-08-15T13:15:58Z

+            # Must point to a Postgres reachable from the remote Docker container.
+            # Override via HOST_OFFSET_JDBC_URL env var when the VM IP changes.
+            connection:
+              url: ${HOST_OFFSET_JDBC_URL:jdbc:postgresql://192.168.252.23:5432/testdb?loggerLevel=OFF}


It looks like the IP here might be specific to your environment? Same for HOST_SCHEMA_HISTORY_JDBC_URL below.

@Naros This part must be removed since we want to use file based offset on the host as default offset/schema storage

@mfvitale, @kmos Agreed; switched to file-based offset and schema history storage. The host's data directory (/opt/debezium/data/) is bind-mounted into the container at /debezium/data. Offset data survives container restarts. Also removed the hardcoded 192.168.252.23 JDBC URLs from application.yml.

Naros · 2026-08-15T13:20:02Z

+        controller.undeploy(999L);
+
+        // Give the executor time to process
+        Thread.sleep(500);


Could we not use this instead?

assertThat(latch.await(ASYNC_WAIT_SECONDS, TimeUnit.SECONDS)).isTrue();

Thanks! Replaced all Thread.sleep(500) calls with CountDownLatch.await(ASYNC_WAIT_SECONDS, TimeUnit.SECONDS) , same pattern used in HostProvisioningServiceTest. No more timing-dependent flakiness.

…cycle Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

d1vyanshu-kumar · 2026-08-18T14:06:41Z

Hey @mfvitale @Naros @kmos - pushed a commit addressing all the review feedback here is summary:

Migration immutability - Reverted V3.6.0 to its original heartbeat-only content and moved all host deployment DDL into a new V3.7.0.2 migration file. Flyway checksum stays intact for existing installations.
LoadCounter out of DeployStrategy - Removed it from the interface. The deployment count query now lives inside RoundRobinStrategy via direct EntityManager injection, so future strategies don't need to deal with load counting at all.
HostPipelineMapper delegation - Refactored to delegate to
PipelineMapper.map().asConfiguration().getAsMapSimple() and only override offset/schema storage from JDBC to file-based for host mode.
Container naming - Now uses pipeline.getName() instead of a numeric ID prefix, matching the K8s convention. docker ps on the host shows the actual pipeline name.
HostContainerRuntime interface - Extracted deploy/undeploy/stop/start/logs into an interface. AnsibleContainerRuntime implements it. The controller depends only on the interface, so swapping to the Host Agent later won't require controller changes.
@ Dependent → @ApplicationScoped - Fixed the CDI scope on HostPipelineController so the shutdown observer fires on the real singleton that holds the running thread pool.
copyContent injection fix - Switched from content='...' to writing a local temp file and using src= instead. Config content never appears in the Ansible command string now.
File-based offset/schema storage - Host data directory is bind-mounted at /debezium/data. Removed the hardcoded JDBC URLs from application.yml. Verified offsets.dat is created on the host after the first snapshot.
Thread.sleep → CountDownLatch Replaced all Thread.sleep(500) in async tests with CountDownLatch.await() for deterministic test behavior.

Also fixed a container permission issue that came up during live testing - the Debezium Server container couldn't write to the root-owned data directory. Per @mfvitale's suggestion, added --user $(id -u):$(id -g) to the docker run command so the container process matches the host directory owner. Tested and verified that offsets.dat is now created successfully.

As we discussed, I've opened a separate issue for the shared group provisioning improvement as a follow-up: dbz#2451

I have verified everything end-to-end on a live host server, and it is working smoothly. Please feel free to pull the latest changes and test it on your end, and let me know if anything else needs attention!

kmos

I have done a round of review, my suggestion is to take a look to the following article and internal links and apply a refactoring.

kmos · 2026-08-18T15:13:07Z

+     * @param hostStatus    the locked and selected host entity
+     * @param allocatedPort the unique port assigned for this deployment
+     */
+    public record HostAllocation(HostStatusEntity hostStatus, int allocatedPort) {


move this record outside this class

kmos · 2026-08-18T15:14:10Z

+    @Transactional(REQUIRES_NEW)
+    public Optional<HostDeploymentEntity> findByPipelineId(Long pipelineId) {
+        return em.createQuery(
+                "SELECT d FROM host_deployment d WHERE d.pipeline.id = :pipelineId",


extract the query

kmos · 2026-08-18T15:15:42Z

+    @Transactional(REQUIRES_NEW)
+    public List<HostDeploymentEntity> findByStatus(DeploymentStatus status) {
+        return em.createQuery(
+                "SELECT d FROM host_deployment d WHERE d.deploymentStatus = :status",


extract the query

kmos · 2026-08-18T15:18:03Z

+     * @return the persisted deployment entity
+     */
+    @Transactional(REQUIRES_NEW)
+    public HostDeploymentEntity createDeployment(Long pipelineId, Long hostStatusId,


is it really necessary to return an HostDeploymentEntity? it could be void?

kmos · 2026-08-18T15:18:40Z

+     */
+    @Transactional(REQUIRES_NEW)
+    public HostDeploymentEntity createDeployment(Long pipelineId, Long hostStatusId,
+                                                 String containerName, String imageVersion,


reduce the number of parameter creating a new record class. Try to follow the rule max 3 params

kmos · 2026-08-18T15:34:59Z

+
+            if (!finished) {
+                process.destroyForcibly();
+                String timeoutMessage = "Ansible ad-hoc command timed out after "


kmos · 2026-08-18T15:38:41Z

+            return new CommandResult.Failure(output);
+        }
+        catch (IOException e) {
+            logger.errorv(e, "Failed to start Ansible ad-hoc process");


same as the previous comment related to stacktrace

kmos · 2026-08-18T15:48:08Z

+    }
+
+    @Override
+    public HostStatusEntity select(List<HostStatusEntity> readyHosts) {


Why using the entity? You could use a domain class

kmos · 2026-08-18T15:54:39Z

+ * @see AnsibleHostProvisioner
+ */
+@ApplicationScoped
+public class AnsibleCommandRunner {


you could refactor using command pattern: https://refactoring.guru/design-patterns/command

kmos · 2026-08-18T15:55:42Z

+        String configPath = configDir + PATH_SEPARATOR + CONFIG_FILE_NAME;
+        String dataDir = hostConfig.dataBasePath() + PATH_SEPARATOR + containerName;
+
+        // 1. Create config directory


Instead of using comment in the code, express your intentions: extract classes that describe the business

mfvitale · 2026-08-19T08:55:50Z

-
-- Host-based pipeline deployment tables
-- Purely additive: no changes to existing tables, sequences, indexes, or constraints.
-
-create sequence host_status_SEQ start with 1 increment by 50;
-
-create sequence host_deployment_SEQ start with 1 increment by 50;


@d1vyanshu-kumar Existing file should not be modified. If you need to add column to an already existing table you just need to crate the alter DDL statement on a new migration file.

@mfvitale ok, I see it now. I moved the host tables out to a new file, but deleting them from V3.6.0 still counts as modifying it, so that's why this popped up again. My bad, makes sense now.

I've reverted V3.6.0 and V3.7.0.2 back to their original state and added the deployed_at column + the ON DELETE CASCADE change through a new migration file instead, so neither of the existing files get touched. Let me know if that looks right. 🙂

…eployment Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

d1vyanshu-kumar added 3 commits August 13, 2026 23:12

debezium/dbz#2101 Add pipeline deployment lifecycle with Ansible-base…

df3376c

…d container management Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

debezium/dbz#2101 Fix host-mode container mounting, become timeouts, …

74cb19b

…format mapping, and offset JDBC URLs Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

debezium/dbz#2101 Fix host-mode pipeline deletion and idempotent rede…

93ec73d

…ployment Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

kmos requested review from kmos and mfvitale August 13, 2026 22:14

mfvitale reviewed Aug 14, 2026

View reviewed changes

Naros reviewed Aug 15, 2026

View reviewed changes

d1vyanshu-kumar mentioned this pull request Aug 18, 2026

Provision a shared OS group during host setup for secure container volume access debezium/dbz#2451

Open

debezium/dbz#2101 Address review feedback for host-mode pipeline life…

7e0fe33

…cycle Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

d1vyanshu-kumar force-pushed the debezium/dbz#2101 branch from f24260c to 7e0fe33 Compare August 18, 2026 13:56

d1vyanshu-kumar requested review from Naros and mfvitale August 18, 2026 14:18

kmos reviewed Aug 18, 2026

View reviewed changes

mfvitale reviewed Aug 19, 2026

View reviewed changes

debezium/dbz#2101 Add deployed_at column and cascade delete on host_d…

1b81e7f

…eployment Signed-off-by: divyanshu_Kumar <154233802+d1vyanshu-kumar@users.noreply.github.com>

Uh oh!

Uh oh!

Conversation

d1vyanshu-kumar commented Aug 13, 2026

Description

Changes

Testing

PR Checklist

Uh oh!

d1vyanshu-kumar commented Aug 13, 2026

Uh oh!

d1vyanshu-kumar commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mfvitale commented Aug 13, 2026

Uh oh!

kmos commented Aug 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfvitale Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d1vyanshu-kumar Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfvitale Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d1vyanshu-kumar commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kmos left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

d1vyanshu-kumar commented Aug 13, 2026 •

edited

Loading

mfvitale Aug 14, 2026 •

edited

Loading

d1vyanshu-kumar Aug 18, 2026 •

edited

Loading

mfvitale Aug 17, 2026 •

edited

Loading

d1vyanshu-kumar commented Aug 18, 2026 •

edited

Loading

kmos left a comment •

edited

Loading