Skip to content

DLPX-99529 Masking : Create new branch and upgrade to version 2.17 for forked repo Apache hop - #17

Merged
srusia-delphix merged 2 commits into
delphix/hop-2.17from
dlpx/pr/srusia-delphix/78d2d83b-d74d-449d-a2c9-1854a8eb0801
Oct 7, 2026
Merged

srusia-delphix merged 2 commits into
delphix/hop-2.17from
dlpx/pr/srusia-delphix/78d2d83b-d74d-449d-a2c9-1854a8eb0801

Conversation

@srusia-delphix

@srusia-delphix srusia-delphix commented Oct 6, 2026 •

Copy link
Copy Markdown

Problem

The masking engine is moving from Hop 1.2.0 to a 2.x line. Hop 2.15 was the original target, but it carries a performance regression in org.apache.hop.core.Condition that makes delimited-file masking roughly 3x slower (see DLPX-99491 and #16): evaluate() rebuilds the right-hand ValueMeta on every row, and under the masking SecurityManager each rebuild costs a policy evaluation.

Apache fixed that regression in 2.17.0, so targeting 2.17 removes the need to carry a Delphix patch for it at all.

Evaluation

2.17.0 is the only version that both fixes the regression and runs on Java 17. Upstream's fix, introduced in 2.17.0 and absent in 2.15/2.16, caches the constant right value in Condition itself:

// Right value is a constant - cache it for performance
if (!rightValueCached) {
  cachedFieldMeta2 = rightValue.createValueMeta();
  cachedField2     = rightValue.createValueData();
  rightValueCached = true;
}

invalidated via clearFieldPositions(), which FilterRows.init() calls once per transform copy per run.

Solution

New delphix/hop-2.17 branch cut from upstream 2.17.0, with the existing Delphix patch set re-applied as a single commit.

This PR replicates #12 (the 2.15 branch creation) onto 2.17 with no additional changes.

Testing Done

CI on this PR: build pass, style checks pass, Jira checks pass.

Compatibility and risk verification performed against upstream artifacts (all reproducible from published jars and the Apache repo):

Notes to Reviewers

One file fewer than the 2.15 patch set. #12 also patched ui/.../PreviewRowsDialogTest.java, which is not carried here as it is removed in upstream 2.17.

@srusia-delphix
srusia-delphix force-pushed the dlpx/pr/srusia-delphix/78d2d83b-d74d-449d-a2c9-1854a8eb0801 branch from 463bc3f to 53eba09 Compare October 6, 2026 06:26
@srusia-delphix
srusia-delphix force-pushed the dlpx/pr/srusia-delphix/78d2d83b-d74d-449d-a2c9-1854a8eb0801 branch from 53eba09 to ea46734 Compare October 6, 2026 06:38
@srusia-delphix
srusia-delphix marked this pull request as ready for review October 6, 2026 07:05
Comment thread core/src/main/java/org/apache/hop/core/database/Database.java

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Database metadata handling, polling configuration, and batch exception reporting contain unresolved functional regressions.

Review effort: Balanced
Findings: 5 Medium severity · 2 Low severity

Open (7)
What changed in this PR

Ports the Delphix masking customizations onto Apache Hop 2.17, including database compatibility, runtime configuration, and internal artifact publishing.

Changes:

  • Configures Delphix Artifactory publishing and dependency resolution.
  • Reapplies database metadata, fetch-size, and exception-reporting customizations.
  • Updates launch classpaths and related tests.
File Description
settings.xml Adds Delphix Artifactory settings.
pom.xml Configures internal artifact deployment.
plugins/​databases/​oracle/​.../​OracleValueMetaBaseTest.java Updates Oracle binary metadata expectations.
engine/​.../​BaseTransform.java Changes input polling timeout behavior.
core/​.../​ValueMetaBaseTest.java Updates null-string conversion expectation.
core/​.../​ValueMetaBase.java Adds database type and JDBC handling customizations.
core/​.../​HopDatabaseBatchException.java Reports chained SQL exceptions.
core/​.../​Database.java Adds configurable fetch sizing and logging propagation.
assemblies/​.../​hop-run.sh Extends runtime classpath.
assemblies/​.../​hop-gui.sh Extends GUI classpath.
.gitignore Ignores local snapshots.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread core/src/main/java/org/apache/hop/core/database/Database.java
Comment thread core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
Comment thread core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
Comment thread core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
Comment thread core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java
Comment thread core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java

@sandeep-delphix sandeep-delphix left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just small NIt's

@srusia-delphix
srusia-delphix merged commit 9d7f09c into delphix/hop-2.17 Oct 7, 2026
5 checks passed
@srusia-delphix
srusia-delphix deleted the dlpx/pr/srusia-delphix/78d2d83b-d74d-449d-a2c9-1854a8eb0801 branch October 7, 2026 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

5 participants