GH-5385: Improve performance of MongoStepExecutionDao::getLastStepExecution#5405
Open
seonwooj0810 wants to merge 1 commit into
Open
Conversation
…:getLastStepExecution Push the step name filter, ordering (createTime DESC, then stepExecutionId DESC), and limit(1) down to MongoDB instead of fetching all step executions for the job instance and filtering/sorting in Java. This mirrors the optimization applied to the JDBC variant in spring-projects#4798 and addresses the existing `// TODO optimize the query` marker. Signed-off-by: Seonwoo Jung <laborlawseon@kap.kr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5385.
Summary
MongoStepExecutionDao#getLastStepExecutioncurrently fetches all step executions for the job instance into memory, then filters by step name and sorts on(createTime, stepExecutionId)in Java. The single-row contract makes this an O(N) cost that grows with the step-execution history despite the O(1) result.This PR pushes the step name filter, ordering (
createTimeDESC, thenstepExecutionIdDESC) andlimit(1)down to MongoDB, mirroring the optimization PR #4798 applied to the JDBC variant. It also removes the existing// TODO optimize the querymarker.The contract
StepExecutionDao#getLastStepExecution("Retrieve the lastStepExecutionfor a givenJobInstanceordered by creation time and then id") is preserved.Changes
MongoStepExecutionDao#getLastStepExecution: filter (name), sort (createTimeDESC,stepExecutionIdDESC), andlimit(1)are now applied via theQueryand resolved byMongoOperations#findOne(which respectsQuery#getSortObjectby delegating tofindwithlimit(1)).MongoStepExecutionDaoIntegrationTests: addedtestGetLastExecutionFiltersByStepNameto assert correct filtering when multiple step names share the same job instance, and to assert that a non-existent step name returnsnull. Existing tests (testSaveAndGetLastExecution,testSaveAndGetLastExecutionWhenSameStartTime) remain unchanged and continue to cover the createTime-then-id ordering contract.Test plan
./mvnw -pl spring-batch-core compile -DskipTests— passes./mvnw -pl spring-batch-core test-compile -DskipTests— passes./mvnw -pl spring-batch-core spring-javaformat:validate— passes@Testcontainers(disabledWithoutDocker = true)and could not be executed in my local environment (no Docker daemon access). They are exercised by upstream CI.Verification done
5385: none found at the time of writing..javaonly.main(// TODO optimize the query+ in-memory filter/sort atMongoStepExecutionDao.java:115-147).JdbcStepExecutionDao::getLastStepExecution#4798 (Improve performance of JdbcStepExecutionDao::getLastStepExecution) — same shape of fix on the JDBC side, mirrored here for Mongo.