Skip to content

build: upgrade Flink from 1.18.1 to 1.20.5 - #248

Merged
pallakartheekreddy merged 2 commits into
Sunbird-Lern:masterfrom
romitshah02:flink-v1.20.5
Aug 12, 2026
Merged

build: upgrade Flink from 1.18.1 to 1.20.5#248
pallakartheekreddy merged 2 commits into
Sunbird-Lern:masterfrom
romitshah02:flink-v1.20.5

Conversation

@romitshah02

Copy link
Copy Markdown
Contributor

Upgrades Apache Flink from 1.18.1 to 1.20.5 across all job modules. Migrates off the deprecated flink-streaming-scala API (removed in Flink 2.0) to flink-streaming-java/Java DataStream types, bumps flink-connector-kafka to 3.4.0-1.20, and fixes a stale log4j pin in the Docker image that crashed the TaskManager on startup against the new base image.

Dependencies required: org.apache.flink:flink-connector-kafka:3.4.0-1.20, org.apache.flink:flink-streaming-java:1.20.5 (replacing flink-streaming-scala_2.12), log4j2 2.24.3 (matches what Flink 1.20.5 already bundles).

Type of change

Please choose appropriate options.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@romitshah02
romitshah02 marked this pull request as ready for review August 11, 2026 11:46

@sntiwari1 sntiwari1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Clean, well-scoped Flink 1.18.1 → 1.20.5 upgrade. The core migration pattern — dropping flink-streaming-scala (removed in Flink 2.0) for flink-streaming-java/Java DataStream, switching OutputTag construction to the explicit (name, TypeInformation) constructor now that the implicit Scala macro is gone, and widening Iterable[I]java.lang.Iterable[I] in the WindowBaseProcessFunction/TimeWindowBaseProcessFunction process() signatures — is applied consistently across every job module (lms-jobs, ml-jobs, notification, user-org-jobs). I verified:

  • No leftover references to org.apache.flink.streaming.api.scala or flink-streaming-scala anywhere in the tree (git grep clean).
  • WindowBaseProcessFunction/TimeWindowBaseProcessFunction have no subclasses in the repo, so the breaking process() signature change is safe.
  • The stringTypeInfo/notificationMetaTypeInfo/userFeeMetaTypeInfo implicits referenced by the new OutputTag constructors are already declared in each touched config class, so nothing is missing.
  • flink-connector-kafka:3.4.0-1.20 usage in FlinkKafkaConnector already relies on the KafkaSource builder API (not the legacy FlinkKafkaConsumer), so no additional migration is needed there.
  • The add-jira-description CI failure on this PR is an unrelated Jira automation check, not a build/test failure — no CI job actually compiles/tests this PR currently.

Only minor, non-blocking observations below — nothing I'd hold the merge for.

Comment thread jobs-distribution/Dockerfile
Comment thread jobs-core/src/main/scala/org/sunbird/job/BaseProcessFunction.scala
Comment thread lms-jobs/credential-generator/collection-cert-pre-processor/pom.xml
Comment thread pom.xml
@romitshah02 romitshah02 changed the title build: upgrade Flink from 1.18.1 to 1.20.5- #910 build: upgrade Flink from 1.18.1 to 1.20.5 Aug 11, 2026
@pallakartheekreddy
pallakartheekreddy merged commit f36189f into Sunbird-Lern:master Aug 12, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants