CASSANALYTICS-173: Add Apache Spark 4.1 support and upgrade Java driv… - #1389
Open
msmygit wants to merge 4 commits into
Open
CASSANALYTICS-173: Add Apache Spark 4.1 support and upgrade Java driv…#1389msmygit wants to merge 4 commits into
msmygit wants to merge 4 commits into
Conversation
added 2 commits
June 5, 2026 23:50
…er to 4.19.2
Upgrade the connector to Apache Spark 4.1.2 and Cassandra Java driver 4.19.2.
Spark 4.x requires Java 17 and Scala 2.13, so Scala 2.12 and Java 8/11 are
dropped.
Build:
* Spark 4.1.2, Cassandra Java driver 4.19.2, Jetty 11.0.26, Scala 2.13.17
* Build Scala 2.13 only; target Java 17 (-release 17 / javac 17)
* Add JDK 17 --add-opens java options for forked (test) JVMs
* Remove obsolete -XX:MaxPermSize from the sbt launcher
* Remove unused scala-2.12 source directories
Spark 4 API migration (connector):
* Use org.apache.spark.sql.classic.{Strategy, SparkSession, Dataset} where the
Connect/Classic split moved the implementations
* Convert Column <-> Expression via classic ClassicConversions/ColumnConversions
* DataSourceV2Relation gained a timeTravelSpec field; update pattern matches
* Replace removed Dataset.sqlContext / SQLContext.conf with sparkSession /
sessionState.conf
* Adapt to error-class based exception constructors (AnalysisException,
NoSuch/AlreadyExists exceptions)
Docs/CI: update README version matrix, quick start and other docs to Spark 4.1
/ Scala 2.13 / Java 17, add CHANGES entry, and run CI on Java 17 + Scala 2.13.
Spark 4 repackaged the streaming internals and split SQLImplicits/SparkSession. Update the integration-test fixtures accordingly: * Import LongOffset/SerializedOffset/StreamingQueryWrapper from the new org.apache.spark.sql.execution.streaming.runtime package * Use classic Dataset.ofRows with the classic SparkSession in the fake stream sources; make offset branches consistently Long * Use the session's built-in implicits instead of a hand-rolled SQLImplicits subclass (SQLImplicits gained abstract members) * Cast to the classic SparkSession for cloneSession() Also point generateDocs.sh at the scala-2.13 output directory.
added 2 commits
June 8, 2026 11:23
…17 opens to IT forks * Bump Scala to 2.13.18 (build + CI matrix) and Cassandra Java driver to 4.19.3 * Add the JDK 17 --add-opens options to the integration-test group fork JVMs in Testing.scala; makeTestGroups builds its own ForkOptions and would otherwise drop them, making Spark fail under Java 17 during it:test
…k 4 (found via it:test) Running the integration tests against Cassandra surfaced three Spark 4 regressions: * setDirectJoin: Spark 4's .load() returns a lazy UnresolvedDataSource as the parsed logical plan, so transforming queryExecution.logical matched nothing and the directJoin() hint was silently dropped. Transform the analyzed plan instead, and match the CassandraTable inside DataSourceV2ScanRelation (its first field is the relation in Spark 4, not the table). * CassandraCatalog.tableMissing/namespaceMissing: Spark 4 removed the free-form message constructors from NoSuchTable/NamespaceException, which dropped the NameTools 'Couldn't find / did you mean' suggestions. Override getMessage on the typed exception to preserve the suggestion text. * CassandraCatalogSpecBase: Spark 4 no longer treats spark.sql.defaultCatalog as the current catalog; select it explicitly via catalogManager.setCurrentCatalog after setting the active session (plugin loading resolves against it). Verified against Cassandra 5.0.4 on JDK 17: CassandraDirectJoinSpec 35/35, CassandraDataSourceSpec 33/33, CassandraDataFrameSpec 14/14, TableWriterSpec 66/66, CassandraCatalogNamespaceSpec 14/14, CassandraCatalogTableSpec 21/21. Unit tests remain green (245 + 316).
|
Hi Team, Please check if the support for default keyspace can be added in clCatalog. As like MySQL Catalog, there is option to set defaultDatabase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…er to
4.19.3Upgrade the connector to Apache Spark
4.1.2and Cassandra Java driver4.19.3. Spark4.xrequires Java 17 and Scala 2.13, so Scala 2.12 and Java 8/11 are dropped.Build:
Spark 4 API migration (connector):
Docs/CI: update README version matrix, quick start and other docs to Spark 4.1 / Scala 2.13 / Java 17, add CHANGES entry, and run CI on Java 17 + Scala 2.13.
Description
How did the Spark Cassandra Connector Work or Not Work Before this Patch
Describe the problem, or state of the project that this patch fixes. Explain
why this is a problem if this isn't obvious.
Example:
"When I read from tables with 3 INTS I get a ThreeIntException(). This is a problem because I often want to read from a table with three integers."
General Design of the patch
How the fix is accomplished, were new parameters or classes added? Why did you
pursue this particular fix?
Example: "I removed the incorrect assertion which would throw the ThreeIntException. This exception was incorrectly added and the assertion is not actually needed."
Fixes: CASSANALYTICS-173
gh-pagesPR is #1388How Has This Been Tested?
Almost all changes and especially bug fixes will require a test to be added to either the integration or Unit Tests. Any tests added will be automatically run on travis when the pull request is pushed to github. Be sure to run suites locally as well.
**Expand / Collapse for local testing results **
Checklist: