question

shehzadjahagirdar_185613 avatar image
shehzadjahagirdar_185613 asked shehzadjahagirdar_185613 published

Getting compilation error in IntelliJ after upgrading Spark connector to version 3.1.0

We are currently using spark cassandra connector version 2.4.2 without any issue when compiling in intellij with sbt package.

But after upgrade to the latest version,we are now facing compilation error in intellij with sbt package using spark cassandra connector version 3.1.0.

PFA below error.

[IJ]package
[info] compiling 9 Scala sources to
/home/abc/IdeaProjects/completion_2021_spark3/target/scala-2.12/classes
...
[error] /home/abc/IdeaProjects/completion_2021_spark3/src/main/scala/completeVisit.scala:255:30:
value getDateTime is not a member of
com.datastax.spark.connector.CassandraRow
[error] maledate.equals(r.getDateTime("male_dt")))
[error] ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 9 s, completed Apr 8, 2022 6:25:39 AM
[IJ]
  • minimal code which replicates the issue
val rdd=sparkSession.sparkContext.cassandraTable(keySpaceName, tableName).select(columnRef.map(ColumnName(_)): _*)
.where(s"empid in (123)"


rdd.filter(visit =>
isAgeAboveBeforeStart(visit, m, true, 12,CalenderUnit.YEAR)
&& isDateEqual(visit, m, "firstDate","compareDate")
)

def isDateEqual(r: CassandraRow, m: MeasureProperty, firstDate: String, compareDate: String): Boolean = {
var isExist = false
isExist = (r.getDateTime(firstDate).equals(r.getDateTime(compareDate))
isExist;
}


In spark cassandra connector 2.4.2 cassandra row object supports row.getDateTime() method whereas in spark cassandra connector 3.1.0 row.getDateTime() is not supported any alternative way for this issue.


build.sbt

import com.typesafe.sbt.SbtNativePackager.autoImport.NativePackagerHelper._

import sbt.Package.ManifestAttributes

name := "Sample-opt"

version := "1.5.8"

scalaVersion := "2.12.10"

val sparkVersion = "3.1.1"

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % sparkVersion,

"org.apache.spark" %% "spark-sql" % sparkVersion,

"com.datastax.cassandra" % "cassandra-driver-core" % "3.11.1",

"com.datastax.spark" %% "spark-cassandra-connector" % "3.1.0",

"com.datastax.spark" %% "spark-cassandra-connector-driver" % "3.1.0",

"org.apache.tinkerpop" % "gremlin-core" % "3.5.2",

"joda-time" % "joda-time" % "2.10.13",

"com.twitter" % "jsr166e" % "1.1.0",

"org.jvnet.mock-javamail" % "mock-javamail" % "1.9" % "test",

"com.upplication" % "s3fs" % "2.2.2",

"org.apache.logging.log4j" % "log4j-api" % "2.17.1",

"org.postgresql" % "postgresql" % "9.3-1102-jdbc41",

"junit" % "junit" % "4.13.2" % Test,

"org.scalatest" %% "scalatest" % "3.0.5" % Test,

"com.novocode" % "junit-interface" % "0.11" % Test exclude("junit", "junit-dep"),

"com.google.cloud" % "google-cloud-storage" % "2.4.4"


)


spark-cassandra-connector
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered shehzadjahagirdar_185613 commented

You haven't provided enough information for us to provide you with a meaningful answer.

The only thing I can say is that you need to refactor your application before you can compile it. The spark-cassandra-connector version 2.4.2 only works with Spark 2.4 whereas version 3.1 of the connector only works with Spark 3.1.

If you update your original question with the following:

  • minimal code which replicates the issue
  • details of how you've configured your IntelliJ environment

we'd be happy to review it. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

shehzadjahagirdar_185613 avatar image shehzadjahagirdar_185613 commented ·

Hi @Erick Ramirez Below are the details requested by you.

  • minimal code which replicates the issue
val rdd=sparkSession.sparkContext.cassandraTable(keySpaceName, tableName).select(columnRef.map(ColumnName(_)): _*)
.where(s"empid in (123)"


rdd.filter(visit =>
isAgeAboveBeforeStart(visit, m, true, 12,CalenderUnit.YEAR)
&& isDateEqual(visit, m, "firstDate","compareDate")
)

def isDateEqual(r: CassandraRow, m: MeasureProperty, firstDate: String, compareDate: String): Boolean = {
var isExist = false
isExist = (r.getDateTime(firstDate).equals(r.getDateTime(compareDate))
isExist;
}


In spark cassandra connector 2.4.2 cassandra row object supports row.getDateTime() method whereas in spark cassandra connector 3.1.0 row.getDateTime() is not supported any alternative way for this issue.

0 Likes 0 ·