Cannot connect to Cassandra with Spark connector 3.1.0 and Spark 3.1.2

Question

I'm trying to connect cassandra, using spark-cassandra-connector, but the following message appears:

spark.version: 3.1.2

cassandra.connector.version: 3.1.0

Caused by: java.io.IOException: Failed to open native connection to Cassandra at {10.99.249.84:9042} :: org/apache/tinkerpop/gremlin/structure/io/BufferFactory
at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:173)
at com.datastax.spark.connector.cql.CassandraConnector$.$anonfun$sessionCache$1(CassandraConnector.scala:161)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:103)
at com.datastax.spark.connector.datasource.CassandraCatalog$.com$datastax$spark$connector$datasource$CassandraCatalog$$getMetadata(CassandraCatalog.scala:455)
at com.datastax.spark.connector.datasource.CassandraCatalog$.getTableMetaData(CassandraCatalog.scala:421)
at org.apache.spark.sql.cassandra.DefaultSource.getTable(DefaultSource.scala:68
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:334)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:317)
... 1 more
Caused by: java.lang.NoClassDefFoundError: org/apache/tinkerpop/gremlin/structure/io/BufferFactory
at com.datastax.dse.driver.internal.core.graph.GraphRequestAsyncProcessor.<init>(GraphRequestAsyncProcessor.java:48)

Answer 1

The stack trace indicates to me that you're trying to run a Graph-Analytics query which suggests you're connecting to a DataStax Enterprise (DSE) cluster.

The first issue is that the latest version of DSE (which is 6.8) ships with Apache Spark 2.4. You cannot run Spark 3.1 jobs against DSE 6.8 since it only supports Spark 2.4.

The second issue is that the open-source Spark connector cannot access DSE-only features including Graph. DSE features are only accessible using the DSE version of the Spark connector which is included in DSE. You will not be able to access DSE features using the OSS version of the Spark connector. Cheers!