'Multi-writer error for persistence function in Structure stream
I'm trying to write into 2 differents databases in my structure streaming process.
for this, I'm using this code:
df.writeStream
.trigger(Trigger.ProcessingTime("15 seconds"))
.option("checkpointLocation", "/FileStore/checkpoints/check")
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
batchDF.persist()
//write 1
//write 2
batchDF.unpersist()
}
.start()
However, I'm getting this error:
error: overloaded method value foreachBatch with alternatives:
(function: org.apache.spark.api.java.function.VoidFunction2[org.apache.spark.sql.Dataset[org.apache.spark.sql.Row],java.lang.Long])org.apache.spark.sql.streaming.DataStreamWriter[org.apache.spark.sql.Row] <and>
(function: (org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], scala.Long) => Unit)org.apache.spark.sql.streaming.DataStreamWriter[org.apache.spark.sql.Row]
cannot be applied to ((org.apache.spark.sql.DataFrame, scala.Long) => org.apache.spark.sql.DataFrame)
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
^
I read that could be a Spark and scala version, but I'm using the databricks runtime version 10.3 (includes Apache Spark 3.2.1, Scala 2.12)
any idea how could I handle this??
Thanks in advance!!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
