'Multi-writer error for persistence function in Structure stream

I'm trying to write into 2 differents databases in my structure streaming process.

for this, I'm using this code:

df.writeStream
  .trigger(Trigger.ProcessingTime("15 seconds"))
  .option("checkpointLocation", "/FileStore/checkpoints/check")
  .foreachBatch { (batchDF: DataFrame, batchId: Long) =>
    
 
      batchDF.persist()
      
     //write 1
     //write 2
       
      
      batchDF.unpersist()
    }

  .start()

However, I'm getting this error:

error: overloaded method value foreachBatch with alternatives:
  (function: org.apache.spark.api.java.function.VoidFunction2[org.apache.spark.sql.Dataset[org.apache.spark.sql.Row],java.lang.Long])org.apache.spark.sql.streaming.DataStreamWriter[org.apache.spark.sql.Row] <and>
  (function: (org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], scala.Long) => Unit)org.apache.spark.sql.streaming.DataStreamWriter[org.apache.spark.sql.Row]
 cannot be applied to ((org.apache.spark.sql.DataFrame, scala.Long) => org.apache.spark.sql.DataFrame)
  .foreachBatch { (batchDF: DataFrame, batchId: Long) => 
  ^

I read that could be a Spark and scala version, but I'm using the databricks runtime version 10.3 (includes Apache Spark 3.2.1, Scala 2.12)

any idea how could I handle this??

Thanks in advance!!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source