'Spark StreamingContext
I have a problem running Spark Streaming. Can someone please help me below?
Solution 1:[1]
Since you are using /Filestore, I believe you are using databricks. Below code would help you to start a spark streaming context.
If you are using databricks, clear all the states and run the below code.
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
ssc = StreamingContext(spark.sparkContext, 1)
dstream = ssc.textFileStream("<Folder/File location")
dstream.saveAsTextFiles("<Destination folder/file location")
ssc.start()
ssc.awaitTermination()
I would suggest you to start using spark structured streaming, instead of using standard streaming option.
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | act_coder |