'Structured Streaming output is not showing on Jupyter Notebook
I have two notebooks. First notebook is reading tweets from twitter using tweepy and writing it to a socket. Other notebook is reading tweets from that socket using spark structured streaming (Python) and writing it's result to console. Unfortunately I'm not getting output on jupyter console. Code is working fine on pycharm.
spark = SparkSession \
.builder \
.appName("StructuredStreaming") \
.getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
# This is Spark Structured Streaming Code which is reading streams from twitter and showing them on console.
tweets = spark \
.readStream \
.format("socket") \
.option("host", "127.0.0.1") \
.option("port", 7000) \
.load()
query = tweets \
.writeStream \
.option("truncate", "false") \
.outputMode("append") \
.format("console") \
.start()
query.awaitTermination()
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
