'WriteStream is not able to write Data in Delta Table
I am trying to connect Streaming Json files from the streaming path using the below code
Schema1= "customerId STRING,orderId STRING,products ARRAY<STRUCT<productId: STRING,quantity: STRING,soldPrice: STRING>>,salesRepId STRING,shippingAddress STRUCT<address: STRING,attention: String,city: STRING,state: STRING,zip: STRING>,submittedAt TIMESTAMP";
streamingDF = (spark.readStream.schema(Schema1)\
.option("maxFilesPerTrigger", 1).json(stream_path))
After few transformations in streamingDF Streaming Dataset and trying to write to a Delta Table using below code
streamingDF.writeStream.outputMode("append")\
.option("checkpointLocation", orders_checkpoint_path)\
.partitionBy("submitted_yyyy_mm")\
.table("sachin")
But those records are not inserted into our delta table and also when I checked the dashboard it shows that numInputRows is 0
Screenshot of streaming while writestream being executed
Why those records are not append into delta table?
Solution 1:[1]
instead of using .table() use .start() then path the table path instead of the table name:
streamingDF.writeStream.outputMode("append")
.option("checkpointLocation", orders_checkpoint_path)
.partitionBy("submitted_yyyy_mm")
.start("/pathtotable/sachin")
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | user3322581 |
