'Mapping Data Flow failing with error on Livy State = Dead, how to resolve this?
I created a mapping data flow to process flat files and load into Snowflake. It was working fine earlier. From yesterday, I'm receiving this error and pipeline is running for like 2 hours and failing. Anyone has faced the same issue? Earlier, it used to complete in 4 minutes.
Solution 1:[1]
Dataflow runs on Azure Databricks. This problem occurred in the Spark when Spark cluster incorrectly defined argument value. Once you corrected that, the error will go away, and the job is finished. The error seems generic though, so it could be any number of configuration items.
Refer this link
You can also Refer this link
Solution 2:[2]
sorted the issue, for some reason the dependencies in the solution had gotten 'knocked out of whack'. Dependencies sorted, which appears to have resolved the issue, I'm still a bit perplexed as to how it would run in debug mode, but not compile but don't need to dwell on the issue, just happy that I can get back to productive work on this application.
Thanks for your feedback.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | AbhishekKhandave-MT |
| Solution 2 | Paul Johnson |