'Spark 2.x to 3.x upgrade Error - The SQL expression for node [ SQLNode7 ] is invalid. Reason: [ Table or view not found: tempView; line 3 pos 19 ]
I am trying to upgrade my spark 2.x to 3.x
I am getting following error
The SQL expression for node [ SQLNode5 ] is invalid. Reason: [ Table or view not found: tempView; line 3 pos 19 ]
My company internal forum suggest 2 solutions for this
- Remove
drop viewstatement from query
or
- use the inline configuration
--conf spark.sql.legacy.storeAnalyzedPlanForView=true
Recommendation from internal forum is "Remove drop view statement " if it is not possible then only use option 2
My question: 1
What will be the impact if i remove " drop view" statement from query ? I guess it will increase the memory foot print and may impact other jobs in cluster .
is it ok to remove that "drop view" statement
My question: 2
Is there any other way to reduce the memory foot print without using drop view statement
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
