'Spark 2.x to 3.x upgrade Error - The SQL expression for node [ SQLNode7 ] is invalid. Reason: [ Table or view not found: tempView; line 3 pos 19 ]

I am trying to upgrade my spark 2.x to 3.x

I am getting following error

The SQL expression for node [ SQLNode5 ] is invalid. Reason: [ Table or view not found: tempView; line 3 pos 19 ] 

My company internal forum suggest 2 solutions for this

  1. Remove drop view statement from query

or

  1. use the inline configuration --conf spark.sql.legacy.storeAnalyzedPlanForView=true

Recommendation from internal forum is "Remove drop view statement " if it is not possible then only use option 2

My question: 1

What will be the impact if i remove " drop view" statement from query ? I guess it will increase the memory foot print and may impact other jobs in cluster .

is it ok to remove that "drop view" statement

My question: 2

Is there any other way to reduce the memory foot print without using drop view statement



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source