'Error due to the string name in withColumn

I have this for loop

for j in jdbc_db.schema.fields:
  drop_df = drop_df.withColumn(j.name,col(j.name).cast(j.dataType))

And I get an error when j is StructField(Location.Authorized,DoubleType,true) this the error: AnalysisException: cannot resolve '`Location.Authorized`' given input columns: [ALL THE COLUMNS];

I'm pretty sure it's because of the name of the column(Location.Authorized), but if possible it should keep the name.

So, in case I am right, I would like to know a way to execute the withColumn without errors.

But if the name is not the problem, I would like know what is the real problem

pyspark azure-databricks

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Error due to the string name in withColumn

Sources

Related Questions