MyPy on PySpark error: "DataFrameLike" has no attribute "values"

Question

I am developing a program in PySpark 3.2.1.

Mypy == 0.950

One of the operations requires to transform information of a small DataFrame into a list.

The code is:

result = df.select("col1","col2","col3").toPandas().values.tolist()

I need to convert it to a list because I then broadcast the information and a pyspark broadcast can't be a DataFrame

For this code I get the following mypy error:

error: "DataFrameLike" has no attribute "values"

I there something I might do to avoid the mypy error?

Answer 1

This is working fine for me.

>>> df=spark.read.option('header','true').csv("C:/Users/pc/Desktop/myfile.txt")
>>> df
DataFrame[col1: string, col2: string, col3: string]
>>> result = df.select("col1","col2","col3").toPandas().values.tolist()
>>> result
[['1', '100', '1001'], ['2', '200', '2002'], ['3', '300', '1421'], ['4', '400', '24214'], ['5', '500', '14141']]

what is Mypy here ?