'pickle.PicklingError: Cannot pickle files that are not opened for reading: a error msg in pyspark code

I have written the below code which need to be parse json data and get ID from json data. While executing the code, I want to log that data as well for the debugging purpose but when code execute, it throw error as "pickle.PicklingError: Cannot pickle files that are not opened for reading: a". If anyone has any idea about this error. Request you to help here.


id_list = current_value_df.rdd.map(lambda row:
                                       (logger.info("row_value {b}".format(b=str(row[0]))))
                                       (json.loads(str(row[0]))['id'])
                                       ).collect()

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'pickle.PicklingError: Cannot pickle files that are not opened for reading: a error msg in pyspark code

Sources

Related Questions