'Calculate MD5 hash CSV file in synapse notebook using pyspark

I want to calculate a hash (md5) from a .CSV in pyspark stationed in ADLS gen2. It seems that pyspark can't transform a df into bytes.

import hashlib

df = spark.read.csv('abfss://*******@***********.dfs.core.windows.net/filetomd5.csv')
file_md5_digest = hashlib.md5(df.encode()).hexdigest()

this generates the following error: 'AttributeError: 'DataFrame' object has no attribute 'encode'

I need help calculating the entire .csv in a single hash.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source