'Calculate MD5 hash CSV file in synapse notebook using pyspark
I want to calculate a hash (md5) from a .CSV in pyspark stationed in ADLS gen2. It seems that pyspark can't transform a df into bytes.
import hashlib
df = spark.read.csv('abfss://*******@***********.dfs.core.windows.net/filetomd5.csv')
file_md5_digest = hashlib.md5(df.encode()).hexdigest()
this generates the following error: 'AttributeError: 'DataFrame' object has no attribute 'encode'
I need help calculating the entire .csv in a single hash.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
