'In Azure Synapse Analytics (Pyspark) note book , Using Spark Context Hadoop File system. I'm not able to move/copy or rename the files

In Azure Synapse Analytics (Pyspark) note book , Using Spark Context Hadoop File system. I'm able to delete a folder or file but not able to move/copy or rename the files , keep getting the error Below is the snipped I used:

from pyspark.sql import SparkSession
# prepare spark session
spark = SparkSession.builder.appName('filesystemoperations').getOrCreate()
# spark context
sc = spark.sparkContext

# File path declaration
containerName = "ZZZZZZZ"
fullRootPath =  "abfss://{cName}@{cName}.dfs.core.windows.net".format(cName=containerName)
tablePath = "/ws/tables/"
evnetName = 'abcd'
tableFilename= "Game_"+eventName+".kql"
tableFile = fullRootPath + tablePath + tableFilename
tempTableFile = fullRootPath + tablePath + tempPath + tableFilename

#empty the paths
sc._jsc.hadoopConfiguration().set('fs.defaultFS', fullRootPath)
fs = (sc._jvm.org.apache.hadoop.fs.FileSystem.get(sc._jsc.hadoopConfiguration()))
fs.delete(sc._jvm.org.apache.hadoop.fs.Path(fullRootPath+tablePath), True)
## This one worked fine.


# dfStringFinalF contains a string
spark.sparkContext.parallelize([dfStringFinalF]).coalesce(1).saveAsTextFile(tempTunctionFile)
fs.copy(tempTunctionFile+'/part-00000' , tableFile)
#Copy , rename, cp, mv, nothing working on fs

Please help

azureapache-sparkpysparkazure-synapse


Solution 1:[1]

It seems in Azure Synapse Analytics is having limitations with Spark Context and shutil libraries. mssparkutils library helps in copying/moving files in the BLOB containers.

Here is the code sample

from notebookutils import mssparkutils
fullRootPath =  "abfss://{cName}@{cName}.dfs.core.windows.net".format(cName=zzzz)

fsmounting = mssparkutils.fs.mount( 
    fullRootPath, 
    "/ws", 
    {"linkedService":"abcd"}
)

fsMove = mssparkutils.fs.mv(tempTableFile+'/part-00000',tableFile )

More reference from the below link : https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-file-mount-api

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions