'Renaming cached RDD/Dataframe description in Spark UI

In Scala, there's a method called setName which enables users to prescribe a user-friendly display of their cached RDDs/Dataframes under Spark UI's Storage tab. (I'm using Databricks for this operation)

Note: I've already attempted to use setName method available using the Python API, but this doesn't appear to update the descriptions of the RDD name in the Storage tab in Spark UI (each "RDD Name" is shown in query form) -

Does anyone know how we can display this in a user friendly manner? If no mechanism is currently available feel free to also let me know this!

Solution 1:^[1]

it's the same for Python - just use setName function on pyspark.RDD (see doc):

df.rdd.setName("myName")

Solution 2:^[2]

I think you can try to create a SQL view and using the SQL context to cache, that works for me on pyspark.

df.createOrReplaceTempView(name)
sqlContext.cacheTable(name)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Alex Ott
Solution 2	eh0005

'Renaming cached RDD/Dataframe description in Spark UI

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]