Facing nullpointerexception on spark3 while using foreach on dataframe

Question

I have the following code snippet where I am trying to access some fields inside foreach. I dont wish to use dataframe apis as its some custom logic i am trying it.

IT works perfectly fine on local but breaks on my databricks cluster. Not sure if its a driver-executor problem, because logic seems straight forward to me

code snippet:

dataframe.rdd.foreachPartition(partitions => {
        partitions.foreach(line => {
          try {
            logger.error("Event type is" + line.getAs[String]("event_type"))
          } catch {
            case ex: Exception => {
              logger.error(ex.getMessage, ex)
            }
          }
        })
      })

The error I face is :

(10.40.20.199 executor 3): java.lang.NullPointerException
    at scala.collection.Iterator.foreach(Iterator.scala:941)
    at scala.collection.Iterator.foreach$(Iterator.scala:941)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)

I am using scala 2.12 with spark 3.1.2

Any help would be highly appreciated.