'Spark Scala code does not compile when used with a encoder + by name

This is a simplified version of my code that is failing to compile:

object CompileFail {
  import org.apache.spark.sql.{ Encoder, SparkSession }

  case class Foo(x: Int)

  def run(spark: SparkSession): Int => Int = {
    import spark.implicits._
    add(bar(encode[Foo]))
  }

  def bar(f: => Int): Int = 0

  def encode[A : Encoder]: Int = 0

  def add(a: Int)(b: Int) = a + b
}

It is failing with following non sensical message:

[error] Error while emitting CompileFail.scala
[error] value $u
[error] one error found
[error] Compilation failed

I am on Scala 2.12.15 and Spark 3.1.2 (but it fails on older Spark versions too).

Interesting to note:

  1. If I change add(a)(b) to add(a, b) it compiles
  2. If I change bar(f: => Int) to bar(f: Int) it compiles
  3. If I change add(bar(encode[Foo])) to add(bar(encode[String])) it compiles

What am I doing wrong?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source