Category "scala"

scala add _2 s in a list of Tuple 2

I have the following mutable Hashmap in Scala: HashMap((b,3), (c,4), (a,8), (a,2)) and need to be converted to the following: HashMap((b,3), (c,4), (a,10)) I

command 'scala' not found after set up by coursier

I use coursier to set up the environment of Scala using command ./cs setup in Ubuntu 20.04 LTS. It was said that all the requirements were installed. However, w

Update a highly nested column from string to struct

Spark - Update a nested column to string

What causes AbstractMethodError for scala implicit val?

I have a gears library, depending on "org.typelevel" %% "cats-core" % "2.7.0", "org.typelevel" %% "cats-effect" % "2.5.4"

Can't figure out how to change Prometheus content type header

So my metrics all appear in one line at my end-point, not in new line per metric. I use micrometer, spring, prometheus and scala. My controller: @RequestMapping

Generate Kafka message with Headers using Apache Spark

I have an ETL (spark-scala). After writing in a table, a message with "header" must be sent to Kafka. I couldn't add the header in the message. I have a spark D

How to get list of all leaf folders from ADLS Gen2 path via Scala code?

We have folders and subfolders in it with year,month, day folders in it. How can we get only the last leaf level folder list using dbutils.fs.ls utility? Exampl

Create a collection of case class names

I'm working in Spark 3.1 with Scala 2.12.10. I'd like to create a collection (Seq, whatever) of case classes that have implemented a common trait, so I can exec

ChannelClosedException - Upstream Address: Not Available

Our scala/thrift service is using twitter finagle 2.12-18.10.0 with java8. The Service is working fine for single request or handful of requests but when we try

tail recursion count, split & get the prefix of a list functions

Language: Scala I'm working on some tail recursion questions in Scala. (GOAL) A function that counts the number of ones in a list. The function takes a list of

Performing a groupBy on a dataframe while limiting the number of rows

I have a dataframe that contains an "id" column and a "publication" column. The "id" column contains duplicates, and represents a researcher. The "publication"

Using K0.ProductInstances in shapeless3

I have a library with typeclasses that I am migrating to Scala 3 using shapeless-3. One of my typeclasses is: trait Parser[T] { def parse(ctx: Context): (Opt

Cannot pass arguments using sbt to gatling simulation

Regarding to Gatling SBT execute a specific simulation topic is there any way to pass argument to simulation? I've been trying passing command from any CLI like

flink cluster with zookeeper HA always shutdown: [RECEIVED SIGNAL 15: SIGTERM]

Environment: flink1.14.4 standalone application mode in kubernetes according to official steps: flink cluster: https://nightlies.apache.org/flink/flink-docs-rel

How can I use snowflake jar in Bitnami Spark Docker container?

I was able to create docker based bitnami stand alone spark instance and run spark jobs on it. However I'm not able not able to write data to snowflake from the

Extract value from ArrayType column in Scala and reshape to long

I have a DataFrame that consists of Column that is ArrayType, and the array may have a different length in each row of the data. I have provide some example cod

How to find position of substring in another column of dataframe using spark scala

I have a Spark scala DataFrame with two columns, text and subtext, where subtext is guaranteed to occur somewhere within text. How would I calculate the positio

Create Map from Elements from List of case class

case class Student(id:String, name:String, teacher:String ) val myList = List( Student("1","Ramesh","Isabela"), Student("2","Elena","Mark"),Student(

Spark 3.2.1 fetch HBase data not working with NewAPIHadoopRDD

Below is the sample code snippet that is used for data fetch from HBase. This worked fine with Spark 3.1.2. However after upgrading to Spark 3.2.1, it is not wo

Category "scala"

Other Categories