I am submitting Spark job with following specification:(same program has been used to run different size of data range from 50GB to 400GB) /usr/hdp/2.6.0.3-8/
I am trying to use snowpark(0.6.0) via Jupiter notebooks(after installing Scala almond kernel). I am using Windows laptop. Had to change the examples here a b
I have a Scala stream of bytes that I'd like to write to a file. The stream has too much data to buffer all of it memory. As a first attempt, I created an Inp
I have two data frames. The first contains a list of latitude and longitude points along with an ID number associated with the person who was at those coordina
I have a Dataframe and wish to divide it into an equal number of rows. In other words, I want a list of dataframes where each one is a disjointed subset of the
I have a method that returns a Try object: def doSomething(p: SomeParam): Try[Something] = { // code } I now want to test this with ScalaTest. Currently I
I have two records of type RDD[T] For example: val a: RDD[Integer] = .... val b: RDD[Integer] = ... when I perform val z = a.union(b) println(z) I find the s
In Haskell, I can define a Tree: data Tree a = Empty | Node a (Tree a) (Tree a) How could I write this in Scala? I'm not sure how to keep the type paramete
I want to add a new field to jsonObject and this new field's name will be based on a value of another field. To be clear, this an examples of what I want to ach
I want to add a new field to jsonObject and this new field's name will be based on a value of another field. To be clear, this an examples of what I want to ach
I have a large data set which I am reading from Postgres. It has an ID column, a timestamp column and several other columns which may have been updated. For eac
I have a column of type map, where the key and value changes. I am trying to extract the value and create a new column. Input: ----------------+ |symbols
I am not sure whether mergestrategy or exclude jars is the best option here. Any help with how do I proceed further with this error will be great! [sameert@pzx
My database has numeric value, which is up to 256-bit unsigned integer. However, spark's decimalType has a limit of Decimal(38,18). When I try to do calculatio
I'm using Circe to deserialize json containing a list. Sometimes a few items in the json list are corrupted, and that causes the entire deserialization to fail.
I try to redirect/forward a TCP stream to another Sink with Akka 2.4.3. The program should open a server socket, listen for incoming connections and then consum
val askNameFlatMap: ZIO[Console, IOException, Unit] = putStrLn("What is your Name? ") *> getStrLn.flatMap(name => putStrLn(s"Hello $name")) val askNa
I'd like to detect events that share the same property. Suppose I have a simple case class: case class Record(name: String, value: Int) Suppose there is the
I have a dataframe like this: JoiKey period Age Amount Jk1 2022-02 2 200 Jk1 2022-02 3 450 Jk2 2022-03 5 500 Jk3 2022-03 0 200 Jk2 2022-02 8 300 Jk3 2022-03 9
I've written a very simple Sort scala program with Spark. object Sort { def main(args: Array[String]): Unit = { if (args.length < 2) {