I have a Spark Table, which contains 400+ millions records/rows. I used spark.table to convert it into a DF. The DF looks like this below id pub_date
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(
I am trying to create a Spark application running on Scala that reads a .csv file that is located in src/main/resources directory and saves it on the local hdfs
I am loading data via pipelines in ADLS gen2 container. Now I want to create a table that has details that when the pipeline start running and then completed. l
I have a Seq which contains randomly generated words. I want to calculate the occurrence count of each word using map reduce. Now, I have been able to map the w
package com.knoldus import org.apache.flink.api.java.utils.ParameterTool import org.apache.flink.streaming.api.scala._ import org.apache.flink.streaming.api.win
I need to use window function that is paritioned by 2 columns and do distinct count on the 3rd column and that as the 4th column. I can do count with out any is
With scala 2.11 and spark-streaming-kafka-0-8_2.11 I could do import org.apache.spark.streaming.kafka.KafkaCluster val params = Map[String, Object]( "bootstr
I have a pipe delimited file I need to strip the first two rows off of. So I read it into and RDD, exclude the first two rows, and make it into a data frame. va
I am trying to upgrade from Jackson-2.5 to 2.10 Below deserialization code was working for me, before but post upgrade the solution is failing with following er
I'm trying to use sbt-avro in a Scala project to generate Scala classes from an Avro schema. Here is the project structure: multi-tenant/ build.sbt proj
I'm trying to instantiate a SparkContext inside a SBT console, using the following scala commands: import org.apache.spark.SparkConf import org.apache.spark.Spa
I have junit tests in my scala sbt project. I know, that I can generate html reports for ScalaTest with: testOptions in Test += Tests.Argument(TestFrameworks.
I have junit tests in my scala sbt project. I know, that I can generate html reports for ScalaTest with: testOptions in Test += Tests.Argument(TestFrameworks.
Say I have an dataframe which contains a column (called colA) which is a seq of row. I want to to append a new field to each record of colA. (And the new filed
I am running databricks 7.3LTS and having errors while trying to use scala bulk copy. The error is: object sqldb is not a member of package com.microsoft. I hav
I'm trying to figure out a simple operation that takes a list of (Int, Int) tuples and multiplies the tuples internally and then sums those results. Example: v
This code prints List(1, 2, ()) def f1(i:Int): Either[String,Int] = if (i > 0) Right(1) else Left("error 1") def f2(i:Int): Either[String,Int] = if
I want to convert Spark dataframe all TIMESTAMP columns into String columns. Could anybody say how to do that automatically for each dataframe? The position of
I am forming a query in a String Builder like below : println(dataQuery) Execution started at 2019-10-31 02:58:24.006019 PST res245: String = " SELECT transac