In this post, we will explore how to read data from Apache Kafka in a Spark Streaming application. Apache Kafka is a distributed streaming platform that provides a reliable and scalable way to publish and subscribe to streams of records. Problem Statement We want to develop a Spark Streaming applicationRead More →
Requirement In our Read JSON file in Spark post, we have read a simple JSON file into a Spark Dataframe. In this post, we are moving to handle an advanced JSON data type. We will read nested JSON in spark Dataframe. Sample Data We are going to use below sampleRead More →
Requirement In this post, we are going to learn how to check if Dataframe is Empty in Spark. This is a very important part of the development as this condition actually decides whether the transformation logic will execute on the Dataframe or not. Solution Let’s first understand how this canRead More →
Requirement In this post, we are going to get the last modified date of a file in Spark using the File System. It helps while processing the data based on the last modified date time. Solution Here, let’s assume your files are available in hdfs or any storage. We willRead More →
Requirement Let’s say we have a data file with a TSV extension. It is the same as the CSV file. What is the difference between CSV and TSV? The difference is separating the data in the file The CSV file stores data separated by “,”, whereas TSV stores data separatedRead More →
Requirement: There is spark dataframe, in which it is needed to add multiple columns altogether, without writing the withColumn , multiple times, As you are not sure, how many columns would be available. Solution : Step 1: A spark Dataframe Enter into your spark-shell , and create a sample dataframe,Read More →
Requirement In this post, we will learn how to get last element in list of dataframe in spark. Solution Create a dataframe with dummy data: val df = spark.createDataFrame(Seq( (“1100”, “Person1”, “Street1#Location1#City1”, null), (“1200”, “Person2”, “Street2#Location2#City2”, “Contact2”), (“1300”, “Person3”, “Street3#Location3#City3”, null), (“1400”, “Person4”, null, “Contact4”), (“1500”, “Person5”, “Street5#Location5#City5”, null) )).toDF(“id”,Read More →
Requirement In this post, we are going to extract or get column value from Data Frame as List in Spark. Let’s take an example, you have a data frame with some schema and would like to get a list of values of a column for any further process. Solution Let’sRead More →
Requirement In this post, we will learn how to get or extract a value from a row. Whenever we extract a value from a row of a column, we get an object as a result. For example, if we have a data frame with personal details like id, name, location,Read More →
Requirement In this post, we will see how to convert column type in spark dataframe. Let’s assume a scenario, we used to get a CSV file from the source. As it is a CSV file, we will see mostly the datatype of the field would be String. In this scenario,Read More →
Requirement In this post, we are going to learn about how to compare data frames data in Spark. Let’s see a scenario where your daily job consumes data from the source system and append it into the target table as it is a Delta/Incremental load. There is a possibility toRead More →
Requirement : There is a Map in scala which is being used by one of your application ,which have all the configurations for your spark app. you need to remove some of conf in Map. Solution : Step 1: Creation of Map Map is the combination of key value pair. Read More →
Requirement In this post, we will convert RDD to Dataframe in Spark with Scala. Solution Approach 1: Using Schema Struct Type //Create RDD: val dummyRDD = sc.parallelize(Seq( (“1001”, “Ename1”, “Designation1”, “Manager1”) ,(“1003”, “Ename2”, “Designation2”, “Manager2”) ,(“1001”, “Ename3”, “Designation3”, “Manager3”) )) val schema = StructType( StructField(“empno”, StringType, true) ::Read More →
Requirement In this post, we are going to create an RDD and then will read the content in Spark. Solution //Create RDD: val dummyRDD = sc.parallelize(Seq( (“1001”, “Ename1”, “Designation1”, “Manager1”) ,(“1003”, “Ename2”, “Designation2”, “Manager2”) , (“1001”, “Ename3”, “Designation3”, “Manager3”) )) //Read RDD dummyRDD.collect().foreach(println(_)) //Read specific Column: dummyRDD.collect().foreach(data => println(data._1,Read More →
Requirement In this post, we are going to learn how to create an empty dataframe in Spark with and without schema. Prerequisite Spark 2.x or above Solution We will see create an empty DataFrame with different approaches: PART I: Empty DataFrame with Schema Approach 1:Using createDataFrame Function import org.apache.spark.sql.types.{StructType, StructField,Read More →
