Requirement Suppose we are having a text format data file which contains employees basic details. When we load this file in Spark, it returns an RDD. Our requirement is to find the number of partitions which has created just after loading the data file and see what records are storedRead More →
Requirement In spark-shell, it creates an instance of spark context as sc. Also, we don’t require to resolve dependency while working on spark shell. But it all requires if you move from spark shell to IDE. So how to create spark application in IntelliJ? In this post, we are goingRead More →
Requirement You have marks of all the students of a class with roll number in CSV file. It is needed to calculate the percentage of marks of students in Spark using Scala. Given: Download the sample CSV file which have 7 columns, 1st column is Roll no and other 6Read More →
Requirement Assume you have the hive table named as reports. It is required to process this dataset in spark. Once we have data of hive table in the Spark data frame, we can further transform it as per the business needs. So let’s try to load hive table in theRead More →
Requirement Suppose the source data is in a file. The file format is a text format. The requirement is to load the text file into a hive table using Spark. In addition to this, read the data from the hive table using Spark. Therefore, let’s break the task into sub-tasks:Read More →
Requirement In the last post, we have demonstrated how to load JSON data in Hive non-partitioned table. This time we are having the same sample JSON data. The requirement is to load JSON Data into Hive Partitioned table using Spark. The hive table will be partitioned by some column(s). TheRead More →
Requirement Suppose there is a source data which is in JSON format. The requirement is to load JSON data into Hive non-partitioned table using Spark. Let’s break the requirement into two tasks: Load JSON data in spark data frame and read it Store into hive non-partition table Components Involved ToRead More →
