Requirement: You have sample dataframe which you want to load in to parquet files using scala. Solution: Step 1: Sample Dataframe use below command: spark-shell Note: I am using spark 2.3 version. To Create a sample dataframe , Please refer Create-a-spark-dataframe-from-sample-data After following above post ,you can see thatRead More →
Requirement : You have parquet file(s) present in the hdfs location. And you need to load the data into the spark dataframe. Solution : Step 1 : Input files (parquet format) Here we are assuming you already have files in any hdfs directory in parquet format. In case if youRead More →
Requirement Suppose we are having a text format data file which contains employees basic details. When we load this file in Spark, it returns an RDD. Our requirement is to find the number of partitions which has created just after loading the data file and see what records are storedRead More →
Requirement There are many data files available. It is required to find out the total number of records available in data files using Map Reduce. Components Involved HDFS: To store the input data which will be passed to the Map Reduce as an input and also store the map reduce output.Read More →
Requirement Suppose you get data files which are having user’s basic information like first name, last name, designation, city, etc. These basic details are separated by ‘,’ delimiter. Now, the requirement has come to find out all the duplicate value of any field of information. So, here the requirement isRead More →
Requirement Suppose you have a file in which many words are repeated. Now the requirement is to get distinct words from the file using Map Reduce. If you compare with the SQL, then we have to write a map reduce program which is similar to the DISTINCT function of SQL.Read More →
