Requirement In this post, we are going to get the last modified date of a file in Spark using the File System. It helps while processing the data based on the last modified date time. Solution Here, let’s assume your files are available in hdfs or any storage. We willRead More →
Requirement In this post, we are going to import data from RDBMS to Hadoop. Here, we have MySQL as an RDBMS database. We will use Sqoop to import data from RDBMS to Hadoop. Components Involved MySQL – For source data HDFS – To store source data in Hadoop Sqoop –Read More →
Requirement Suppose we are having a text format data file which contains employees basic details. When we load this file in Spark, it returns an RDD. Our requirement is to find the number of partitions which has created just after loading the data file and see what records are storedRead More →
Requirement Suppose you have a data files which are having duplicate records i.e. a line of a file is occurring more than one times. You want those records which are present multiple times in the file. So the requirement is to find the duplicate record using Map Reduce. Components InvolvedRead More →
Requirement There are many data files available. It is required to find out the total number of records available in data files using Map Reduce. Components Involved HDFS: To store the input data which will be passed to the Map Reduce as an input and also store the map reduce output.Read More →
Requirement Suppose you get data files which are having user’s basic information like first name, last name, designation, city, etc. These basic details are separated by ‘,’ delimiter. Now, the requirement has come to find out all the duplicate value of any field of information. So, here the requirement isRead More →
Requirement Suppose you have a file in which many words are repeated. Now the requirement is to get distinct words from the file using Map Reduce. If you compare with the SQL, then we have to write a map reduce program which is similar to the DISTINCT function of SQL.Read More →
