Requirement Suppose we are having some data in a hive table. The table contains information about company’s quarterly wise profit. Now, the requirement is to find max profit of each company from all quarters. Sample Data The record having 5 columns – company name, quarter 1 as Q1, quarter 2Read More →
Requirement Suppose we have data in Hive table. We want the same data into HBase table. So, our requirement is to migrate the data from Hive to HBase table. Components Involved Hive – Source table HBase – Target Table Solution We cannot load data directly into HBase table from theRead More →
Requirement Suppose, there is a table named EMPLOYEE in MySQL database. We want this table data in Hadoop ecosystem. So, the requirement is to import data from MySQL into Hive using Sqoop. Once data is available in Hive, we can process it. Components Involved In order to achieve the requirement,Read More →
Requirement You have marks of all the students of a class with roll number in CSV file. It is needed to calculate the percentage of marks of students in the hive. Given: Download the sample CSV file which have 7 columns, 1st column is Roll no and other 6 columnsRead More →
Requirement You have one table in hive, and it is needed to process the data of that hive table using pig. To load data directly from the file we generally use PigStorage(), but to load data from hive table we need different loading function. Let’s go into detail step byRead More →
Requirement Assume you have the hive table named as reports. It is required to process this dataset in spark. Once we have data of hive table in the Spark data frame, we can further transform it as per the business needs. So let’s try to load hive table in theRead More →
Requirement Suppose the source data is in a file. The file format is a text format. The requirement is to load the text file into a hive table using Spark. In addition to this, read the data from the hive table using Spark. Therefore, let’s break the task into sub-tasks:Read More →
Requirement In the last post, we have demonstrated how to load JSON data in Hive non-partitioned table. This time we are having the same sample JSON data. The requirement is to load JSON Data into Hive Partitioned table using Spark. The hive table will be partitioned by some column(s). TheRead More →
Requirement You have one hive script which is expecting some variables. The variables need to be passed from a shell script. Say the name of hive script is daily_audit.hql. It is expecting three variables which are as follows: • schema • tablename • total_emp Solution Step 1: Hive Script Let’sRead More →
Requirement: Generally we receive data from different sources which usually have different types of date formats. When we create a hive table on top of these data, it becomes necessary to convert them into date format which is supported by hive. Hive support yyyy-MM-dd date format. So output format ofRead More →
Requirement You have one hive table named as infostore which is present in bdp schema. One more application is connected to your application, but it is not allowed to take the data from hive table due to security reasons. It is required to send the data of infostore table intoRead More →
Requirement Suppose there is a source data which is in JSON format. The requirement is to load JSON data into Hive non-partitioned table using Spark. Let’s break the requirement into two tasks: Load JSON data in spark data frame and read it Store into hive non-partition table Components Involved ToRead More →
Requirement Suppose there is a source data, which is required to store in the hive partitioned table. So our requirement is to store the data in the hive table with static and dynamic partitions. With an understanding of partitioning in the hive, we will see where to use the staticRead More →
Requirement Suppose you are having an XML formatted data file. This file contains some empty tag. The requirement is to parse XML data in Hive and assign any default value to the empty tags. Components Involved Hive Maven Java Solution There are many solutions for parsing XML data into hiveRead More →
Requirement Suppose, you have one table in hive with one column and you want to split this column into multiple columns and then store the results into another Hive table. Solution Assume the name of hive table is “transact_tbl” and it has one column named as “connections”, and values in connectionsRead More →
