Need an expert ? You are more than welcomed to leave your contact info and we will be in touch shortly.
team@bigdataprogrammers.com
2017-03-10
Secondary Navigation Menu
Remove Header of CSV File in hive
Requirement You have one CSV file which is present at Hdfs location, and you want to create a hive layer on top of this data, but CSV file is having two headers on top of it, and you don’t want …
Start Learning
Start Learning
csv, Hive, hive, remove header
Load CSV file in hive
Requirement If you have comma separated file and you want to create a table in the hive on top of it (need to load CSV file in hive). Solution Step 1: Sample CSV File Create a sample CSV file named …
Start Learning
Start Learning
Load csv file in pig
Requirement Assume that you want to load CSV file in pig and output of pig should be pipe delimited and should be stored in one directory. Solution Please follow the below steps:- Step 1: Sample CSV file Create a sample …
Start Learning
Start Learning
Split one column into multiple columns in hive
Requirement You have one table in hive with one column and you want to split this column into multiple columns and store the results into another hive table. Solution Assume the name of hive table is “transact_tbl” and it has one …
Start Learning
Start Learning
Parse XML data in Hive
Requirement Suppose you are having an XML formatted data file. This source file contains some empty tag. The requirement is to parse XML data in Hive and read data with handling some tag which is empty in the source data …
Start Learning
Start Learning
Partitioning in Hive
Requirement Suppose there is a source data, which is required to store in hive partition table. So our requirement is to store the data in the hive table with static and dynamic partition. With an understanding of partition in the …
Start Learning
Start Learning
Load JSON Data in Hive non-partitioned table using Spark
Requirement Suppose there is a source data which is in JSON format. The requirement is to load JSON data in Hive non-partitioned table using Spark. Let’s break the requirement into two task: Load JSON data in spark data frame and …
Start Learning
Start Learning
Export hive data into file
Requirement You have one hive table named as infostore which is present in bdp schema.one more application is connected to your application, but it is not allowed to take the data from hive table due to security reasons. And it …
Start Learning
Start Learning
Load CSV file into hive ORC table
Requirement You have comma separated file and you want to create an ORC formatted table in hive on top of it, then please follow below mentioned steps. Solution Step 1: Sample CSV File Create a sample CSV file named as …
Start Learning
Start Learning
Join in pig
Requirement You have two table named as A and B. and you want to perform all types of join in pig Latin. It will help you to understand, how join works in pig. Solution Step 1: Input Files Download file …
Start Learning
Start Learning
String to Date conversion in hive
Requirement: Generally we receive data from different sources which usually have different types of date formats. When we create a hive table on top of these data, it becomes necessary to convert them into date format which is supported by …
Start Learning
Start Learning
Filter records in pig
Requirement: In source data, you have user’s information of mobile connection type and Id.You have four type of possible connection “POSTP, PREP, CLS, PEND” .But it is required to get Id of only those users whose connection type is in …
Start Learning
Start Learning
Pass variables from shell script to hive script
Requirement You have one hive script which is expecting some variables which need to be passed from a shell script.Say name of hive scripts is daily_audit.hql .it is expecting three variables which are as follows • schema • tablename • …
Start Learning
Start Learning
Hive, hive, parameters, shell, shell
Load JSON Data into Hive Partitioned table using PySpark
Requirement In the last post, we have demonstrated how to load JSON data in Hive non-partitioned table. This time having the same sample JSON data. The requirement is to load JSON Data into Hive Partitioned table using Spark. The hive …
Start Learning
Start Learning
Load Text file into Hive Table Using Spark
Requirement Suppose the source data is in a file. The file format is a text format. The requirement is to load the text file into hive table using Spark. In addition to this, read the data from the hive table …
Start Learning
Start Learning