Shell script tutorials

Shell Script

Pass variables from shell script to hive script

Requirement You have one hive script which is expecting some variables which need to be passed from a shell script.Say

Remove Header of CSV File in hive

Requirement You have one CSV file which is present at Hdfs location, and you want to create a hive layer on top of this data, but CSV file is having two headers on top of it, and you don’t want …
Start Learning

csv, Hive, hive, remove header

Load CSV file in hive

Requirement If you have comma separated file and you want to create a table in the hive on top of it (need to load CSV file in hive). Solution Step 1: Sample CSV File Create a sample CSV file named …
Start Learning

Hive

Load csv file in pig

Requirement Assume that you want to load CSV file in pig and output of pig should be pipe delimited and should be stored in one directory. Solution Please follow the below steps:- Step 1: Sample CSV file Create a sample …
Start Learning

csv file, load, Pig, pig

Split one column into multiple columns in hive

Requirement You have one table in hive with one column and you want to split this column into multiple columns and store the results into another hive table. Solution Assume the name of hive table is “transact_tbl” and it has one …
Start Learning

Hive, hive, hive columns, split

Parse XML data in Hive

Requirement Suppose you are having an XML formatted data file. This source file contains some empty tag. The requirement is to parse XML data in Hive and read data with handling some tag which is empty in the source data …
Start Learning

Hive, hive, load xml, parse xml

Partitioning in Hive

Requirement Suppose there is a source data, which is required to store in hive partition table. So our requirement is to store the data in the hive table with static and dynamic partition. With an understanding of partition in the …
Start Learning

dynamic partition, Hive, hive, partitioning, static partition

Load JSON Data in Hive non-partitioned table using Spark

Requirement Suppose there is a source data which is in JSON format. The requirement is to load JSON data in Hive non-partitioned table using Spark. Let’s break the requirement into two task: Load JSON data in spark data frame and …
Start Learning

Hive, hive, json, pyspark, Spark, spark with python

Export hive data into file

Requirement You have one hive table named as infostore which is present in bdp schema.one more application is connected to your application, but it is not allowed to take the data from hive table due to security reasons. And it …
Start Learning

export, Hive, hive

Load CSV file into hive ORC table

Requirement You have comma separated file and you want to create an ORC formatted table in hive on top of it, then please follow below mentioned steps. Solution Step 1: Sample CSV File Create a sample CSV file named as …
Start Learning

Hive

Join in pig

Requirement You have two table named as A and B. and you want to perform all types of join in pig Latin. It will help you to understand, how join works in pig. Solution Step 1: Input Files Download file …
Start Learning

join, Pig, pig

String to Date conversion in hive

Requirement: Generally we receive data from different sources which usually have different types of date formats. When we create a hive table on top of these data, it becomes necessary to convert them into date format which is supported by …
Start Learning

date conversion, Hive, hive

Filter records in pig

Requirement: In source data, you have user’s information of mobile connection type and Id.You have four type of possible connection “POSTP, PREP, CLS, PEND” .But it is required to get Id of only those users whose connection type is in …
Start Learning

filter, Pig, pig

Pass variables from shell script to hive script

Requirement You have one hive script which is expecting some variables which need to be passed from a shell script.Say name of hive scripts is daily_audit.hql .it is expecting three variables which are as follows • schema • tablename • …
Start Learning

Hive, hive, parameters, shell, shell

Load JSON Data into Hive Partitioned table using PySpark

Requirement In the last post, we have demonstrated how to load JSON data in Hive non-partitioned table. This time having the same sample JSON data. The requirement is to load JSON Data into Hive Partitioned table using Spark. The hive …
Start Learning

Hive, hive, hive partition, json file, load json, python, Spark, spark with python

Load Text file into Hive Table Using Spark

Requirement Suppose the source data is in a file. The file format is a text format. The requirement is to load the text file into hive table using Spark. In addition to this, read the data from the hive table …
Start Learning

Hive, hive, Spark, spark with python