1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
https://www.youtube.com/watch?v=FG1rpwnVR6U Step 1 : Login to hive using below command: hive Step 2 : Suppose you want to create a schema named as mydb1 , use below command create schema mydb1; Step 3 : To use this schema ,use below command : use mydb1; Step 4 : To see allRead More →
Requirement You have comma separated file and you want to create an external table in the hive on top of it (need to load CSV file in hive), then follow the below steps. Solution Step 1: Sample CSV File Create a sample CSV file named as sample_1.csv. You can downloadRead More →
Requirement You have one hive table named as infostore which is present in bdp schema. It is needed to get the data into Excel file. Solution. Let’s say the location where output file should present is /root/local_bdp/posts/export-hive-data-into-file Step 1: Create Output directory mkdir /root/local_bdp/posts/export-hive-data-into-file/output Step 2: Go to hive CLIRead More →
1.There are 50 columns in one spark data frame say df.it is needed to cast all the columns into string. But to make the code more generic. It is not recommended to cast individual columns by writing column name.How would you achieve it in spark using scala? Answer : AsRead More →
1. Let’s say a Hive table is created as an external table. If we drop the table, will the data be accessible? Answer: The data will be accessible even if the table gets dropped. We can get the data from the table’s HDFS location. 2. A Hive table is createdRead More →
Requirement You have comma separated(CSV) file and you want to create Parquet table in hive on top of it, then follow below mentioned steps. Solution Step 1: Sample CSV File Create a sample CSV file named as sample_1.csv file. download from here (You can skip this step if you alreadyRead More →
What is bucketing and what is the use of it? Answer: Bucketing is an optimization technique which is used to cluster the datasets into more manageable parts, which helps to optimize the query performance. Check the below post with the example. https://bigdataprogrammers.com/bucketing-in-hive/ What is Over Partitioning and what is theRead More →
What is Hive and why it is useful? Hive is a data warehouse application where data gets stored in the structure format. It is used to querying and managing large datasets. It provides a SQL-like interface to access the data which is also called HiveQL(HQL). What is the advantage ofRead More →
Requirement You have two table named as A and B. and you want to perform all types of join in hive . It will help you to understand, how join works in hive. Solution Step 1: Input Files Download file and from here. And place them into a local directory. FileRead More →
#execute a query in Silent(S)Mode hive -S -e “your hive query” #execute a query hive -e “your hive query” #show current database hive>set hive.cli.print.current.db=true; #show all column names for a table hive>set hive.cli.print.header=true; #to run all pre required commands in one go 1.create .hiverc fileRead More →
Requirement There is an uncertain number of columns present in the hive table. Sometimes a table can have many numbers of columns and sometimes it can have few numbers of columns. If we want the value of all the columns from the table, then there is no any challenge asRead More →
Requirement In this post, we are going to understand what is hive_default_partition in hive and why it gets created. Components Involved HDFS HIVE Sample Data We will use below sample data for the task. This is the sample data of employee details. Some employees are the member of company’s sportsRead More →
Requirement Suppose we are having a hive partition table. This table is partitioned by the year of joining. Our requirement is to drop multiple partitions in hive. Components Involved Hive HDFS Sample Data Let’s say we are having given sample data: Here, 1 record belongs to 1 partition as weRead More →
Requirement There are two files which contain employee’s basic information. One file store employee’s details who have joined in the year of 2012 and another is for the employees who have joined in the year of 2013. Now, we want to load files into hive partitioned table which is partitionedRead More →
