Requirement In this post, we will learn how to store the processed dataframe to delta table in databricks with overwrite mode. The overwrite mode delete the existing data of the table and load only new records. Solution For this exercise, we will use the below data: empno ename designation managerRead More →
Requirement In this post, we will learn how to store the processed dataframe to delta table in databricks in append mode. The append mode helps when we need to store the new data into an existing table without impacting old data in the table. Solution For this exercise, we willRead More →
Requirement In the last post, we have imported the CSV file and created a delta table in Databricks. In this post, we are extending the same exercise with the Partition. Partitioning is a very important concept when it comes to storing the data in tables. In this post, we areRead More →
Requirement In the last post, we have imported the CSV file and created a table using the UI interface in Databricks. In this post, we are going to create a delta table from a CSV file using Spark in databricks. Solution Let’s use the same sample data: empno ename designationRead More →
Requirement In the last post, we have imported the CSV file and created a table using the UI interface in Databricks. In this post, we are going to create a delta table from a CSV file using Spark in databricks. Solution Let’s use the same sample data: empno ename designationRead More →
Requirement Suppose you have received a data set. It is in a CSV format. Now, you want to analyze this data using spark in databricks. In this post, we will see how to import data into DBFS (Databricks File System) from a local system for analysis. Solution Let’s take theRead More →
1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
Requirement The CSV file format is a very common file format used in many applications. Sometimes, it contains data with some additional behavior also. For example comma within the value, quotes, multiline, etc. In order to handle this additional behavior, spark provides options to handle it while processing the data.Read More →
Requirement Let’s say we are getting data from multiple sources, but we need to ingest these data into a single target table. These data can have different schemas. We want to merge these data and load/save it into a table. Sample Data Emp_data1.csv empno,ename,designation,manager,hire_date,sal,deptno,location 9369,SMITH,CLERK,7902,12/17/1980,800,20,BANGALORE 9499,ALLEN,SALESMAN,7698,2/20/1981,1600,30,HYDERABAD 9521,WARD,SALESMAN,7698,2/22/1981,1250,30,PUNE 9566,TURNER,MANAGER,7839,4/2/1981,2975,20,MUMBAI 9654,MARTIN,SALESMAN,7698,9/28/1981,1250,30,CHENNAI 9369,SMITH,CLERK,7902,12/17/1980,800,20,KOLKATARead More →
Requirement In this post, we are having data in a CSV file. This file contains basic information about Employees. We want to import CSV data into HBase table. Components Involved HDFS HBASE Sample Data Our sample data looks like below: 7369,SMITH,CLERK,7902,12/17/1980,800,20 7499,ALLEN,SALESMAN,7698,2/20/1981,1600,30 7521,WARD,SALESMAN,7698,2/22/1981,1250,30 7566,TURNER,MANAGER,7839,4/2/1981,2975,20 7654,MARTIN,SALESMAN,7698,9/28/1981,1250,30 7698,MILLER,MANAGER,7839,5/1/1981,2850,30 7782,CLARK,MANAGER,7839,6/9/1981,2450,10 7788,SCOTT,ANALYST,7566,12/9/1982,3000,20 7839,KING,PRESIDENT,NULL,11/17/1981,5000,10 7844,TURNER,SALESMAN,7698,9/8/1981,1500,30Read More →
Requirement Suppose we have a dataset which is in CSV format. We want to read the file in spark using Scala. So the requirement is to create a spark application which read CSV file in spark data frame using Scala. Components Involved Following components are involved: Spark RDD/Data Frame ScalaRead More →
Requirement You have marks of all the students of a class with roll number in CSV file. It is needed to calculate the percentage of marks of students in the hive. Given: Download the sample CSV file which have 7 columns, 1st column is Roll no and other 6 columnsRead More →
Requirement You have one CSV file which is present at Hdfs location, and you want to create a hive layer on top of this data, but CSV file is having two headers on top of it, and you don’t want them to come into your hive table, so let’s solveRead More →
