December 2017

Import CSV data into HBase

Requirement In this post, we are having data in a CSV file. This file contains basic information about Employees. We want to import CSV data into HBase table. Components Involved HDFS HBASE Sample Data Our sample data looks like below: 7369,SMITH,CLERK,7902,12/17/1980,800,20 7499,ALLEN,SALESMAN,7698,2/20/1981,1600,30 7521,WARD,SALESMAN,7698,2/22/1981,1250,30 7566,TURNER,MANAGER,7839,4/2/1981,2975,20 7654,MARTIN,SALESMAN,7698,9/28/1981,1250,30 7698,MILLER,MANAGER,7839,5/1/1981,2850,30 7782,CLARK,MANAGER,7839,6/9/1981,2450,10 7788,SCOTT,ANALYST,7566,12/9/1982,3000,20 7839,KING,PRESIDENT,NULL,11/17/1981,5000,10 7844,TURNER,SALESMAN,7698,9/8/1981,1500,30Read More →

Data migration from Hive to HBase

Tagged: data migration, external table, hbase, hive

Requirement Suppose we have data in Hive table. We want the same data into HBase table. So, our requirement is to migrate the data from Hive to HBase table. Components Involved Hive – Source table HBase – Target Table Solution We cannot load data directly into HBase table from theRead More →

Import data from MySQL to HBase using Sqoop

Tagged: hbase, import, ingestion, mysql, sqoop

Requirement Suppose, we have data in a table EMPLOYEE in MySQL database. We want to ingest same data in NoSQL HBase table. So, the requirement is to import data from MySQL to HBase using Sqoop. Components Involved Following components are involved in this requirement MySQL – Source for the Ingestion.Read More →

Import data from MySQL into Hive using Sqoop

Tagged: hive, mysql, sqoop

Requirement Suppose, there is a table named EMPLOYEE in MySQL database. We want this table data in Hadoop ecosystem. So, the requirement is to import data from MySQL into Hive using Sqoop. Once data is available in Hive, we can process it. Components Involved In order to achieve the requirement,Read More →

Read CSV file in Spark Scala

Tagged: case-class, csv, dataframe, IntelliJ, sbt, scala, Spark

Requirement Suppose we have a dataset which is in CSV format. We want to read the file in spark using Scala. So the requirement is to create a spark application which read CSV file in spark data frame using Scala. Components Involved Following components are involved: Spark RDD/Data Frame ScalaRead More →