mysql

Merging Two Dataframes in Spark

Tagged: dataframe, merge, mysql, Spark, spark sql, union

Requirement Let’s say we are getting data from two different sources (i.e. RDBMS table and File), and we need to merge these data into a single dataframe. Both the source data having the same schema. Sample Data MySQL Table Data: empno,ename,designation,manager,hire_date,sal,deptno 7369,SMITH,CLERK,7902,12/17/1980,800,20 7499,ALLEN,SALESMAN,7698,2/20/1981,1600,30 7521,WARD,SALESMAN,7698,2/22/1981,1250,30 7566,TURNER,MANAGER,7839,4/2/1981,2975,20 7654,MARTIN,SALESMAN,7698,9/28/1981,1250,30 CSV File Data: empno,ename,designation,manager,hire_date,sal,deptnoRead More →

Load data from MySQL in Spark using JDBC

Tagged: jdbc, mysql, Spark, spark dataframe, spark sql

Requirement We have data in an RDBMS table say MySQL table. The requirement is to load data from MySQL in Spark using JDBC connection. Sample Data We will use below sample data which contains the basic details of an employee like employee number, employee name, designation, manager, hire date, salaryRead More →

Import data from MySQL to HBase using Sqoop

Tagged: hbase, import, ingestion, mysql, sqoop

Requirement Suppose, we have data in a table EMPLOYEE in MySQL database. We want to ingest same data in NoSQL HBase table. So, the requirement is to import data from MySQL to HBase using Sqoop. Components Involved Following components are involved in this requirement MySQL – Source for the Ingestion.Read More →

Import data from MySQL into Hive using Sqoop

Tagged: hive, mysql, sqoop

Requirement Suppose, there is a table named EMPLOYEE in MySQL database. We want this table data in Hadoop ecosystem. So, the requirement is to import data from MySQL into Hive using Sqoop. Once data is available in Hive, we can process it. Components Involved In order to achieve the requirement,Read More →