Requirement In this post, we will convert RDD to Dataframe in Pyspark. Solution Let’s create dummy data and load it into an RDD. After that, we will convert RDD to Dataframe with a defined schema. # Create RDD empData = [(7389, “SMITH”, “CLEARK”, 9902, “2010-12-17”, 8000.00, 20), (7499, “ALLEN”, “SALESMAN”,Read More →
Requirement In this post, we will see how to print RDD content in Pyspark. Solution Let’s take dummy data. We are having 2 rows of Employee data with 7 columns. empData = [(7389, “SMITH”, “CLEARK”, 9902, “2010-12-17”, 8000.00, 20), (7499, “ALLEN”, “SALESMAN”, 9698, “2011-02-20”, 9000.00, 30)] Load the data intoRead More →
Requirement Do you want to explore Spark? Azure provides a cloud service platform named databricks which is built on top of the Spark. In this post, we are going to create a databricks cluster in Azure. Solution Follow the below steps to create the databricks cluster in Azure. Step 1:Read More →
Requirement In the previous post, we have learned about when and how to use SELECT in DataFrame. It is useful when we want to select a column, all columns of a DataFrames. Let’s say we want to add any expression in the query like length, case statement, etc, then SELECTRead More →
Requirement In this post, we will learn how to convert a table’s schema into a Data Frame in Spark. Sample Data empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499 ALLEN SALESMAN 7698 2/20/1981 1600 30 HYDERABAD 9521 WARD SALESMAN 7698 2/22/1981Read More →
Requirement In this post, we will learn how to handle NULL in spark dataframe. There are multiple ways to handle NULL while data processing. We will see how can we do it in Spark DataFrame. Solution Create Dataframe with dummy data val df = spark.createDataFrame(Seq( (1100, “Person1”, “Location1”, null), (1200,Read More →
Requirement In this post, we will learn how to select a specific column value or all the columns in Spark DataFrame with different approaches. Sample Data empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499 ALLEN SALESMAN 7698 2/20/1981 1600 30 HYDERABADRead More →
Requirement : You have parquet file(s) present in the hdfs location. And you need to load the data into the spark dataframe. Solution : Step 1 : Input files (parquet format) Here we are assuming you already have files in any hdfs directory in parquet format. In case if youRead More →
Requirement Let’s say we are getting data from two different sources (i.e. RDBMS table and File), and we need to merge these data into a single dataframe. Both the source data having the same schema. Sample Data MySQL Table Data: empno,ename,designation,manager,hire_date,sal,deptno 7369,SMITH,CLERK,7902,12/17/1980,800,20 7499,ALLEN,SALESMAN,7698,2/20/1981,1600,30 7521,WARD,SALESMAN,7698,2/22/1981,1250,30 7566,TURNER,MANAGER,7839,4/2/1981,2975,20 7654,MARTIN,SALESMAN,7698,9/28/1981,1250,30 CSV File Data: empno,ename,designation,manager,hire_date,sal,deptnoRead More →
Requirement: You have a dataframe which you want to save into hive table for future use. But you do not want to create the hive table first. Instead you need to save dataframe directly to the hive. Given: Sample data: 101, “alex”,88.56 102, “john”,68.32 103, “peter”,75.62 104, “jeff”,92.67 105, “mathew”,89.56Read More →
Requirement Let’s say we have a set of data which is in JSON format. The file may contain data either in a single line or in a multi-line. The requirement is to process these data using the Spark data frame. In addition to this, we will also see how toRead More →
Requirement Let’s take a scenario where we have already loaded data into an RDD/Dataframe. We got the rows data into columns and columns data into rows. The requirement is to transpose the data i.e. change rows into columns and columns into rows. Sample Data We will use below sample data.Read More →
Requirement Suppose we have a dataset which is in CSV format. We want to read the file in spark using Scala. So the requirement is to create a spark application which read CSV file in spark data frame using Scala. Components Involved Following components are involved: Spark RDD/Data Frame ScalaRead More →
Requirement Assume you have the hive table named as reports. It is required to process this dataset in spark. Once we have data of hive table in the Spark data frame, we can further transform it as per the business needs. So let’s try to load hive table in theRead More →
