Requirement Suppose we have a process that consumes data from upstream. This data includes both new and updated information. We now need to consume and ingest this information into the table in the same manner. It means we have to insert all the new data and update the modified data.Read More →
Requirement In the last post, we have imported the CSV file and created a delta table in Databricks. In this post, we are extending the same exercise with the Partition. Partitioning is a very important concept when it comes to storing the data in tables. In this post, we areRead More →
Requirement In this post, we are going to get the last modified date of a file in Spark using the File System. It helps while processing the data based on the last modified date time. Solution Here, let’s assume your files are available in hdfs or any storage. We willRead More →
Requirement : There is a Map in scala which is being used by one of your application ,which have all the configurations for your spark app. you need to remove some of conf in Map. Solution : Step 1: Creation of Map Map is the combination of key value pair. Read More →
Print,Println,Printf Go to Spark-shell or choose environment of your choice to run scala code. Write below code : print(“hello data engineers”) If you type print and press the tab ,you can see three functions ,Related to print ,these are print,printf and println. like above line, it just print whatever isRead More →
Requirement: You have sample dataframe and you want to delete some columns from it. Solution: Step 1: Sample Dataframe use below command: spark-shell Note: I am using spark 2.3 version. To Create a sample dataframe , Please refer Create-a-spark-dataframe-from-sample-data After following above post ,you can see that students dataframeRead More →
Requirement: You have sample dataframe which you want to load in to parquet files using scala. Solution: Step 1: Sample Dataframe use below command: spark-shell Note: I am using spark 2.3 version. To Create a sample dataframe , Please refer Create-a-spark-dataframe-from-sample-data After following above post ,you can see thatRead More →
Requirement The UDF is a user-defined function. As its name indicate, a user can create a custom function and used it wherever required. We do create UDF when the existing build-in functions not available or not able to fulfill the requirement. Sample Data empno ename designation manager hire_date sal deptnoRead More →
Requirement Let’s say we are getting data from multiple sources, but we need to ingest these data into a single target table. These data can have different schemas. We want to merge these data and load/save it into a table. Sample Data Emp_data1.csv empno,ename,designation,manager,hire_date,sal,deptno,location 9369,SMITH,CLERK,7902,12/17/1980,800,20,BANGALORE 9499,ALLEN,SALESMAN,7698,2/20/1981,1600,30,HYDERABAD 9521,WARD,SALESMAN,7698,2/22/1981,1250,30,PUNE 9566,TURNER,MANAGER,7839,4/2/1981,2975,20,MUMBAI 9654,MARTIN,SALESMAN,7698,9/28/1981,1250,30,CHENNAI 9369,SMITH,CLERK,7902,12/17/1980,800,20,KOLKATARead More →
Requirement : To Load application properties from text file. In this tutorial we will learn how to load properties or configs for any spark application. Solution : Step 1 : Preparation of required configs Many times parameters for any application vary from environment to environment. Or two run multiple instanceRead More →
Requirement : You have parquet file(s) present in the hdfs location. And you need to load the data into the spark dataframe. Solution : Step 1 : Input files (parquet format) Here we are assuming you already have files in any hdfs directory in parquet format. In case if youRead More →
Don’t miss the tutorial on Top Big data courses on Udemy you should BuyRead More →
Requirement You have two table named as A and B. and you want to perform all types of join in spark using scala. It will help you to understand, how join works in spark scala. Solution Step 1: Input Files Download file and from here. And place them into a localRead More →
Requirement Suppose we have a dataset which is in CSV format. We want to read the file in spark using Scala. So the requirement is to create a spark application which read CSV file in spark data frame using Scala. Components Involved Following components are involved: Spark RDD/Data Frame ScalaRead More →
