Requirement Let’s say we have a data file with a TSV extension. It is the same as the CSV file. What is the difference between CSV and TSV? The difference is separating the data in the file The CSV file stores data separated by “,”, whereas TSV stores data separatedRead More →
Requirement As we received data/files from multiple sources, the chances are high to have issues in the data. Let’s say, we have received a CSV file, and most of the columns are of String data type in the file. We found some data missing in the target table after processingRead More →
Requirement In the last post, we have imported the CSV file and created a table using the UI interface in Databricks. In this post, we are going to create a delta table from a CSV file using Spark in databricks. Solution Let’s use the same sample data: empno ename designationRead More →
Requirement In this post, we are going to learn to create a delta table from the dataframe in Databricks. This scenario comes when we consume data from any file, source database table, etc., at last, we used to have the data in a dataframe. We can store this data inRead More →
In this post, we will walkthrough of the Databricks Notebook. This is the code base area in Databricks. Overview The Databricks Notebook is a kind of document which keeps all the commands, visualization in a cell. You can either create a cell for the entire code or can keep anRead More →
Requirement: There is spark dataframe, in which it is needed to add multiple columns altogether, without writing the withColumn , multiple times, As you are not sure, how many columns would be available. Solution : Step 1: A spark Dataframe Enter into your spark-shell , and create a sample dataframe,Read More →
Requirement In this post, we will learn how to get last element in list of dataframe in spark. Solution Create a dataframe with dummy data: val df = spark.createDataFrame(Seq( (“1100”, “Person1”, “Street1#Location1#City1”, null), (“1200”, “Person2”, “Street2#Location2#City2”, “Contact2”), (“1300”, “Person3”, “Street3#Location3#City3”, null), (“1400”, “Person4”, null, “Contact4”), (“1500”, “Person5”, “Street5#Location5#City5”, null) )).toDF(“id”,Read More →
Requirement In this post, we are going to extract or get column value from Data Frame as List in Spark. Let’s take an example, you have a data frame with some schema and would like to get a list of values of a column for any further process. Solution Let’sRead More →
Requirement In this post, we will learn how to get or extract a value from a row. Whenever we extract a value from a row of a column, we get an object as a result. For example, if we have a data frame with personal details like id, name, location,Read More →
Requirement In this post, we will see how to convert column type in spark dataframe. Let’s assume a scenario, we used to get a CSV file from the source. As it is a CSV file, we will see mostly the datatype of the field would be String. In this scenario,Read More →
Requirement In this post, we are going to learn about how to compare data frames data in Spark. Let’s see a scenario where your daily job consumes data from the source system and append it into the target table as it is a Delta/Incremental load. There is a possibility toRead More →
1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
Requirement In the last post, we have learned how to create Delta Table from Path in Databricks. In this post, we will learn how to create Delta Table with the partition in Databricks. Solution The partition is basically split the data and then stored. You can learn more about theRead More →
Requirement In the last post, we have learned how to create Delta Table in Databricks. In this post, we will learn how to create Delta Table from Path in Databricks. Solution Let’s first understand what is the use of creating a Delta table with Path. Using this, the Delta tableRead More →
Requirement Let’s say there is a system which used to extract the data from any source (can be Databases, Rest API, etc.) and dumping into Azure Data Lake Storage aka. ADLS Gen2 storage. Now, we want to access and read these files in Spark for further processing for our businessRead More →
