In this post, I will show you how to read data from Cosmos DB using Spark in Databricks. This is a common scenario for many data pipelines that need to process data from a NoSQL database like Cosmos DB. Requirement Let’s say you have a Cosmos DB account with aRead More →
Requirement In this, we are going to read and load TSV file data in Spark Dataframe. The TSV file contains data with a tab separator. Sample Data Let’s use the same sample data: empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499Read More →
Requirement In this post, we will perform a task to read an Excel file, process the data, and store the data in a Delta Table in Databricks. Sample Data We will take the below sample data for this exercise: empno ename designation manager hire_date sal deptno location 9369 SMITH CLERKRead More →
In this post, we will walkthrough of the Databricks Notebook. This is the code base area in Databricks. Overview The Databricks Notebook is a kind of document which keeps all the commands, visualization in a cell. You can either create a cell for the entire code or can keep anRead More →
1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
Requirement In the last post, we have learned how to create Delta Table from Path in Databricks. In this post, we will learn how to create Delta Table with the partition in Databricks. Solution The partition is basically split the data and then stored. You can learn more about theRead More →
Requirement In the last post, we have learned how to create Delta Table in Databricks. In this post, we will learn how to create Delta Table from Path in Databricks. Solution Let’s first understand what is the use of creating a Delta table with Path. Using this, the Delta tableRead More →
Requirement In our previous post, we have learned about Delta Lake and Delta Table in Databricks. In this post, we are going to create a Delta table with the schema. Solution For creating a Delta table, below is the template: CREATE TABLE <table_name> ( <column name> <data type>, <column name>Read More →
In our last post, we have learned about Delta Lake in Databricks. In this post, we are going to learn about Delta Table in Databricks. Overview The delta table in Databricks is a table created in Delta Lake. The delta table provides flexibility to performed CRUD operations. Features There areRead More →
In this post, we will learn about Delta Lake in Databricks. In addition to this, will see what are the benefits of using Delta Lake. Overview of Delta Lake Delta Lake is an open-source storage layer that provides ACID transactions. It is provided by Databricks and runs on top ofRead More →
Requirement Let’s say there is a system which used to extract the data from any source (can be Databases, Rest API, etc.) and dumping into Azure Data Lake Storage aka. ADLS Gen2 storage. Now, we want to access and read these files in Spark for further processing for our businessRead More →
Requirement Let’s say there is a system which used to extract the data from any source (can be Databases, Rest API, etc.) and dumping into Azure Data Lake Storage aka. ADLS Gen2 storage. Now, we want to access and read these files in Spark for further processing for our businessRead More →
Requirement In this post, we are going to create a mount point in Azure Databricks to access the Azure Datalake data. This is a one-time activity. Once we create the mount point of blob storage, we can directly use this mount point to access the files. Earlier, in one ofRead More →
Requirement Let’s say you have applications and wanted to access or modify the resource. In such scenarios, we can create an identity for the app and use this identity. This identity is known as Service Principal. Now, how to know when you really need this. Suppose, you are working withRead More →
Requirement We have data in Azure Data Lake (blob storage). We want to read and process these data using Spark in Databricks. Before reading and processing the data, it is required to access the Azure Data Lake. In this post, we are going to create a mount point in AzureRead More →
