In this post, I will show you how to read data from Cosmos DB using Spark in Databricks. This is a common scenario for many data pipelines that need to process data from a NoSQL database like Cosmos DB. Requirement Let’s say you have a Cosmos DB account with aRead More →
Requirement Assume you have a large amount of data. You frequently perform ad-hoc analyses on this dataset. However, retrieving the results of your query takes time. In order to achieve a fast result, we must perform some optimizations. Bloom Filter Index is one of the optimizations available. In this post,Read More →
Requirement In our previous post, we learned how to create Delta tables and Parquet tables. The purpose of this post is to compare Delta vs Parquet Tables. Solution Both format tables are helpful. It is all about what your requirement is. Below are the following details about Delta vs ParquetRead More →
Requirement The purpose of this post is to explore all of the commands available in the Delta table of Databricks. We used some of the commands in our previous post. If you checked it, you might have seen them. Our next step will be to list all the commands andRead More →
Requirement In the last post, We covered how to vacuum a delta table in our previous post. In this article, we’ll talk about Auto Optimize Delta table in Databricks. Solution Before moving on to the solution, let’s first clarify what Auto Optimize is and how it functions before talking aboutRead More →
Requirement As an example, let’s say you have created a Delta table in Databricks. The Delta table is used to keep track of versions of the data. It is a good idea to remove older versions of data from delta tables. This is where the vacuum comes into play. InRead More →
Requirement In our previous post, we learned about the Temporary view in Databricks. In this post, we are going to learn about Global View in Databricks or can say in Spark. We will also see when to create a temporary view and how to access it. Solution For this exercise,Read More →
Requirement In this post, we are going to learn about Temporary View in Databricks or can say in Spark. We will also see when to create temporary view and how to access it. Solution For this exercise, we are going to use the below sample dataset: empno ename designation managerRead More →
Requirement Suppose we have a process that consumes data from upstream. This data includes both new and updated information. We now need to consume and ingest this information into the table in the same manner. It means we have to insert all the new data and update the modified data.Read More →
Requirement Let’s say you have DELTA tables in databricks, which point to data available in the data lake in Delta format. You now need to do some analysis on this data but from a different databricks workspace. Depending on your requirements, you can create a table in the newly createdRead More →
Requirement In this post, we will learn how to store the processed dataframe to delta table in databricks with overwrite mode. The overwrite mode delete the existing data of the table and load only new records. Solution For this exercise, we will use the below data: empno ename designation managerRead More →
Requirement In this post, we will learn how to store the processed dataframe to delta table in databricks in append mode. The append mode helps when we need to store the new data into an existing table without impacting old data in the table. Solution For this exercise, we willRead More →
Requirement In our Read JSON file in Spark post, we have read a simple JSON file into a Spark Dataframe. In this post, we are moving to handle an advanced JSON data type. We will read nested JSON in spark Dataframe. Sample Data We are going to use below sampleRead More →
Requirement In this, we are going to read and load TSV file data in Spark Dataframe. The TSV file contains data with a tab separator. Sample Data Let’s use the same sample data: empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499Read More →
Requirement In this post, we will perform a task to read an Excel file, process the data, and store the data in a Delta Table in Databricks. Sample Data We will take the below sample data for this exercise: empno ename designation manager hire_date sal deptno location 9369 SMITH CLERKRead More →
