Requirement In our Read JSON file in Spark post, we have read a simple JSON file into a Spark Dataframe. In this post, we are moving to handle an advanced JSON data type. We will read nested JSON in spark Dataframe. Sample Data We are going to use below sampleRead More →
Requirement In this post, we are going to read a JSON file using Spark and then load it into a Delta table in Databricks. Solution We can use the below sample data for the exercise. In our case, we have placed this file is located in FilteStore in the DatabricksRead More →
Requirement In this post, we will learn how to convert a table’s schema into a Data Frame in Spark. Sample Data empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499 ALLEN SALESMAN 7698 2/20/1981 1600 30 HYDERABAD 9521 WARD SALESMAN 7698 2/22/1981Read More →
Requirement When we ingest data from source to Hadoop data lake, we used to add some additional columns with the existing data source. These columns basically help to validate and analyze the data. So, in this post, we will walk through how we can add some additional columns with theRead More →
Requirement Suppose there is a source data which is in JSON format. The requirement is to load JSON data into Hive non-partitioned table using Spark. Let’s break the requirement into two tasks: Load JSON data in spark data frame and read it Store into hive non-partition table Components Involved ToRead More →
