1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
Requirement In this post, we are going to learn how to create an empty dataframe in Spark with and without schema. Prerequisite Spark 2.x or above Solution We will see create an empty DataFrame with different approaches: PART I: Empty DataFrame with Schema Approach 1:Using createDataFrame Function import org.apache.spark.sql.types.{StructType, StructField,Read More →
Requirement In this post, we will learn how to convert a table’s schema into a Data Frame in Spark. Sample Data empno ename designation manager hire_date sal deptno location 9369 SMITH CLERK 7902 12/17/1980 800 20 BANGALORE 9499 ALLEN SALESMAN 7698 2/20/1981 1600 30 HYDERABAD 9521 WARD SALESMAN 7698 2/22/1981Read More →
