Q.1 There is a json file with following content :- {“dept_id”:101,”e_id”:[10101,10102,10103]} {“dept_id”:102,”e_id”:[10201,10202]} And data is loaded into spark dataframe say mydf, having below dtypes dept_id: bigint, e_id: array<bigint> What will be the best way to get the e_id individually with dept_id ? Answer : we can use the explode functionRead More →
Requirement: You have sample dataframe and you want to delete some columns from it. Solution: Step 1: Sample Dataframe use below command: spark-shell Note: I am using spark 2.3 version. To Create a sample dataframe , Please refer Create-a-spark-dataframe-from-sample-data After following above post ,you can see that students dataframeRead More →
Requirement: You have sample dataframe which you want to load in to parquet files using scala. Solution: Step 1: Sample Dataframe use below command: spark-shell Note: I am using spark 2.3 version. To Create a sample dataframe , Please refer Create-a-spark-dataframe-from-sample-data After following above post ,you can see thatRead More →
