Requirement: You have sample data of some students and you want to create a dataframe to perform some operations. Given: Sample data: 101, “alex”,88.56 102, “john”,68.32 103, “peter”,75.62 104, “jeff”,92.67 105, “mathew”,89.56 106, “alan”,72.57 107, “steve”,96.12 108, “mark”,98.45 109, “adam”,76.25 109, “david”,78.45 Solution: Step 1: go to spark shell useRead More →
Requirement: You have a dataframe which you want to save into hive table for future use. But you do not want to create the hive table first. Instead you need to save dataframe directly to the hive. Given: Sample data: 101, “alex”,88.56 102, “john”,68.32 103, “peter”,75.62 104, “jeff”,92.67 105, “mathew”,89.56Read More →
Requirement When we ingest data from source to Hadoop data lake, we used to add some additional columns with the existing data source. These columns basically help to validate and analyze the data. So, in this post, we will walk through how we can add some additional columns with theRead More →
Requirement Let’s say we have a set of data which is in JSON format. The file may contain data either in a single line or in a multi-line. The requirement is to process these data using the Spark data frame. In addition to this, we will also see how toRead More →
Requirement The spark-shell is an environment where we can run the spark scala code and see the output on the console for every execution of line of the code. It is more interactive environment. But, when we have more line of code, we prefer to write in a file andRead More →
Don’t miss the tutorial on Top Big data courses on Udemy you should BuyRead More →
