Find max value in Spark RDD using Scala


Suppose we are having a source file, which contains basic information about Employees like employee number, employee name, designation, salary etc. The requirement is to find max value in spark RDD using Scala. With this requirement, we will find out the maximum salary, the second maximum salary of an employee.

Components Involved

  • Spark RDD
  • Scala
  • IntelliJ
  • SBT

Sample Data

Below is the sample data for this requirement. The source file is in text format with a comma (‘,’) separated.

  1. empno,ename,designation,manager,hire_date,sal,deptno
  2. 7369,SMITH,CLERK,7902,1980-12-17,800.00,20
  3. 7499,ALLEN,SALESMAN,7698,1981-02-20,1600.00,30
  4. 7521,WARD,SALESMAN,7698,1981-02-22,1250.00,30
  5. 7566,JONES,MANAGER,7839,1981-04-02,2975.00,20
  6. 7654,MARTIN,SALESMAN,7698,1981-09-28,1250.00,30
  7. 7698,BLAKE,MANAGER,7839,1981-05-01,2850.00,30
  8. 7782,CLARK,MANAGER,7839,1981-06-09,2450.00,10
  9. 7788,SCOTT,ANALYST,7566,1982-12-09,3000.00,20
  10. 7839,KING,PRESIDENT,NULL,1981-11-17,5000.00,10
  11. 7844,TURNER,SALESMAN,7698,1981-09-08,1500.00,30
  12. 7876,ADAMS,CLERK,7788,1983-01-12,1100.00,20
  13. 7900,JAMES,CLERK,7698,1981-12-03,950.00,30
  14. 7902,FORD,ANALYST,7566,1981-12-03,3000.00,20
  15. 7934,MILLER,CLERK,7782,1982-01-23,1300.00,10

Download the data from here:



We will create a spark application with the MaxValueInSpark using IntelliJ and SBT. If you want to setup IntelliJ on your system, then you can check this post.

Step 1: Create Spark Application

First of all, open IntelliJ. Once it opened, Go to File -> New -> Project -> Choose SBT

Click next and provide all the details like Project name and choose scala version. In my case, I have given project name MaxValueInSpark and have selected 2.10.4 as scala version.

Step 2: Resolve Dependency

Adding below dependency:

  1. libraryDependencies++=Seq(
  2. "org.apache.spark"%"spark-core_2.10"%"1.6.0",
  3. "org.apache.spark"%"spark-sql_2.10"%"1.6.0"
  4. )

It will download all the required packages.

Step 3: Write Code

In this step, we will write the code to get the maximum salary, minimum salary, second maximum salary, second minimum salary, salary with the employee name. Let’s start with loading dataset in RDD.

  1. valemp_data=sc.textFile("src\\main\\resources\\emp_data.txt")

In this line of code, we are loading employee data which is available at local path src\\main\\resources. It will return an RDD.

Once data has been loaded, read the first record of the file using function first

  1. valemp_header=emp_data.first()
  2. println(emp_header)


first() – Return the first record.

The first record might be the header of the file. So let’s remove it from out RDD.

  1. valemp_data_without_header=emp_data.filter(line=>!line.equals(emp_header))

Now, we have another RDD which is without the header. Let’s check how many partitions have been created.

  1. println("No.ofpartition="+emp_data_without_header.partitions.size)


partitions – It will return the number of partitions of a RDD.

Here, I have used partitions function which is a predefined function which returns a number of partitions of RDD.
In order to find max salary, we are going to use two different approaches. One approach will use the max function which will give us max value.

  2. println("Highestsalaty:"+emp_salary_list.max())

Here, at first, I have split the record and taken the only salary from the RDD. On the list of salary data, I have used max function which returned the max value among them.
The second approach to sortBy function.

  1. valmax_salary=emp_salary_list.distinct.sortBy(x=>x.toDouble,false,1)
  2. print(max_salary.take(1).foreach(println))


sortBy : This function takes three argument (value, ascending = true, number of partition). By default, sort happens in ascending order.

Here, at first, I have taken only distinct value from the salary. After that sorted the records in descending order and taken the 1st record which is the max value. As we were having 2 partitions, if we don’t provide partition 1, then it will sort in each partition.
Similarly, if we make sorting true instead of false, It will return minimum salary.

  1. valmin_salary=emp_salary_list.distinct.sortBy(x=>x.toDouble,true,1)
  2. print(min_salary.take(1).foreach(println))

Below code snippet give second highest salary.

  1. valsecond_highest_salary=max_salary.zipWithIndex().filter(index=>index._2==1)
  2. print("Secondhighestsalary=",second_highest_salary.foreach(println))


zipWithIndex() : It will zip the RDD with indices. It will provide an index for every record.

Here, we are using zipWithIndex() function which will zip our RDD with index and then we are taking the second record using a filter.

Next, we will find out an employee who has the highest salary.

  2. valmaxSalaryEmployee=salaryWithEmployeeName.groupByKey.takeOrdered(1)(Ordering[Double].reverse.on(_._1))
  3. print(maxSalaryEmployee.foreach(println))


GroupByKey – Return a collection of value for the same key.

TakeOrdered – Return the first N record on the basis of mentioned order.

Here, we have first created a key value with the values of salary and employee name and then have taken the first record by applying for the descending order of the salary.
Once all the code was done, let’s bind together and the code will look like:

  1. import org.apache.spatrk.{SparkConf, SparkContext}
  2. object MaxSalary {
  3.   def main(args: Array[String]): Unit = {
  4.     // Setup configuration and create spark context
  5.     val conf = new SparkConf().setAppName("Emp Dept Assignment").setMaster("local[*]")
  6.     val sc = new SparkContext(conf)
  7.     // Load the source data file into RDD
  8.     val emp_data = sc.textFile("src\\main\\resources\\emp_data.txt")
  9.     println(emp_data.foreach(println))
  10.     // Find first record of the file
  11.     val emp_header = emp_data.first()
  12.     println(emp_header)
  13.     // Remove header from RDD
  14.     val emp_data_without_header = emp_data.filter(line => !line.equals(emp_header))
  15.     println(emp_data_without_header.foreach(println))
  16.     // Get no. of partition of an RDD
  17.     println("No. of partition = " + emp_data_without_header.partitions.size)
  18.     // Find max salary of an employe 1st Other approach
  19.     // Using max function on RDD
  20.     val emp_salary_list ={x => x.split(',')}.map{x => (x(5).toDouble)}
  21.     println("Highest salaty:"+ emp_salary_list.max())
  22.     // Using sortBy function
  23.     val max_salary = emp_salary_list.distinct.sortBy(x => x.toDouble, false, 1)
  24.     print(max_salary.take(1).foreach(println))
  25.     // Find minimum salary
  26.     val min_salary = emp_salary_list.distinct.sortBy(x => x.toDouble, true, 1)
  27.     print(min_salary.take(1).foreach(println))
  28.     // Find Second highest salary
  29.     val second_highest_salary = max_salary.zipWithIndex().filter(index => index._2 == 1).map(_._1)
  30.     print(second_highest_salary.foreach(println))
  31.     // Find second min salary
  32.     val second_min_salary = min_salary.zipWithIndex().filter(index => index._2 == 1).map(_._1)
  33.     print(second_min_salary.foreach(println))
  34.     // Employee who have max salary
  35.     val salaryWithEmployeeName ={x => x.split(',')}.map{x => (x(5).toDouble, x(1))}
  36.     val maxSalaryEmployee = salaryWithEmployeeName.groupByKey.takeOrdered(1)(Ordering[Double].reverse.on(_._1))
  37.     print(maxSalaryEmployee.foreach(println))
  38.   }
  39. }

Step 4: Execute the application

Once code part is done, let’s execute the application. In order to execute the application right click on the script and choose Run

Step 5: Output

You will able to see the output of the application in the console of the IntelliJ.

Wrapping Up

In this post, we have created a spark application with Scala to find out the maximum salary, second maximum salary, minimum salary, second minimum salary from an employee data set. We have seen multiple ways to find out the max and min salary. We have executed in local and validated the output. If you want to download the application then you can download from here. If you don’t have an IntelliJ IDE on your local system, then you try on spark shell with the same code.

Load CSV file into hive AVRO table

Requirement You have comma separated(CSV) file and you want to create Avro table in hive on top of it, then ...
Read More

Load CSV file into hive PARQUET table

Requirement You have comma separated(CSV) file and you want to create Parquet table in hive on top of it, then ...
Read More

Hive Most Asked Interview Questions With Answers – Part II

What is bucketing and what is the use of it? Answer: Bucket is an optimisation technique which is used to ...
Read More
/ hive, hive interview, interview-qa

Spark Interview Questions Part-1

Suppose you have a spark dataframe which contains millions of records. You need to perform multiple actions on it. How ...
Read More

Leave a Reply