Calculate percentage using pig

Calculate percentage using pig

Requirement

You have marks of all the students of a class with roll number in CSV file, It is needed to calculate the percentage of each student using pig.

Given:

Download the sample CSV file marks which have 7 columns, 1st column is Roll no and other 6 columns are subject1 subject2….subject6.

Solution

Step 1: Loading the sample CSV file marks.csv into HDFS.

I have Local directory named as “calculate-percentage-using-pig” in path “/root/local_bdp/problems/”
So I have kept marks.csv file in that path.

You can see sample data in below screen shot:-

Let’s create HDFS directory using below command

 
 
hadoop fs -mkdir -p hdfs://sandbox.hortonworks.com:8020/user/root/bdp/problems/cal_per_pig/ip/

As you can see “ip” directory is created for input files.
Now we can load file into HDFS using below command

 
 
hadoop fs -put /root/local_bdp/problems/calculate-percentage-using-pig/marks.csv hdfs://sandbox.hortonworks.com:8020/user/root/bdp/problems/cal_per_pig/ip/

 

Step 2

Now it’s time to interact with grunt  shell.
Enter the below command

 
 
 pig

It will take you to grunt shell
Create one relation named as ip_marks which will have all the data from file marks.csv, as we know that we have kept the file in HDFS location, so mention this HDFS location in and  7 columns names with datatype in below command. .

 
 
  1. ip_marks = LOAD 'hdfs://sandbox.hortonworks.com:8020/user/root/bdp/problems/cal_per_pig/ip/marks.csv' USING PigStorage(',') AS (roll_no:Int,subject1:Int,subject2:Int,subject3:Int,subject4:Int,subject5:Int,subject6:Int)

To see the data of this Relation  use below command.

 
 
  1. dump ip_marks

Refer below screen shot.

Step 3 : Calculation of Percentage

Use below command to calculate Percentage.

 
 
  1. per_mrks= FOREACH ip_marks generate roll_no,(float)(subject1+subject2+subject3+subject4+subject5+subject6)/6.0 AS percentage;

In above command we are adding marks of all subject and converting the sum in float and  then dividing it by 6,we have assumed that total marks of each subject are 100.You can change the formula if you wish.

Step 4 :Output

Use below command to see the data of per_mrks

 
 
  1. dump  per_mrks

To save the data into file use below command.

 
 
  1. STORE per_mrks INTO 'hdfs://sandbox.hortonworks.com:8020/user/root/bdp/problems/cal_per_pig/op/' using PigStorage(',');

It will create a “op” directory, it should not already exist. You can see the list of output files using below command:-

 
 
  1. hadoop fs -ls hdfs://sandbox.hortonworks.com:8020/user/root/bdp/problems/cal_per_pig/op/

Please refer below screen shot.

Don’t forget to subscribe us. Keep learning. Keep sharing.

2
0

Join in hive with example

Requirement You have two table named as A and B. and you want to perform all types of join in ...
Read More

Join in pyspark with example

Requirement You have two table named as A and B. and you want to perform all types of join in ...
Read More

Join in spark using scala with example

Requirement You have two table named as A and B. and you want to perform all types of join in ...
Read More

Java UDF to convert String to date in PIG

About Code Many times it happens like you have received data from many systems and each system operates on a ...
Read More
/ java udf, Pig, pig, pig udf, string to date, udf

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.