udf

Get last element in list of dataframe in Spark

Tagged: reverse, Spark, spark dataframe, spark sql, split, udf, udf in spark

Requirement In this post, we will learn how to get last element in list of dataframe in spark. Solution Create a dataframe with dummy data: val df = spark.createDataFrame(Seq( (“1100”, “Person1”, “Street1#Location1#City1”, null), (“1200”, “Person2”, “Street2#Location2#City2”, “Contact2”), (“1300”, “Person3”, “Street3#Location3#City3”, null), (“1400”, “Person4”, null, “Contact4”), (“1500”, “Person5”, “Street5#Location5#City5”, null) )).toDF(“id”,Read More →

UDF in Spark

Tagged: scala, Spark, udf

Requirement The UDF is a user-defined function. As its name indicate, a user can create a custom function and used it wherever required. We do create UDF when the existing build-in functions not available or not able to fulfill the requirement. Sample Data empno ename designation manager hire_date sal deptnoRead More →

Java UDF to convert String to date in PIG

Tagged: java udf, pig, pig udf, string to date, udf

About Code Many times it happens like you have received data from many systems and each system operates on a different kind of date format. But in the output you need to have a specific date format. Let’s say you are receiving date string like :- 12-01-2018 12:22:33 2018/12/01 12:22:33Read More →