distinct Archives - BIG DATA PROGRAMMERS

Trim Column in PySpark DataFrame

Tagged: dataframe, distinct, pyspark, Spark, trim

Requirement As we received data/files from multiple sources, the chances are high to have issues in the data. Let’s say, we have received a CSV file, and most of the columns are of String data type in the file. We found some data missing in the target table after processingRead More →

How to get distinct words of a file using Map Reduce

Tagged: distinct, hadoop, hdfs, MapReduce, maven, words

Requirement Suppose you have a file in which many words are repeated. Now the requirement is to get distinct words from the file using Map Reduce. If you compare with the SQL, then we have to write a map reduce program which is similar to the DISTINCT function of SQL.Read More →