In this post, we will explore how to perform sliding window operations in Spark Streaming. Sliding window operations allow us to analyze data over a sliding time window with overlapping intervals, enabling continuous calculations and aggregations on streaming data. Problem Statement We want to develop a Spark Streaming application thatRead More →
In this post, we will explore how to write data to Apache Kafka in a Spark Streaming application. Apache Kafka is a distributed streaming platform that enables high-throughput, fault-tolerant, and scalable data streaming. Problem Statement We want to develop a Spark Streaming application that can process data in real-time andRead More →
In this post, we will explore how to read data from Apache Kafka in a Spark Streaming application. Apache Kafka is a distributed streaming platform that provides a reliable and scalable way to publish and subscribe to streams of records. Problem Statement We want to develop a Spark Streaming applicationRead More →
In this post, we will explore how to create a Spark Streaming application. Spark Streaming is a powerful component of Apache Spark that enables real-time processing and analysis of streaming data. Problem Statement We want to develop a real-time application that can process and analyze streaming data from a dataRead More →
1. If there is a csv file present in hdfs location which has a header . while reading it in to spark, which property needs to be set Answer : while reading in dataframe we need to set an option of header to true like below :- var df1=spark.read.option(“header”,true).csv(“path”) Read More →
Objective :- In this Tutorial we are going to learn how to read data from kafka and how to send data to kafka using console. Create a kafka topic we are creating one kafka topic named as sampleTopic1 For Now we are keeping replication-factor to 1 and partitions to 1.Read More →
