Apache Kafka Vs Apache Spark: Know the Differences
A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. – Dean Wampler (Renowned author of many big data technology-related books)
Dean Wampler makes an important point in one of his webinars. The demand for stream processing is increasing every day in today’s era. The main reason behind it is, processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real time is very essential so that organization can react to changing business conditions in real time.
And hence, there is a need to understand the concept “stream processing “and technology behind it.
Think of streaming as an unbounded, continuous real-time flow of records and processing these records in similar timeframe is stream processing.
AWS (Amazon Web Services) defines “Streaming Data” is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling.
In stream processing method, continuous computation happens as the data flows through the system.
Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. It is also best to utilize if the event needs to be detected right away and responded to quickly.
There is a subtle difference between stream processing, real-time processing (Rear real-time) and complex event processing (CEP). Let’s quickly look at the examples to understand the difference.
We have multiple tools available to accomplish above-mentioned Stream, Realtime or Complex event Processing. Spark Streaming, Kafka Stream, Flink, Storm, Akka, Structured streaming are to name a few.
We will try to understand Spark streaming and Kafka stream in depth further in this article. As historically, these are occupying significant market share.
Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline.
Typically, Kafka Stream supports per-second stream processing with millisecond latency.
Kafka Streams is a client library for processing and analyzing data stored in Kafka. Kafka streams can process data in 2 ways.
It also does not do mini batching, which is “real streaming”.
Note:
Kafka Streams is built upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple (yet efficient) management of application state. It is based on many concepts already contained in Kafka, such as scaling by partitioning.
Also, for this reason, it comes as a lightweight library that can be integrated into an application.
The application can then be operated as desired, as mentioned below:
Spark Streaming receives live input data streams, it collects data for some time, builds RDD, divides the data into micro-batches, which are then processed by the Spark engine to generate the final stream of results in micro-batches. Following data flow diagram explains the working of Spark streaming.
![Apache Spark Streaming]()
Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data.
DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. Think about RDD as the underlying concept for distributing data over a cluster of computers.
It makes it very easy for developers to use a single framework to satisfy all the processing needs. They can use MLib (Spark’s machine learning library) to train models offline and directly use them online for scoring live data in Spark Streaming. In fact, some models perform continuous, online learning, and scoring.
Not all real-life use-cases need data to be processed at real real-time, few seconds delay is tolerated over having a unified framework like Spark Streaming and volumes of data processing. It provides a range of capabilities by integrating with other spark tools to do a variety of data processing.
Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Following table briefly explain you, key differences between the two.
Following are a couple of many industry Use cases where Kafka stream is being used:
Broadly, Kafka is suitable for microservices integration use cases and have wider flexibility.
Following are a couple of the many industries use-cases where spark streaming is being used:
Broadly, spark streaming is suitable for requirements with batch processing for massive datasets, for bulk processing and have use-cases more than just data streaming.
Dean Wampler explains factors to evaluation for tool basis Use-cases beautifully, as mentioned below:
Kafka Streams is still best used in a ‘Kafka -> Kafka’ context, while Spark Streaming could be used for a ‘Kafka -> Database’ or ‘Kafka -> Data science model’ type of context.
Although, when these 2 technologies are connected, they bring complete data collection and processing capabilities together and are widely used in commercialized use cases and occupy significant market share.
Stream Processing: Stream processing is useful for tasks like fraud detection and cybersecurity. If transaction data is stream-processed, fraudulent transactions can be identified and stopped before they are even complete.
Real-time Processing: If event time is very relevant and latencies in the second’s range are completely unacceptable then it’s called Real-time (Rear real-time) processing. For ex. flight control system for space programs
Complex Event Processing (CEP): CEP utilizes event-by-event processing and aggregation (for example, on potentially out-of-order events from a variety of sources, often with large numbers of rules or business logic).
Kafka -> Kafka: When Kafka Streams performs aggregations, filtering etc. and writes back the data to Kafka, it achieves amazing scalability, high availability, high throughput etc. if configured correctly.
Kafka -> External Systems (‘Kafka -> Database’ or ‘Kafka -> Data science model’): Typically, any streaming library (Spark, Flink, NiFi etc) uses Kafka for a message broker. It would read the messages from Kafka and then break it into mini time windows to process it further.
Sources here could be event logs, webpage events etc. etc.
DB/Models would be accessed via any other streaming application, which in turn is using Kafka streams here.
Standalone, in an application server
As a Docker container, or
Directly, via a resource manager such as Mesos.
Elastic, highly scalable, fault-tolerant
Deploy to containers, VMs, bare metal, cloud
Equally viable for small, medium, & large use cases
Fully integrated with Kafka security
Write standard Java and Scala applications
Exactly-once processing semantics
No separate processing cluster required
Develop on Mac, Linux, Windows
The New York Times: The New York Times uses Apache Kafka and Kafka Streams to store and distribute, in real-time, published content to the various applications and systems that make it available to the readers.
Pinterest: Pinterest uses Apache Kafka and the Kafka Streams at large scale to power the real-time, predictive budgeting system of their advertising infrastructure. With Kafka Streams, spend predictions are more accurate than ever.
Zalando: As the leading online fashion retailer in Europe, Zalando uses Kafka as an ESB (Enterprise Service Bus), which helps us in transitioning from a monolithic to a micro services architecture. Using Kafka for processing event streams enables our technical team to do near-real time business intelligence.
Trivago: Trivago is a global hotel search platform. We are focused on reshaping the way travellers search for and compare hotels while enabling hotel advertisers to grow their businesses by providing access to a broad audience of travellers via our websites and apps. As of 2017, we offer access to approximately 1.8 million hotels and other accommodations in over 190 countries. We use Kafka, Kafka Connect, and Kafka Streams to enable our developers to access data freely in the company. Kafka Streams powers parts of our analytics pipeline and delivers endless options to explore and operate on the data sources we have at hand.
Booking.com: We are using Spark Streaming for building online Machine Learning (ML) features that are used in Booking.com for real-time prediction of behaviour and preferences of our users, demand for hotels and improve processes in customer support.
Yelp: Yelp’s ad platform handles millions of ad requests every day. To generate ad metrics and analytics in real-time, they built the ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage a large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a timelier fashion.
Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix, and Pinterest.
Research & References of Apache Kafka Vs Apache Spark: Know the Differences|A&C Accounting And Tax Services
Source
From Admin and Read More here.
A note for you if you pursue CPA licence, KEEP PRACTICE with the MANY WONDER HELPS I showed you. Make sure to check your works after solving simulations. If a Cashflow statement or your consolidation statement is balanced, you know you pass right after sitting for the exams. I hope my information are great and helpful. Implement them. They worked for me. Hey.... turn gray hair to black also guys. Do not forget HEALTH?
Competency Progression is actually the number 1 critical and significant consideration of achieving true success in just about all professions as everyone observed in all of our society in addition to in Across the world. As a result happy to focus on together with everyone in the subsequent related to just what exactly thriving Expertise Enhancement is;.
precisely how or what procedures we get the job done to get objectives and in the end one will certainly get the job done with what anyone likes to accomplish all working day for the purpose and meaningful of a comprehensive living. Is it so amazing if you are competent to grow properly and locate achievement in just what exactly you thought, steered for, disciplined and did wonders really hard just about every afternoon and surely you turned into a CPA, Attorney, an person of a great manufacturer or even a medical professionsal who may seriously contribute wonderful help and valuations to many people, who many, any world and local community unquestionably popular and respected. I can's believe that I can allow others to be leading professional level just who will bring about considerable answers and pain relief values to society and communities presently. How joyful are you if you turned out to be one similar to so with your individual name on the title? I get landed at SUCCESS and conquer all the difficult areas which is passing the CPA examinations to be CPA. Furthermore, we will also protect what are the downfalls, or many other troubles that is perhaps on your current option and the best way I have personally experienced them and is going to indicate you the best way to get over them.
0 Comments