Today, we're going to take a look at data streaming from a reactive Kafka-based WebFlux REST server to a Webflux client in a non-blocking manner.
Below designed architecture can be used to:
- Push data to external or internal apps in near real-time.
- Push data onto the files and securely copy them to cloud services.
- Push the same data over to multiple clients from a Kafka topic.
Let's get started!
Before we execute a sample application to demonstrate Server-Sent Events (SSE) using Spring WebFlux and Reactive Kafka, let's understand the fundamental concepts:
**What are Server-Sent Events?
**Server-Sent Events (SSE) is a Server Push technology that allows a client to receive automatic server updates through the HTTP connection.
The SSE can be used to:
- Replace Long polling (which creates a new connection for every pull) by maintaining a single connection and keeping a continuous event stream going through it.
- Enable apps that use one-way data communication (eg: e-commerce websites, live stock price updates).
What is Spring WebFlux?
Spring WebFlux framework is a fully asynchronous and non-blocking reactive web stack that enables the handle of a massive number of concurrent connections. WebFlux supports Reactive Streams back pressure and runs on such servers as Netty. It enables us to vertically scale the services to handle the greater load on the same hardware.
What is Reactive Kafka?
Reactive Kafka is a reactive API for Kafka based on project Reactor and the Kafka producer/Consumer API. It enables data to be published and consumed from Kafka using functional API with non-blocking back pressure and low overheads which allows reactive Kafka to be integrated with other reactor systems and provide an end-end reactive pipeline.
Note: to gain a thorough grasp of Webflux and Reactive Kafka, make sure to understand the terminology.
As per the architecture shown in image-1, we will build a WebFlux Server using the Spring WebFlux framework and reactive Kafka, exposing a REST API for the clients to make secure HTTP requests.
Once a secure connection is established between the client and the web flux server, it consumes messages from Kafka topics and pushes the data asynchronously without closing connection with the client unless required.
Rather than build a reactive Kafka producer, we will leverage the existing producer example on the reactor repository. Additionally, instead of building a web flux client, we will test the server's SSE response by using a curl command on the terminal.
- Java v1.8+
- Apache Kafka + basic understandings
- Intellij or Eclipse or Sprint Tool Suite
- Kafka Conduktor tool
Let's create a Spring boot application using the following dependencies shown below.
Let's create a Kafka Receiver Configuration, which is a consumer as shown below. It is configured with generic GROUP_ID_CONFIG since we are working on handling a single client for now by enabling auto commits & always reading the earliest messages but we can also update it to the _latest.
If we enable multiple clients, each client can receive messages from the same topic based on the last committed offset.
In order to keep things simple, we will be dealing with String Deserializers which can be extended to generic JSON/AVRO schemas.
After the configurations are ready, we will create a REST controller that consumes messages from Kafka topics and sends back responses as the flux of data. Use MediaType.TEXT_EVENT_STREAM_VALUE as the content-type. This tells the client that a connection will be established and the stream is open for sending events from the server to the client.
Before we test this application, here is a sample producer we're leveraging from the reactor repository with StringSerializer:
Now, start the Kafka server in the localhost:9092 and create a topic as shown in the above configurations.
The following commands would be helpful.
$ bin/zookeeper-server-start.sh config/zookeeper.properties $ bin/kafka-server-start.sh config/server.properties $ bin/kafka-topics.sh — create — bootstrap-server localhost:9092 — replication-factor 1 — partitions 1 — topic <topic_name> $ bin/kafka-topics.sh — list — bootstrap-server localhost:9092
The image below shows the topic created with partition 1 and without any message (Count=0) being produced from Kafka Sender.