Conveyor

This proof of concept shows the use of Spring Boot and Kafka together to execute long-running tasks asynchronously with a REST API.

Built With

• Java
• Spring Boot
• Apache Kafka
• H2 Database
• Docker
• Apache Maven

Prerequisites

Kafka runs with Docker Compose, which is integrated into Spring Boot. A working Docker setup must therefore be available to start the project. Java 21 and Maven are also required.

Build & Run

1. Clone the repo

   git clone https://github.com/murygin/malware-scanner.git

2. Compile

   ./mvnw clean compile

3. Run

   ./mvnw spring-boot:run

Usage

The API provides an endpoint for starting jobs and an endpoint for loading the job results. If the service is started with ./mvnw spring-boot:run the base url is http://localhost:8080.

Bruno

Bruno is a Git-friendly, offline-first API client built for developers who want fast local workflows, plain-text collections, and better collaboration through Git.

You can use the Bruno collection in folder src/test/bruno/Conveyor to test the API.

POST /jobs

Starts ta new job. The job is started asynchronously. The result is not returned directly in the response. The response contains a confirmation of the start with the ID of the job. The response header Location contains the URL for loading the result.

Request:

{
  "data": "conveyor.wait.ms=3500",
  "type": "slacker"
}

Response:

• Status: 202 Accepted
• Header: Location: /jobs/b3a5896f-387b-4363-a631-cfbf467db1ce

{
   "state": "CREATED",
   "results": [],
   "id": "b3a5896f-387b-4363-a631-cfbf467db1ce"
}

GET /jobs/<UUID>

Loads the result of a job. If the job has not yet been started, the status CREATED is returned. If the job is currently running, the status RUNNING is returned. When the job is completed, the status FINISHED and a result is returned.

Response:

• Status: 200 OK

{
   "state": "FINISHED",
   "results": [
      {
         "state": "OK",
         "name": "slacker",
         "details": "I finished this job in just 3503 ms, no problem"
      }
   ],
   "id": "b3a5896f-387b-4363-a631-cfbf467db1ce"
}

How it works

The REST endpoint POST /jobs can be used to trigger a new job. When the endpoint is called, the method create is called in the controller. The Spring Boot REST Controller o.d.c.rest.JobsController contains the methods that are executed when the endpoints are called. The controller is only a facade and passes the calls on to the o.d.c.service.JobService.

If a new job is requested, the controller calls the method createJob in the JobService. The job is not started directly. The job is only triggered by the Kafka event. This has the advantage that the caller of the REST endpoint is not blocked and has to wait, but receives a response immediately. This method createJob in JobService creates a o.d.c.model.Job with the status CREATED and saves it in the database. A o.d.c.model.JobEvent is then sent to event streaming platform Kafka.

The jobEvents are consumed by the o.d.c.kafka.KafkaTopicListener. After receiving the event, the KafkaTopicListener set the status of the Job to RUNNING and starts the job by calling the executeJob method in the o.d.c.service.JobExecutionService.

After the job is finished in the JobExecutionService is completed, an o.d.c.model.JobResultEvent is sent to Kafka. The JobResultEvent is consumed by the KafkaTopicListener. The KafkaTopicListener takes the result of the job from the event and saves it in the Job The status of the job is set to FINISHED. Now the result of the job can be loaded from the client via the REST endpoint GET /jobs/<UUID>.

Improvements & Enhancements

• The service should only be able to be used if a client is authenticated.
• The API should be documented with Spring SpringDoc, OpenAPI and Swagger.
• Test coverage should be improved. Integration tests are to be implemented for the controller calls and the processing of Kafka events.
• A load test needs to be written to test how the system performs when many requests have to be processed simultaneously.
• Exception handling should be improved if an invalid request body is sent to the POST /jobs endpoint.

Articles

With the articles in this section you can learn more about frameworks and systems that are used in this application.

Kafka

• Apache Kafka Quickstart
• Run Kafka Streams Demo Application
• Is a Key Required as Part of Sending Messages to Kafka?
• What should I use as the key for my Kafka message?
• Kafka Integration Testing with Spring Boot
• Testing Kafka and Spring Boot

API Design

• REST API Design for Long-Running Tasks

Spring Boot

• Docker Compose Support in Spring Boot 3.1
• Getting started with Spring Boot 3, Kafka over docker with docker-compose.yaml
• Building REST services with Spring
• Spring Boot With H2 Database
• Building REST services with Spring
• Getting started with unit testing in spring boot

Contact

Daniel Murygin - linkedin.com/in/murygin - daniel.murygin@gmail.com

Project Link: https://github.com/murygin/conveyor