An introduction to event-driven architecture and Apache Kafka

Application development based on concepts like microservices and so-called serverless functions is emerging as the new normal for enterprise software delivery. It’s a logical progression; we now live in a cloud-native world, and these modern development models are a natural fit for distributed, cloud-based environments. But simply building and deploying services and functions isn’t enough. 

On its own, a single microservice doesn’t accomplish much. You’ll also need a way to wire up those components—that is, to connect them so they can exchange data, forming a true application. In fact, it’s one of the most important architectural decisions to make.

Various mechanisms for doing this have been developed over the years, ranging from message queues to enterprise service buses (ESBs), Java Message Service (JMS), RESTful APIs, and other, bespoke connection methods. More recent offerings, most notably Apache Kafka, have been designed around the concept of streaming data.

Interest in this latter category is growing, in part because streaming data is seen as a useful tool for implementing event-driven architecture—a software design pattern in which application data is modeled as streams of events, rather than as operations on static records. Let’s examine how this model works, and then explore why Kafka, in particular, is often considered the default platform for managing data in event-driven applications. 

What’s an event, anyway?

Even if the event-driven software model is new to you, the basic concept of events should be familiar. Simply put, events are things that happen, within a software system or, more broadly, during the operation of a business or other human process. An event-driven application is one that is organized around reacting to these events. Examples of events might include:

• A new customer requests an online account login

• A parts warehouse updates its inventory count

• A payment transaction completed

• An unauthorized access attempt was denied

• A shipment arrived at its destination

Each of these events is likely to trigger one or more actions or processes in response. One example might be simply to log the event for monitoring purposes. Others, based on the list of events above, might include:

• The customer is sent an email confirmation that the new account was created

• Supply-chain management software places orders for materials that are running low

• The fulfilment center is directed to assemble an order for delivery

• The account is locked and security personnel are notified

• The sales ticket is closed

As you can see, the concept of events in software systems closely aligns with how most of us think about our day-to-day lives. There’s no hard-and-fast rule about what constitutes an event or how granular events in a given software system should be (in fact, deciding these things often takes a great deal of thought). What’s more, how events are handled will depend on the specific type of architecture you build.

The benefits of thinking about application design in this way, however, are compelling. For one thing, organizing around events makes it easier to develop business logic that accurately models real-world processes. It can also make it easier to insert new processes into the workflow at a later date; for example, if a new government regulation requires an additional step in the process to ensure compliance, the service that handles that action can be triggered by an existing event, rather than rewriting application logic in multiple, tightly coupled components to accommodate the new business process. And organizing around events can help cut down the number of one-to-one connections within a distributed system and improve code reuse, increasing the value of the microservices you build.

Patterns of event-driven architecture

When it comes time to implement an event-driven architecture, naturally there's more than one way to do it. Some patterns might be easier to implement, while others can be more adaptable to complex needs. Which one is best for a given use case will depend on a number of factors, including how many microservices are in play, how tightly coupled they should be, and how much “data chatter” the system can accommodate. Examples of popular event-driven patterns include:

Event notification 

In this model, a microservice sends events merely to notify other systems of a change in its domain. For example, a user account service might send a notification event when a new login is created. What other systems choose to do with that information is largely up to them; the service that issued the notification just carries on with its business. Notification events usually don’t carry much data with them, resulting in a loosely coupled system with minimal network traffic spent on messaging.

Event-carried state transfer

A step up from simple notification, in this model the recipient of an event also receives the data it needs to perform further work. The aforementioned user account service, for example, might issue an event that includes a data packet containing the new user’s login ID, full name, hashed password, and other pertinent details. This model can be appealing to developers familiar with RESTful interfaces, but depending on the complexity of the system, it can lead to a lot of data traffic on the network and data duplication in storage.

Event-sourcing

The goal of this model is to represent every change of state in a system as an event, each recorded in chronological order. In so doing, the event stream itself becomes the principal source of truth for the system. For example, it should be possible to “replay” a sequence of events to recreate the state of a SQL database at a given point in time. This model presents a lot of intriguing possibilities, but it can be challenging to get right, particularly when events require participation from external systems.

This list is not exhaustive, and still more event-driven patterns may emerge over time. For a deeper dive, check out this article by Martin Fowler, one of the more prominent experts in modern architecture.

The case for Kafka

As mentioned earlier, various methods of wiring up services have been embraced over the years, with Apache Kafka being one of the newer and more popular ones. That being said, traditional message queues such as RabbitMQ still have plenty of use cases, including in more loosely coupled event-driven models, such as event notification, which don’t necessarily require a full featured data platform.

But one reason Kafka is so popular right now is because it has a lot of good things going for it, including its pedigree:

•  It was created at LinkedIn, where one of the design requirements was that it could sustain operations at massive scale.

• It’s an open source project maintained under the Apache Foundation.

• An enterprise-grade platform based on Kafka is offered by Confluent, a company founded by some of Kafka’s original developers.

In addition, Kafka has characteristics that set it apart from traditional enterprise message queues and service buses:

• Kafka is a distributed platform, meaning data can be replicated across a cluster of servers for fault tolerance, including geo-location support.

• As compared to the traditional queueing and publish-subscribe models, Kafka offers a hybrid model that combines the advantages of both: message-processing remains scalable even when messages are received by multiple consumers.

• It offers strong guarantees that messages will be received in the chronological order in which they were published.

• Kafka’s storage system can efficiently scale to terabytes of data, with a configurable retention period—meaning even when outages that last whole days occur, once service is restored, the event stream proceeds in order.

• Configurable retention also means Kafka is equally suitable for real-time streaming applications and periodic batch data pipelines.

• In addition, Kafka offers a stream-processing API that allows for complex transformations of data as it’s passed between service endpoints.

These and other features make Kafka an attractive fit for more advanced event-driven patterns, such as event-sourcing, where message queues are not a good fit. The Apache Kafka website has plenty of further information regarding use cases, APIs, security, and other implementation details, but there’s also a growing amount of information online regarding concepts and best practices.

Learn more about Kafka and event-driven architecture

One place to start is this 15-minute podcast crash course featuring Neha Narkhede, co-founder and CTO of Confluent and a co-creator of Kafka, in which she discusses the growing importance of streaming data and event-driven architecture. Other great resources include:

• Putting Apache Kafka To Use: A Practical Guide to Building an Event Streaming Platform, by Confluent’s Jay Kreps (Part 1 and Part 2)

• How to Deliver an Event-Driven Architecture, a dive into architectural details from Pivotal’s Richard Seroter

• Apache Kafka Streaming Platform Explained, a video of Neha Narkhede’s talk from the SpringOne Platform 2018 conference

• Spring Boot and Kafka: The New Enterprise Platform, a video of Pivotal’s James Watters’ keynote at Kafka Summit 2019

Getting started

Event-driven architecture is gaining in popularity, and with good reason. From a technical perspective, it provides an effective method of wiring up microservices that can help future-proof systems; for example, interest in serverless functions—such as AWS Lambda, Azure Functions, or Knative—is growing, and these are inherently event-based. Moreover, when coupled with modern streaming data tools like Apache Kafka, event-driven architectures become more versatile, resilient, and reliable than with earlier messaging methods.

But perhaps the most important “feature” of the event-driven pattern is that it models how businesses operate in the real world. Organizations that want to explore building applications in this way should first map out the events that need to be modeled in their workflows. This planning phase usually calls for direct collaboration between business-line managers and app development groups. 

And that may be event-driven architecture’s real “secret sauce”: it’s not only powerful, but it also helps foster true alignment between business objectives and IT.

About the Author

Neil McAllister is a veteran technology journalist covering enterprise technology. He is currently a Senior Product Marketing Manager at Pivotal.

More Content by Neil McAllister