Tech Insights / Event Streaming

Event streaming:
Build applications that keep up with business

Event streaming reflects how businesses actually work—thousands of small changes happening, all day, every day. Shouldn’t your applications work the same way?

What is an event?

In the context of applications, events simply are things that happen within a software system. Oftentimes, they’re tied to actions that affect how a business operates or how a user navigates through a process (applying for a credit card, for example). Examples of events might include a customer logging into a service, a distribution center updating its inventory, or a payment transaction completing.

Each event should trigger one or more actions or processes in response. A common and very simple action is logging the event for monitoring purposes. However, other actions can be more complex or proactive in nature. Based on the examples above, for example, a service might respond to a specific event by serving a verification request, updating an e-commerce database, or closing an open sales ticket.

Most likely, an event will actually trigger a series of events, as is common with almost every interaction. Take e-commerce, for example: When a user clicks “purchase,” the credit card is processed, the shopping cart is emptied, the inventory database is updated, a confirmation email is sent, and the multi-step shipping process kicks off.

What is event streaming: Event-driven architecture

One of the inherent challenges with microservices is the coupling that can occur between the services. In a conventional ask, don’t tell architecture, data is gathered on demand. Service A asks Services B, C, and D, “What’s your current state?” This assumes B, C, and D are always available to respond. However, if they moved or they’re offline, they can’t respond to Service A’s query.

To compensate, microservice architectures tend to include workarounds (such as retries) to deal with any ill effects caused by changes in deployment topology or network outages. This adds an extra layer of complexity and cost.

Event streaming attempts to solve this problem by inverting the communication process among services. An event-driven architecture utilizes a tell, don’t ask approach, in which Services B, C, and D publish continuous streams of data as events. Service A subscribes to these event streams—processing the facts, collating the results, and caching them locally. However, Service A only needs to act, or perform its function, when it is delivered a specific type of event.

Using event streams in this way can deliver a number of benefits, including:

Systems that more closely mimic real-world processes, because software teams have carefully considered the optimal flow.
Increased utilization of scale-to-zero functions (or serverless computing), because more services can remain idle until they’re needed.
Improved flexibility, because new requirements or features can be captured in loosely coupled microservices instead of rewriting the business logic of a large monolith.

What is event streaming: Streaming data

Streaming data means a constant flow of data, each containing information about an event or change of state. Streaming data is processed in real-time as it’s delivered to a system, although the type of data or the nature of events typically will affect any resulting action. Notably, any given event flowing in from a data stream might not be immediately actionable.

Streaming data is unbounded, meaning it has no real beginning and no real end—each event is processed as it occurs and managed accordingly. Events can trigger specific application behaviors and feed real-time business dashboards, as well as make their way into other systems for later batch processing. This is in contrast to legacy approaches, where, by default, data is sent to a backend data warehouse or Hadoop cluster to be analyzed every day, week, month, or quarter.

The stock market is a common example. Every time a stock price changes, a new message is created containing, among other things, the time of day, the stock’s code, and its new trade price. Because an exchange might house thousands of individual stocks and handle millions of trades per day, the result (at least during trading hours) is a constant stream of data. How an organization analyzes or acts on that data—and how frequently it does either—depends on any number of factors specific to the organization’s business.

How to prepare for an event-driven future

If your organization wants to adapt to future business demands more quickly, it should consider event streaming. Streaming architectures can power everything from simple event notifications—for example, sending an alert when a stock price slumps—to real-time machine learning models to detect suspicious trade activity. Even in batch operations, streaming data can improve analytics and business intelligence by connecting specific events with the times they occurred, or by highlighting historical trends.

Modern stream processing systems such as Apache Kafka can also act as a stateful source of truth about the business. Because they can store and process events—while also holding onto historical data—systems can analyze and aggregate data in real time without reaching out to external data sources.

However simple or advanced the implementation, event streaming can help organizations future-proof their applications. A common benefit realized by VMware customers at all levels is their ability to add new features or eliminate problematic ones simply by adding or removing subscribers to a data stream. There’s no monolithic business logic to reconfigure; just microservices to plug or unplug.

In the years to come, the ability to think in events and build applications around them will become ever more important. Apart from the business improvements that go along with being faster, making better use of data, and designing more intuitive user experiences, there are also technologies such as serverless and functions that have the notion of events baked into their cores. The best enterprises will take full advantage of these innovations to build better, more successful applications.

Event streaming at VMware

As a contributor to the Spring Framework, RabbitMQ, and many other open source projects related to modern application development, VMware supports developers building streaming data applications in many ways. Here are just a few highlights:

VMware is also working closely in partnership with Confluent, provider of the leading commercial event-streaming platform based on Apache Kafka, to bring enterprise-grade Kafka to our ecosystem. The Confluent Operator automates and simplifies the provisioning and management of the Confluent Platform on Tanzu Kubernetes Grid. Enterprises can:

  • Deploy Kafka as a cloud native application on Kubernetes to simplify and automate provisioning, and minimize the operating burden of managing Kafka clusters at scale.
  • Run applications and their Kafka service on any (public or private) cloud without modification or the dreaded “vendor lock-in.”

Solace PubSub+ for VMware Tanzu, another partner integration, registers a service broker on VMware Tanzu Application Service and exposes the preconfigured service plans in the VMware Marketplace for developers to choose. Solace, together with VMware, also developed the first third-party, commercially developed binder to ensure optimal integration with Spring Cloud Stream. It’s been released as an open source project and is available on GitHub and Maven.

The tile, along with Spring Cloud Stream Binder for PubSub+, gives developers the tools they need to deploy event-driven microservices and steaming applications to VMware Tanzu Application Service.