Benchmarking Stream Performance With Spring XD 1.2 and Apache Kafka

June 17, 2015 Sabby Anandan

sfeatured-XD-KafkaOne of the goals for the Spring XD 1.2 release was to obtain the baseline performance metrics on a typical cluster of machines and then optimize stream performance where necessary. Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. Our testing drove several optimizations to increase streaming performance. The benchmarks found that a single, single threaded Spring XD stream can handle over 2 million (100 byte) events a second, using Apache Kafka as a transport. If the business case allows the stream to run on one container, XD can handle ~13 million (100 byte) events a second using direct (in-memory) binding.
To gain a perspective on these results, the graph below shows XD’s performance in megabytes per second, compared to Kafka’s own performance benchmarks with various message sizes.






















This graph shows that a customer will get all of the benefits of Spring XD with near-native performance for the underlying transport. Spring XD 1.2 uses the new Spring Integration Kafka adapter, which offers a richer feature set than the standard Kafka client library. The other takeaway from our tests is that these results were obtained using Spring XD out of the box, requiring minimal configuration to set up the Kafka batch size, mode, and the connectivity to the Kafka transport.

For anyone interested in reproducing these numbers, please refer to the engineering blog, which describes in detail the tests performed and infrastructure used.

Learn more:

About the Author

Sabby Anandan

Sabby Anandan is a Product Manager on the Spring Team at VMware. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining VMware, Sabby worked in engineering and management consulting positions. He holds a Bachelor’s degree in Electrical and Electronics from the University of Madras and a Master’s in Information Technology and Management from Carnegie Mellon University.

More Content by Sabby Anandan
The Data Science Behind Predictive Maintenance For Connected Cars
The Data Science Behind Predictive Maintenance For Connected Cars

Pivotal has extensive experience with platforms for the connected car. In this post, three of our data scie...

Living On The Edge: Why Route Services Matter
Living On The Edge: Why Route Services Matter

Route Services in Pivotal Cloud Foundry (PCF) are a new type of marketplace service that developers insert ...

SpringOne 2021

Register Now