Real-Time Performance Analysis of Data-Processing Pipelines with Spring Cloud Data Flow, Micrometer

October 18, 2019

Real-time monitoring is a critical capability for building and operating distributed, data-intensive applications at scale. While Spring Cloud Data Flow (SCDF) helps you orchestrate streaming and batch data pipelines on multiple platforms, the Micrometer integration, on the other hand, provides the mechanism to collect multidimensional performance metrics. SCDF leverages Micrometer to provide a turnkey solution to collect, curate, and visualize key statistics of streaming and batch data pipelines. With practical code examples, we'll demonstrate how to consistently produce structured metrics (both for streaming and batch data pipelines) to Prometheus, and how to visualize them with dashboards such as Grafana. From an observability standpoint, we'll share the best practices to monitor the data-intensive applications running in Cloud Foundry and Kubernetes, which could be the basis for throughput-based elastic autoscaling. Speakers: Christian Tzolov, Principal Software Engineer, Pivotal and Sabby Anandan, Principal Product Manager, Pivotal Filmed at SpringOne Platform 2019 Slides: https://www.slideshare.net/SpringCentral/realtime-performance-analysis-of-dataprocessing-pipelines-with-spring-cloud-data-flow-micrometer

No Previous Videos

Next Video

Event Driven with Spring

In the world of cloud-native distributed architectures, such as microservices, “event driven” is more popul...

Real-Time Performance Analysis of Data-Processing Pipelines with Spring Cloud Data Flow, Micrometer

Next Video

Related content in this Stream

In the world of cloud-native distributed architectures, such as microservices, “event driven” is more popular than ever. But it's a broad category that covers a variety of use cases, ranging from a si

Real-time streaming data ingestion is a common application requirement. In industries like IoT, e-commerce, communications, entertainment, finance, and retail, where so much depends on timely and accu

When it comes time to choose a distributed messaging system, everyone knows the answer: Apache Kafka. But how about when you’re on the hook to choose a world-class, horizontally scalable stream data p

When should I use RabbitMQ and when should I use Kafka? This is a question we're asked all the time on our team. In this talk, we'll walk through the different architecture and design of these two po

Functions-as-a-Service (FaaS) platforms like riff enable users to deploy serverless applications. Serverless applications are event driven and are made up of small “functions” that are chained togethe

Express Scripts, a Cigna corporation company, is reimagining its data and software architecture to bring best-in-class user experience and provide the foundation of next-generation applications. The c

In this talk, we'll explore how Spring Cloud Stream and its support for Apache Kafka can streamline the process of developing event-driven microservices that use Apache Kafka and its high-throughput c

Developers have long employed message queues to decouple subsystems and provide an approximation of asynchronous processing. However, these queuing systems don’t adequately deliver on the promise of e

Events are all around. With Redis Streams, we can capture those and store them in Redis. Redis Streams represent a journal-like data structure that allows sequential processing of messages. Let’s hav

In the advent of volatile cloud-native, distributed architectures ranging from enterprise integration to data streaming to microservices, how can one avoid getting sucked into the vendor/product-speci