VMware Spring Cloud Data Flow Adds Real-time Message Tracing, Redesigned Dashboard

June 15, 2021 Sabby Anandan

The minor releases of VMware Spring Cloud Data Flow for Kubernetes (v1.3) and VMware Spring Cloud Data Flow for VMware Tanzu (v1.11) are now generally available. These commercial offerings build on the v2.8 release of open source Spring Cloud Data Flow, and add enterprise-ready features to boost developer productivity.

The core value proposition of Spring Cloud Data Flow is about providing a lightweight and interactive developer experience to design, develop, and deploy data pipelines made of event-streaming and batch microservices. By relying on these developer experiences, our customers have clocked significant mileage with large-scale event-streaming and batch data pipeline footprints running on both Kubernetes and VMware Tanzu Application Service.

The theme of these two new minor releases is to stabilize orchestration semantics in Spring Cloud Data Flow by pushing scale-out limits across supported platforms and compatible versions. There are also new features that improve the user experiences of application developers, data engineers, and application operators.

So let’s dig into the details.

Stability

Our customers orchestrate large-scale event-streaming and batch data pipelines through Spring Cloud Data Flow on various IaaS platforms running either Tanzu Application Service or Kubernetes, if not both. And there is increasing evidence that most deployments are orchestrated using Spring Cloud Data Flow’s Java DSL to programmatically build and initiate the deployment of data pipelines using CI/CD or GitOps automation.

To provide a stable foundation for our customers, irrespective of the scale of their data processing operations, our automation incrementally runs integration and end-to-end acceptance tests against several versions of Tanzu Application Service and Kubernetes. As a result, based on VMware’s internal testing, we have increased test harnessing by more than 200 percent, and deployments to different platform versions have increased by a factor of 5x!

Real-time traces and observability

The existing integrations of Spring Cloud Data Flow (event-streaming and batch applications with observability tooling), such as with Tanzu Observability, Prometheus with Grafana, and others, now all get a makeover. In addition, we are extending observability mechanics to include message tracing support for event-streaming data pipelines.

With message tracing support, users can inspect and remediate hotspots in real-time streaming data pipelines. Having this instrumentation is critical given the large-scale data processing required to drive downstream business outcomes. Additionally, tracing support can be used to architect message latency-driven triggers, which helps autoscale event-stream processors based on continuously changing throughput demands.

New Task and Batch dashboard

While event-streaming has seen significant adoption in the enterprise, batch data-processing continues to run strong if you look at specific use cases.

Businesses migrate from mainframes and database-stored procedure-driven ETL flows, to microservices-based batch data pipelines, that are triggered either by real-time event streams or through scheduled recurring runs. The choreography and orchestration of these batch-style data processing pipelines is where Spring Cloud Data Flow can help add significant value to our customers’ modernization journey.

The most important feature of this June 2021 release is the complete overhaul and redesign of both task/job workflows as well as the associated user interface and experience of using the Spring Cloud Data Flow dashboard. The tooling improvements can help significantly boost the productivity of application developers, data engineers, and application operations teams as they design, launch, interact with, and troubleshoot or monitor task/job data pipelines deployed by Spring Cloud Data Flow. 

Among our most requested features are the ability to filter based on task/job definition and to clean up/recover from a corrupted state of batch data processing and execution. We’re delighted to include these features in this release.

The screen capture below walks through both the new UX improvements and the workflow automation, showcasing guardrails that help to reduce guesswork and minimize mistakes.

Other notable announcements

Outside of Kubernetes, Tanzu Application Service customers can now also enjoy the enterprise features that come with the v1.11 release of Spring Cloud Data Flow for VMware Tanzu, namely:

  • Multi-IO visualization for Kafka Streams

  • Application version alerting and CI/CD automation

  • Task launches and incremental progress alerting

The recent Spring Cloud Stream applications v2020.0.2 release includes feature enhancements and improvements to out-of-the-box applications for common use cases. Check out our more than 15 case studies and reference architectures to learn more.

Try out the latest Spring Cloud Data Flow on Kubernetes or Tanzu Application Service

Review the documentation and download and install VMware Spring Cloud Data for Kubernetes and Spring Cloud Data Flow for VMware Tanzu. Please reach out in VMware Support or head to the open source StackOverflow forum to ask questions and receive feedback.

About the Author

Sabby Anandan

Sabby Anandan is a Product Manager on the Spring Team at VMware. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining VMware, Sabby worked in engineering and management consulting positions. He holds a Bachelor’s degree in Electrical and Electronics from the University of Madras and a Master’s in Information Technology and Management from Carnegie Mellon University.

More Content by Sabby Anandan
Previous
Announcing the VMware Tanzu Application Service 2.11.1 Release
Announcing the VMware Tanzu Application Service 2.11.1 Release

This update further simplifies the process of deploying and running apps across clouds at scale.

Next
Secure and Up-to-Date Containers with VMware Tanzu Build Service 1.2
Secure and Up-to-Date Containers with VMware Tanzu Build Service 1.2

Now generally available, VMware Tanzu Build Service 1.2 is packed with features that address core build cha...

SpringOne 2021

Register Now