Spring Cloud DataFlow 2.5 Is GA: Level Up!

May 13, 2020 Sabby Anandan

We are pleased to announce the general availability of Spring Cloud Data Flow 2.5, with enhancements in performance, enterprise identity management, starter apps as Java functions, and more. Let’s dive into the details.

Speed thrills

Our customers develop and deliver state-of-the-art streaming solutions that regularly push the limits. And we’re grateful that they challenge us to ensure that Spring Cloud Data Flow (SCDF) always has the agility and bandwidth to meet their needs—today and in the future. In their large-scale IoT scenarios, it isn’t unusual for our customers to challenge the overall resilience and performance of their applications by deploying hundreds of streaming data pipelines.

In SCDF 2.5, we’ve upgraded the performance of the orchestration and monitoring APIs for microservices running in Cloud Foundry (Tanzu Application Service) or Kubernetes. In our tests, we’re seeing that 'runtime/streams', 'runtime/apps', 'streams/definitions', 'streams/deployments', and the associated backing APIs used in the SCDF dashboard/shell are now over 90 percent faster! 

The next time you use the dashboard, shell, and other client tools, get ready to experience the performance boost in action.

Security, now with Azure AD

Security is on everyone’s minds. And as our customers continue to exercise and extend the OAuth2/OIDC-spec-compliant security abstraction in SCDF, we’ve made identity management easier by adding Azure Active Directory support for prescriptive single sign-on. 

All SCDF clients (RESTful APIs, dashboard, Java DSL, shell, TaskLauncher Dataflow, and the Composed Task Runner) can now have native integration with Azure AD for authentication and authorization flows. 

For Application Starters, we’ve gone functional

The SCDF starter applications (about 70 of them!) are evolving, as we port and modernize the full-blown applications to add standalone Java-function versions. These have reduced resource footprints, are easier to understand, and can be composed with other Java functions.

The new Java functions can be built and registered as uber-jars in SCDF, usable in Knative as Java-based event sources, or generally useful as composable data-integration functions to natively run in a platform wherever there's Java.

While the applications are gradually being promoted to top-level Java functions and their application equivalents, we welcome the community to lean in and be part of the action. Please try them out, give us feedback, and contribute. You can start the dialog with us by submitting a pull-request, based on your favorite Spring Cloud Stream's business logic represented as Supplier, Function, or Consumer functions.

Application properties as Docker labels

Instead of loading your application properties into JARs, they can now be represented as native metadata container-image labels. The application container image will include both the application uber-jar and the associated application properties, presented as native Docker labels. This magic is thanks to the Maven plugin (either using Jib or Fabric8), which intercepts and populates the application properties as native Docker labels, assembling the metadata properties at application build time. 

Because users don’t have to think about Maven artifactory and the associated configurations, this is a signification optimization compared with the previous JAR-based method for specifying properties—especially when deploying to Kubernetes. 

Redesigned batch job scheduler

We heard your feedback on the extra TaskLauncher scheduling component required to schedule batch jobs in Kubernetes. For 2.5, this is eliminated by orchestrating a scheduled launch through 'job' and 'cron job' boundaries, letting Kubernetes do the right thing to organize the deployment and manage the lifecycle semantics of ephemeral applications. To help with the migration, we also created a migration tool to export schedules from SCDF 2.3.x and 2.4.x to the latest 2.5.0 release.

Join our community!

We can’t wait for you to try out Spring Cloud Data Flow 2.5. If you want to ask questions or give us feedback, please reach out to us on Gitter, StackOverflow, or GitHub.

About the Author

Sabby Anandan

Sabby Anandan is a Product Manager on the Spring Team at VMware. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining VMware, Sabby worked in engineering and management consulting positions. He holds a Bachelor’s degree in Electrical and Electronics from the University of Madras and a Master’s in Information Technology and Management from Carnegie Mellon University.

More Content by Sabby Anandan
To Kubernetes or Not Kubernetes: What Lies Beneath the Question?
To Kubernetes or Not Kubernetes: What Lies Beneath the Question?

The evidence suggests many are advocating for Kubernetes based on a fear of missing out on the latest trend...

This Month in Spring - May 2020
This Month in Spring - May 2020