Spring Cloud Data Flow 2.1 Centers on Upgrades to Guides, Docs, and Samples

May 21, 2019 Sabby Anandan

Thanks to the wonderful Spring Cloud Data Flow community, in our recent survey, we received lots of suggestions, comments, and feedback with regards to limitations in the reference guides, documentation, and samples for Spring Cloud Data Flow.

You asked us to make it easier to find solutions to your Spring Cloud Data Flow problems!

Consequently, the theme for the 2.1 GA release became “helping the community and users,” and today we’re announcing a brand new Spring Cloud Data Flow Microsite to help developers get more from the product.


All the projects under the Spring umbrella have a significant presence in the community forums including StackOverflow, GitHub, Slack, and Gitter channels.

The projects in the Spring Cloud Data Flow (SCDF) ecosystem are well represented in all these community platforms. On any given day, you’ll find the team busy answering community questions. We are proud to regularly engage with the community.

In answering community questions and customer support tickets, we noticed a high degree of context-switching was required between different projects, their reference guides, and the samples, in order to build streaming and batch processing solutions. This was consistent with the feedback you gave us in the survey results as well.

The main reason for all the context-switching is that there are various projects in the SCDF ecosystem, each of which evolves with different features and release cadences. SCDF brings them all together into a coherent set of developer tools to build, deploy, and manage streaming and batch data pipelines. Therefore, answering community questions typically involves pointing to various project-specific resources.

Is this a side effect to building SCDF as microservices based architecture? Perhaps. This architecture has brought us enormous new benefits due to the modular design. If you count Spring Cloud Stream, Spring Cloud Task, Spring Cloud Skipper, and the other projects in the SCDF ecosystem, it’d easily encompass over 100 releases in the last 1 year!

To minimize this context-switching and to promote the easy discovery of product capabilities, we realized that we needed to deliver a step-by-step developer guide to delve into new features, enhancements, and use-case possibilities.

The Spring Cloud Dataflow Microsite was born...

Our Goals

The first and foremost goal was to go deeper as opposed to just highlighting the breadth of the features. Ultimately, we wanted to answer the questions that a developer would face when building streaming and batch processing solutions using SCDF. Building upon that thinking, we chose to start with some ‘simple-to-complex’ scenarios in an incremental style so that folks can switch to the level that matches their familiarity. Lastly, we focused heavily on repeatability — code fragments, end-to-end samples, and recipes that a developer could learn in order to become proficient with Spring Cloud Data Flow in less time.

Other Notable Updates

While the significant focus of this release was on the new Microsite, there were a few other notable improvements within the ecosystem…

  • Spring Cloud Stream 2.2 GA: The primary goal was to elevate the Spring Cloud Function programming model in Spring Cloud Stream. A developed focused Spring IO blog is in the works, so stay tuned. The RabbitMQ, Apache Kafka, and Kafka Streams binders have had new feature improvements and validation through community contributions. Lastly, Kinesis binder recently also added the support for KCL/KPL and DynamoDB streams.

  • Moving away from bit.ly for Bulk App Registration: For convenience, we have shipped bit.ly based shortened URLs to redirect to the bulk-registration property files that we host in Spring repository. This had brought a lot of complexity over the years, and we are happy to announce that we are moving away from bit.ly to replace that with URLs served from the Spring IO domain instead. Look out for the updated URLs in  reference guide.

  • Deployment Properties at Fingertips: The discovery of platform-specific deployment properties for the supported platforms were tedious. We have seen multiple reports of folks having trouble understanding how to override the deployment experience. It is now improved by a new dropdown in the UI to populate the deployer properties for the selected platform automatically. No more hunting down for properties from the docs/code, you can now choose from the dropdown.

  • Concurrent Task Launching Optimizations: Given the number of customers with filed-ingest use-cases in the cloud, we published a reference architecture to describe the end-to-end solution for Cloud Foundry and Kubernetes. One of the critical pieces in this architecture is the ability to rate-limit and throttle the concurrent task launches to avoid saturating the available resources. While the initial implementation worked for the new dataset, we received feedback on the checkpointing mechanism towards the historical datasets. We have an improved solution that now measures the actual running tasks to make downstream decisions.

  • Docker Compose / Helm Chart Customizations: With growing demands for quick getting-started experience using SCDF’s Docker Compose and the Helm Chart, we have added customizations to switch to different monitoring solutions and messaging middlewares.

It’s just the Beginning

We are barely getting started on the Spring Cloud Data Flow Microsite. We are determined to continue investing in these learning materials in future releases. We wouldn’t be able to do this without you, the community, customers, and the users, so please reach out to us in StackOverflow, GitHub, or Gitter if you see issues or if you have a new topic to include in the Microsite. Please note, everything is plain markdown files, and they get automatically generated as static web resources. It is set up to add and update by everyone quickly — we are looking forward to your contributions!


About the Author

Sabby Anandan

Sabby Anandan is a Product Manager on the Spring Team at VMware. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining VMware, Sabby worked in engineering and management consulting positions. He holds a Bachelor’s degree in Electrical and Electronics from the University of Madras and a Master’s in Information Technology and Management from Carnegie Mellon University.

More Content by Sabby Anandan
Pivotal Spring Runtime: Comprehensive Support for OpenJDK, Spring, and Apache Tomcat
Pivotal Spring Runtime: Comprehensive Support for OpenJDK, Spring, and Apache Tomcat

Pivotal Spring Runtime is comprehensive support for all your Java environments. It includes binaries and su...

Simplified Platform Networking with Pivotal Ingress Router, Powered by Istio and Envoy
Simplified Platform Networking with Pivotal Ingress Router, Powered by Istio and Envoy

Pivotal Service Mesh is a collection of platform services that improve security, traffic management, and ob...


Subscribe to our Newsletter

Thank you!
Error - something went wrong!