This post was co-authored by Vikrant Singh and Miguel Martinez Trivino.
VMware Marketplace is a one stop-shop for VMware customers to discover, try, and deploy various third-party and open source solutions onto their VMware environments. All deployable assets on VMware Marketplace are pre-tested on their respective VMware environments, which empowers users to deploy them with confidence. However, with an increased industry focus on “more early, more often” software release cycles, these marketplace assets run the risk of quickly becoming incompatible with the latest versions of their respective deployment platforms.
VMware is excited to offer continuous and automated validation for all Tanzu ecosystem solutions on VMware Marketplace! Read more about it here.
What is continuous validation?
VMware Marketplace now offers automatic platform compatibility and functional checks on our published, cloud native, container-based solutions. This automatic process validates publisher solutions on pre-selected target platforms (i.e., Tanzu Kubernetes Grid) both during the publishing process and later on, when newer versions of the underlying target platform are released.
In this first phase, continuous validation as a feature is available for Tanzu Kubernetes Grid platforms and Helm-based Kubernetes solutions. Additional VMware-based deployment platforms and form factors will be enabled with this feature soon.
Continuous validation process flow
There are two main validations types:
-
Platform – Application-independent testing on predefined target platforms, with a focus on generic application lifecycle and platform tests. For example, VMware Marketplace will check that the application successfully comes up, self-heals, correctly persists data, and more.
-
Application – Application-specific tests provided by the publisher, which range from probing the live web application to lower-level checks that are executed inside the target container.
Why is continuous validation important?
Continuous validation benefits both sides of VMware Marketplace: the users and the publishers. For users, it guarantees that the latest version of the solution they choose will work as expected on the latest version of their chosen deployment platform. For publishers, it means the latest versions of their applications on the latest version of Tanzu Kubernetes Grid are validated in real time, eliminating the need for manual compatibility testing against every new update. Additionally, publishers have the flexibility to define custom test scenarios on their apps, giving them an extra boost of confidence that their applications will work as expected, every time.
How does it work?
The continuous validation engine input consists of:
-
The application to be tested, along with any additional deployment options
-
The platform on which the system should run the tests
-
The custom test suite, in the form of a container image along with optional test run options, for testing the application
The engine will provision the desired target platform, deploy the application, and run either a predefined common test suite for platform tests or, in the case of application tests, the custom test suite provided by the publisher. The output will contain enough information to troubleshoot and visualize the performed validation in the form of logs and XUnit reports.
Components of continuous validation
The VMware Marketplace continuous validation engine can be triggered by way of either of the two mechanisms—initiating a new solution onboarding process or accessing the self-service validation section on the listing.
Solution onboarding
Whenever a publisher publishes a new solution or a new version of an existing solution on VMware Marketplace, the continuous validation engine will automatically trigger a platform validation that consists of deploying the solution and running a generic set of tests. Publishers can provide custom Helm Chart install options to be used by the engine to deploy the app before running the tests. A successful platform validation is mandatory in order for a solution to be approved on VMware Marketplace. Both application and functional validation are optional, and can be set up and triggered manually.
Solution onboarding
Self-service validation
Additionally, publishers now have access to a new section on VMware Marketplace where they can define, run, visualize, and troubleshoot new and existing platform and application validation for their solutions.
New Marketplace UI with platform and app test features
The user interface provides a way to set up both platform and application tests where the publisher can choose the desired target platform, the application run options, and the test suite, if applicable.
Platform validation setup
Application validation setup
Once set, the publisher has the option to run those tests manually. But more importantly, the continuous validation engine will make sure to automatically re-run platform and application tests if there is a new release of the published solution or the target deployment platform.
Next steps
Continuous and automated validation is a win-win for both publishers offering containerized solutions on Tanzu as well as Marketplace users! And VMware Marketplace is working to extend this capability to other VMware deployment platforms as well as other form factors, such as virtual machines. Continue to watch this space for more exciting updates!
Learn more
Interested in learning more about VMware Marketplace? Read our introductory blog post, follow us on Twitter, or visit the catalog to peruse the more than 2,000 solutions listed. If you have any questions, please reach out to us at [email protected].
Have more questions? Be sure to read the FAQ (which can also be found under the Resources section) as well as our publisher guidelines.
