Managing application configuration at scale is a challenge for every IT organization. We need a better way.
Configuration is stored in many different places and formats, changes over time, differs from deployment to deployment, and must be replicated across increasing numbers of application instances. The push towards cloud scalability and continuous deployment increases the urgency of finding solutions to this problem.
Many organizations have reached for the deployment tools in their configuration management tool chain, like Chef and Puppet, to ensure that the correct configuration information is pulled from a central, well-known, authoritative source with each application instance. This works, if you are using deployment tools, if those deployments originate from a centrally managed repository, and if the only way you change configuration is through redeployment (which is becoming more and more common with the increased adoption of immutable infrastructure).
But what if your deployment tool is Cloud Foundry’s cf push command? Where does configuration information come from? How do you manage the configuration differences between deployment environments? How do you manage a change to a setting that affects many different applications? For many organizations that attempt to run cloud production at scale, sourcing the configuration from individual manifest files or managing the environment through the cf command line is not viable. Easy cross-application management of configuration settings is a necessity.
Configuration Servers to the Rescue
Many organizations are centralizing their configuration into services like etcd, consul, and Apache Zookeeper™. Dedicated configuration servers, like Devnull’s Zuul (not to be confused with the Netflix router of the same name) and our very own Spring Cloud Config Server, are also starting to make an appearance. These servers are intended to be the authoritative source for configuration information, and provide features to easily manage, review, version, and audit, configuration settings.
To leverage configuration servers, applications are expected to pull down their settings from them, instead of (or in addition to) looking for them in their traditional places, like environment variables, configuration files, and command line options. And herein lies the rub—every application now needs to change and integrate with the chosen configuration server, using its particular API, everywhere it consumes configuration information.
Spring Cloud has come to the rescue of Java applications by automating the instrumentation of Spring Boot applications to contact the config server, pull down the applicable configuration, and merge it into the Spring Framework Environment. If your Java code was already well-behaved with respect to reading its environment, it needs no further changes—configuration server settings will appear like settings from any other sources and the application won’t be aware of where they came from.
To create a solution, I set out to integrate the responsibilities of pulling down configuration properties from servers and injecting them into applications running on the Cloud Foundry platform. By doing so, the feature becomes available to all supported languages, extensible to any configuration server, and, ideally, does not require application changes at all. It removes yet another burden from application developers who are looking to migrate to the third platform.
Moving the Integration into the Platform
As mentioned, typical applications, in any programming language, look for configuration information in environment variables or configuration files (of a variety of formats). What if, in a Cloud Foundry environment, we could manipulate those standard sources and populate them with settings that come from a configuration server before the application starts?
That’s exactly the approach I took in a little proof-of-concept for a large customer. I wrote a minimal (Python) utility that runs in the application container before the real application starts. This is accomplished by having the buildpack insert the utility into the droplet, and placing a script to invoke it in .profile.d. The utility looks for service bindings to configuration servers, identified by a predefined set of tags. If it finds a bound configuration server, it connects to it using the provided url and credentials, pulls down the appropriate settings, and inserts those settings into environment variables. Voila…the application starts as it normally would, and its environment now contains the settings from the configuration server. No application changes required.
Using service bindings to create the association between the application and its configuration server enables this to work with a variety of configuration servers. For the PoC, I did a minimal integration with both Zuul (the customer’s choice) and the Spring Cloud Config Server. It also allows different config server instances in different deployments and even running without a config server in developer environments.
The utility could easily be changed to insert the settings into configuration files in any desired format instead of environment variables. Where to place the config can itself be part of the application configuration and passed in through environment variables specified in the application manifest. It would be an elegant way to adjust the utility’s behavior to match the application’s expectations.
Nothing about this mechanism is application-language specific. For my PoC I forked the Go and Python buildpacks, added the utility, and added 3 lines to the compile step. The same set of changes to any other buildpack would enable this mechanism for other languages. And since the behavior is dependent on the presence of service bindings, this buildpack change will not affect existing applications. We could even change the compile step to avoid modifying the droplet unless configuration server bindings were present.
Dynamic Configuration Updates
There is a fair amount of debate over whether we should allow configuration changes to running application instances. Dynamic changes at scale would have to deal with situations where not all instances are responsive. It is hard to guarantee that an instance modified while running will have the exact same behavior as a replacement instance started with the new configuration, especially without placing more burden on the application developer. For the purpose of this post, I won’t take a position on whether it should be done—I will just discuss what can be done.
The easiest way to handle dynamic configuration updates in a Cloud Foundry environment is to restart all applications affected by a configuration change. Restarts are supposed to be routine and quick, and they can be implemented without requiring any changes to applications. In production, we should do the restart in a rolling manner over multiple application instances, so that applications continue to be available during the restart.
Applications can also be instrumented to react to configuration changes while running. Spring Cloud does that for you, instrumenting Java applications to listen for configuration change notifications and re-reading the configuration when a change is detected. Other apps can do this themselves. They either monitor configuration sources directly, for example, by looking at timestamps on configuration files or registering for notifications that tell them when to reread the configuration. To accommodate these types of applications, we could enhance the utility to spawn a long-running process that monitors configuration changes. When a change is detected, it would update the configuration sources for the application, as it does on startup, and optionally send a (configurable) signal to the application to reread its settings.
In Summary
This demonstrates an integration with various configuration servers offered as a Cloud Foundry platform feature. It would be available to application developers using any supported language and without requiring application code changes. There is a lot left to do before we can claim this to be a supported platform feature, but my hope is that this proof of concept can serve to demonstrate the capability to customers interested in exploring the value of this integration.
Learn More:
- Watch Guido’s video of this demo
- Pivotal Cloud Foundry: Product Info | Blog Articles
- Listen to the podcast “Injecting Cloud Native Thinking into Java with Spring Boot”
- Read how Pivotal Cloud Foundry is Igniting Innovation for CenturyLink
- Find in GitHub
Editor’s Note: ©2015 Pivotal Software, Inc. All rights reserved. Apache Zookeeper is a trademark and/or registered trademarks of Pivotal Software, Inc. in the United States and/or other countries. Apache Zookeeper is a registered trademark or trademarks of the Apache Software Foundation in the United States and/or other countries.
About the Author
