An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc). This includes:

  • Resource handles to the database, Memcached, and other backing services
  • Credentials to external services such as Amazon S3 or Twitter
  • Per-deploy values such as the canonical hostname for the deploy

– https://12factor.net/config

We’ve leaned heavily towards using the programming language that fits the task at hand. As a result, we run about a dozen backend services in Node.js, Clojure, and Python where each is focused on a specific task. Some may call these microservices. One caveat of this architecture is that rather than configuring one application, we now have to configure many applications in different languages that may share certain configuration values.

Our solution to this problem is to centralize configuration. In other words, we configure our fleet of backend services by updating a single file. This solution seems obvious, and the benefits are numerous: no duplication, quick updates, easy referencing, no works-on-my-machine issues, etc. In our experience, many PAAS providers neglect this issue. For example, we’ve been using Deis for the latest iteration of our product. It’s an excellent, open-source platform with an amazing community. Its workflow is similar to dotCloud or Heroku. However, it’s an apparent contradiction that these platforms encourage the developer to run many small applications thanks to the ease of creation and deployment, yet they require each application to be configured individually. When multiple applications share a database or access to a 3rd-party service like S3, those configuration values are duplicated in each application.

We’ll get into our specific implementation of a solution, but before we do, it’s worth talking about the problem a bit more. In our case, because we’ve stuck to a 12 factor app pattern, all of our applications, regardless of language, derive their configuration from environment variables. In addition, each app has an associated docker container image. These container images are run on deis, and each app’s environment variables are set using a CLI tool.

$ deis config:set FOO=BAR OOF=RAB ...

Before we go on, we’d like to stress that the fact we’re using Deis isn’t terribly relevant to the formulation of the problem. It’s just necessary for context. The problem would be the same on any container platform.

So…moving on. A developer would rather set these environment variables in one place and have them distributed to every app. The design of a solution has to answer some questions. How will we make the configs easy to update? How will we store them in a secure way? How will we be able to tell who accessed/changed them? How can we revert bad changes? Should we distribute the configs at runtime or build-time? How do we accomplish these things while keeping the benefits of the 12 Factor App pattern?

For storage, we chose Hashicorp’s Vault, and we chose S3 with file versioning as its data backend. With auditing, version control, and secure storage out of the way, we only had to think about how to make the configs easy to update and how to distribute them to our apps.

Vault provides a simple REST interface for a key/value store. All that’s needed to access it is an http client and a secret token. To enable easy editing for our team members, we created a command-line tool that can be accessed as a Deis CLI plugin.

$ deis global-config -h
Edit configs that will be picked up by all apps using the receiver.

usage: deis global-config [options]

Options:
    -n --namespace <internal,user_facing>
        which config you'd like to edit. You can have as many namespaces as 
        you'd like, and one or more apps can source the config values from a
        given namespace.
    -g --github <token>
        github personal access token with "read:org" scope
        see https://github.com/settings/tokens
    -v --vault-addr <https://some-vault.com>
        address associated with the vault store where your global-config is stored

Vault enables access control via Github authentication with no additional work. Namespaces are just the key in Vault where we’ll retrieve the config for a particular environment. We name ours development, staging, and production. A note about the Vault address: since we’re able to run Vault itself on deis, our tool will automatically detect --vault-addr using the deis CLI and looking for an application named “vault”.

Running the global-config tool opens the config, a flat JSON file, in the user’s text editor of choice. Upon saving and exiting, the config is uploaded to Vault. Here’s an example of what the global config for a particular namespace might look like:

{"MONGO_HOST": "delicious.cheeseburger.net",
 "MONGO_PORT": "32123",
 "SOME_SERVICE_API_KEY": "123imakey",
 "S3_KEY": "doodoodoo",
 "S3_SECRET": "sssshhhhh"}

All that remains is determining how to distribute these environment variables to each of our apps. To accomplish this, we first configure each app with the Vault url and secret token for the appropriate namespace using deis config:set. This is acceptable because these values will rarely change, and they are the same across all apps in the namespace. In our docker build, the CMD that is executed when our container runs needs to first source a script that retrieves the config from Vault and exports it to the environment.

RESP=$(curl -s -XGET \
            --retry 20 \
            --retry-delay 5 \
            -H "X-Vault-Token: $GLOBAL_CONFIG_VAULT_TOKEN" \
            $GLOBAL_CONFIG_VAULT_URL)

echo "$RESP" \
    | jq15 -r '.data | to_entries | .[] | "\(.key)=\(.value)"' \
    > /tmp/env \
    || (echo 'Could not retrieve global config' && echo "$RESP" && exit)

while read line; do
    export "$line"
done < /tmp/env

rm /tmp/env

Et voilà! It’s as simple as that using jq and curl. Since our apps already derive their configs from environment variables, they don’t require any changes. Alternatively, we could have chosen to add these values to the environment at build-time. However, a change to the values stored in Vault would then require rebuilding every app’s container image. Sourcing the global config at runtime means that config changes only require a restart of the container. The caveat is that Vault must be available at the time of a restart.

This is a really basic solution (<200 lines of bash in our case), but it’s been a tremendous help to us so far. There is certainly room for improvement. For example, a more advanced solution could poll for config changes and terminate the app process causing the container to restart and pick-up the new config. For now, we’re content with manual restarts.

As we’ve said already, we think the general solution here could be applied to any fleet of 12 factor, containerized apps, but as an added bonus, we’re open-sourcing our particular solution to the problem. Though specific to Deis, parts of it could be applied to solutions for other platforms.

Feel free to direct any questions or feedback for this article to @justinratner.