Docker has been widely adopted at the Washington Post as a way of deploying applications to AWS. With Docker an application can be written in any programming language and yet still use a common build and deployment infrastructure. Once a system tool understands how to work with Docker containers, it is immediately useful to all applications regardless of which programming language is selected.

Docker also encourages a style of deployment called immutable infrastructure. With immutable infrastructure, all configuration and code changes are versioned together and follow a common deployment path from local development to production. This simplifies operations because code and configuration changes can be both deployed and rolled back using a single tool.

Running Docker containers on AWS requires some type of container scheduler, which is a system that decides which server instance to run the container on. Some examples of Docker container schedulers include Fleet, EC2 Container Service, and Kubernetes. The currently available Docker schedulers work really well for stateless applications where it does not matter what server instance the containers run on, but these same schedulers don’t work well with applications like database servers and caches that require the preservation of state between deployments. Most of these database systems use some type of replication system which needs static IP addresses assigned to each container. They also need to attach existing data volumes during a deployment to avoid copying large amounts of data over the network. Since none of the Docker container schedulers support all of these features, we built our own static Docker scheduler called Cloud Compose.

Cloud Compose was built around a simple design principle: startup an EC2 instance with a static set of Docker containers. Cloud Compose is static scheduler that follows the principles of immutable infrastructure: once the instance starts up, nothing ever changes. When you want to deploy a new change, you must terminate an existing instance and launch a replacement instance with the new Docker containers. In most cases, Cloud Compose is used to create static server clusters with a fixed size and static IP addresses. It also supports re-attaching data volumes to reduce data replication costs. These features make it ideal for systems like Kafka, MongoDB, Elasticsearch, and Memcached.

Cloud Compose Architecture

Cloud Compose is implemented as a Python command line tool that uses several configuration files to generate a cloud init script. The cloud init script runs when the instances start and runs the application using Docker Compose.

Cloud Compose needs a minimum of four configuration files: docker-compose.yml, docker-compose.override.yml, cloud-compose.yml, and cluster.sh. To make the configuration more concrete, we will look at typical MongoDB configuration.

docker-compose.yml

The docker-compose.yml defines what Docker containers to run on the EC2 instances. It also defines what ports to open, environment variables, and other Docker run settings.

Example docker-compose.yml

version: "2"
services:
  mongodb:
    ports:
      - "27018:27018"

docker-compose.override.yml

Docker Compose supports override files called docker-compose.override.yml. This feature allows for modification of an existing docker-compose.yml file. When using Cloud Compose, you will have two docker-compose.override.yml files. The local docker-compose.override.yml will contain the values that are needed for local development and the cloud version of the docker-compose.override.yml would contain the same values for deployment to AWS. For example, the local docker-compose.override.yml would have a build command for compiling a Dockerfile to run while the cloud version of docker-compose.override.yml would have the name of the Docker image that is already built.

Example local docker-compose.override.yml

version: "2"
services:
  mongos:
    build: mongodb

Example cloud docker-compose.override.yml

version: "2"
services:
  mongos:
    image: washpost/mongos:3.2

cloud-compose.yml

The cloud-compose.yml provides the parameters needed by Cloud Compose to create a cluster of EC2 instances. It includes values like instance size, subnet, and security group. The cloud-compose.yml also includes a search path, which is a list of directories for finding the other template files like the cloud version of the docker-compose.override.yml and the cluster.sh script.

Example cloud-compose.yml

cluster:
  name: mongodb-1
  search_path:
    - templates
  aws:
    ami: docker:1.10 
    username: centos
    security_groups: sg-0ad1c36d
    instance_type: t2.medium
    keypair: drydock
    nodes:
      - id: 0
        ip: 10.0.0.10
        subnet: subnet-9f0074c6

cluster.sh

The cluster.sh file is used by the cluster plugin to initialize an instance at startup. The cluster.sh script uses the Jinja2 template language so that a single script can be composed of multiple smaller scripts. You can also use variables from your environment or the cloud-compose.yml to make configuration changes easier. The image plugin users a similar script, but it is called image.sh.

Example cluster.sh

#!/bin/bash
{% include "system.mounts.sh" %}
{% include "docker.config.sh" %}
{% include "docker_compose.run.sh" %}
{% include "system.network_conf.sh" %}

Managing Secrets

Cloud Compose loads the secrets (i.e. passwords) from environment variables. Envdir is an excellent tool for managing collections of environment variables.

Sharing and Extending Configuration

With Cloud Compose, all configuration files are stored in the same Github repository. A common way for sharing configuration files is to have a common git project that includes a cloud-compose sub-directory. This sub-directory includes a set of files for creating a cluster. Other teams that want to share this configuration can import that Git repository into their repository by doing a subtree merge. See docker-mongodb for a detailed example.

If you are using a shared set of configuration files, you can also override or extend the files to include additional scripts by simply creating a templates directory in your project, copying the scripts you want to modify into that directory, and then modifying them to suit your needs. Because the scripts are modular, you can just override part of them and leave the rest the same. Once you override a part of the configuration, you also need to adjust the search path attribute in the cloud-compose.yml to refer to your template directory so it will load your configuration files first instead of the default ones provided by the shared configuration.

Components

Cloud Compose is Python library that is extended using a plugin system. There are currently two plugins: the cluster plugin and the image plugin. The image plugin is used to create a new base image that has Docker and Docker Compose already installed. The cluster plugin is the one you will use most often. It has commands for creating and updating clusters of servers.

Execution

To use Cloud Compose, you first need to install the Python library and at least one plugin using the pip command

pip install cloud-compose cloud-compose-cluster

Then you need to open a terminal window and execute the cloud-compose command inside the directory that has your cloud-compose.yml file

cd my-configs/
cloud-compose cluster up

This command will create a new cluster of EC2 instances, each running the same set of Docker containers as defined by your docker-compose.yml file.

Support Applications

To simplify the usage of Cloud Compose, we have open sourced the Cloud Compose library along with example Dockerfiles and Cloud Compose configuration scripts. Those scripts should work well with CentOS 7, but may need to be adapted to other distros if you don't use CentOS.