1. Preface

This Document describes a number of good practices regarding the creation and usage of Container Images.

It is a continuously evolving work published on github.com. Technically speaking there will be editions, as git tags, so that referencing a section is simple.

As of October 2016 we the Authors agreed to revisit major parts of this document.

2. Audience

This documentation has been written specifically for the developers who are writing Dockerfiles to create images. Dockerfiles can become rather complex depending on the application being containerized so we are passing on our experience with Dockerfiles through a series of best practices.

This guide of best practices is not a reference guide of all the Dockerfile instructions. There is plenty of documentation available on the Docker website should you require that sort of content.

3. Overview

Container technology is a popular packaging method for developers and system administrators to build, ship and run distributed applications. Production use of image-based container technology requires a disciplined approach to development. This document provides guidance and recommendations for creating and managing images to control application lifecycle.

In Application Planning we discuss how to deconstruct applications into microservices, common types of images and how planning must consider the target deployment platforms.

Creating Images discusses the details about how to work with Dockerfiles, best practices, tips and tricks, and tools available to developers.

The Build section discusses the importance of automation in building, testing and maintaining images. We discuss ideal workflows, how to plan a test environment, types of testing and image certification.

Finally, Delivery covers how to get images and updates to the end-users, whether that’s inside an enterprise or public infrastructure. A key consideration is access control.

3.1. Content of the Containers is Important

When talking about containers, content is a very important thing. The content is important especially when we compare linux container technology with a classic virtual machine. Both is basically a kind of virtualization for isolating applications, but we cannot consider containers to be the same as virtual machines if it is related to security.

The big difference between linux containers and a virtual machine is the guest’s operating system, because all containers share the kernel with the host. That makes the containers much more efficient, but the fact that the kernel is shared with the host and other containers means, that some unfortunate security flaw in the host kernel creates potential door from the container, which may influence either other containers or the host itself.

That all makes the content of the container very important, because running a malicious container is always a big risk for the host machine. Every container user is advised to pay attention at what containers are running in the infrastructure.

4. Goals

4.1. Provide general guidance on containerizing applications

4.3. Describe a complete end-to-end journey of a containerized application, including the technology, tools and best practices

5. Application Planning

As you begin to contemplate the containerization of your application, there are number of factors that should be considered prior to authoring a Dockerfile. You will want to plan out everything from how to start the application, to network considerations, to making sure your image is architected in a way that can run in multiple environments like Atomic Host or OpenShift.

The very act of containerizing any application presents a few hurdles that are perhaps considered defacto in a traditional Linux environment. The following sections highlight these hurdles and offer solutions which would be typical in a containerized environment.

5.1. Persistent Storage: Simple Database Server

Introduce the basic environment and began to foreshadow the perceieved complications

5.1.1. Traditional database server

One of the simpler environments in the IT world is a database that serves one or more client nodes. A corporate directory is an example most of us can identify with. Consider the figure below.

simple db
Figure 1. Simple database topology

In this scenario, we have a single server running a Linux distribution. The server functions largely as a database server (perhaps postgres) for other clients that can connect to it on the network. The database is capable of connecting to the clients on the network using the standard TCP/IP network stack and typically a combination of TCP socket and port. In the case of posgres, the default port is 5432.

More importantly, many database implementations store the database files on reliable, Enterprise storage such as SANs or robust RAID arrays. This is done to obviously protect the database from data loss. By default, containers have immutable storage; and therefore, if the container is deleted, your data will be lost. As a developer, you will need to understand and design your containerization in a way that will allow for data to persist regardless of the state of the container.

5.1.2. Containerized Environment

In the case of a database server, retaining your data can be critical. The default storage for containers themselves is not persistent but it can be with a little planning. The most common way to allow for data persistence is using one of the several methods already available to docker or your chosen deployment platform. The following figure is a simplified containerized topography of the same database from above.

simple db containerized
Figure 2. Containerized database

Note how the container host, like the traditional Linux deployment, has enterprise storage associated with it. Through the use of a docker volumes, we can assign storage to containers and those volumes will persist irregardless of the state of the container.

For more information about planning for persistent storage, check out the Storage Options section.

5.2. Container Interconnection: Database Server with Local and Distributed Clients

By definition, distributed application components need to communicate with one another. Container technologies encourage developers to make these interconnection points explicit and provide a number of mechanisms to coordinate or otherwise enable communication between containers.

5.2.1. Traditional Database server/environment

Consider the database example in the previous section. Once we have established persistent storage for the database server, we also need to consider how database clients will connect to it. In nearly all cases these connections will occur through a socket, ever over the network or locally via UNIX domain socket special file.

(Diagram placeholder - Block for running container, inset for listening port on top of container block, distinct block outside of container showing shared/mapped directories for UNIX sockets.)

Simple non-distributed applications may assume that a database is co-located on the same server and use an established port number, or UNIX domain socket path, as their default access mechanism.

True multi-node distributed applications may host the database as a distinct node in which case communication must occur via the network. Clients that wish to use the database must be made aware of its location, either via explicit configuration or a service discovery mechanism.

interconnect single
Figure 3. Traditional DB environment using both socket and TCP/IP connections

5.2.2. Container environment - Docker

The previous example shows a traditional database where a single node allows both socket and port (TCP/IP) connections. If we were to "containerize" the database server and the database client into seperate containers, this would present a slight challenge in the architecture due to the container namespacing. Consider the following image:

single node mult containers
Figure 4. Single Node Database with server and client in separate containers

In this setup, there are actually two clients. One is containerized and the other is executing from the container host directly. The database is also containerized but isolated by namespacing as well. The database client executing on the host can still communicate with the containerized database server via TCP/IP because Docker has an internal network for containers to communication with each other and the host. Once an interconnection mechanism has been established a container developer must ensure that service containers are properly configured to allow access to these connections.

Some container coordination frameworks, such as Kubernetes, attempt to simplify this use case for containers co-located on a single node by sharing the network port space between node-local containers.

Further details and examples of networking interconnect options for various contaner frameworks and scenarios can be found in the network considerations section of this document.

For non-network connections between containers on a single node, shared filesystem locations, either for domain sockets or actual filesystem content, must be set up at the time the containers are launched. Docker, for example, allow mapping a host directory to a container directory by adding the following argument to the run command:

-v <host_directory>:<container_directory>

In our DB server example, assuming the database maintains a listening UNIX domain socket in "/var/run/postgres" we could launch both our server and client with the following argument included:

-v /var/run/postgres:/var/run/postgres

This will ensure that both the server and client see the same directory content, exported from the host, allowing the client to connect using the expected/default location.

Further details and example can be found in the storage considerations section of this document.

Another iteration on this scenario would be where the database server and clients are on different nodes and require network access to communicate. In this case, you must ensure that Docker not only exposes a port for the database container but that a port is also exposed to the network so other clients can communicate with it.

multi node single container
Figure 5. Multiple node deployment where server and client are separated

Notice how in this scenario, the database server is still containerized but the client resides on a different node. For network connections, Docker provides a simple directive in the Dockerfile to expose a port from the running container. For example, to create a Postgres DB server container that listens on the default Postgres port, you would add the following line:

EXPOSE 5432

You then also need to ensure that you perform the port mapping when the container runs using either the -P or -p flags.

5.2.3. Networking in OpenShift

Establishing network connection between containers in OpenShift is different from the standard Docker container linking approach. OpenShift uses a built-in DNS so that services can be reached by the service DNS and service IP address. In other words, applications running in a container can connect to another container using the service name. For example, if a container running the MySQL database is a database service endpoint, database will be used as a hostname to connect to it from another container. In addition to DNS record, you can also use the environment variables with IP address of the service which are provided for every container running in the same project as the service. However, if the IP address (environment variable) changes, you will need to redeploy the container. Using service names is therefore recommended.

For details, see OpenShift Documentation.

5.3. Data Initialization

A truly stateless application component acquires all configuration information via combination of discovery or injection via a cloud, container or configuration management framework. It assumes that all local storage is ephemeral and that any data that requires persistence beyond a shutdown, reboot or termination must be stored elsewhere.

In practice, many applications are not truly stateless in the sense defined above. Instead they require at least some element of persistent state or storage to be made available or "attached" to the component at the time it is launched. Frequently, this storage must be initialized the first time the application is run.

5.3.1. Examples

  • Creation of schema/tables and initial population of a relational database

  • Initialization of configuration data, such as the location of a central server or executive to which the application component should connect.

  • Initialization of unique identifying information such as a UUID, key pair or shared secrets

5.3.2. Key Issues

  • Tracking whether initialization has occurred to ensure it only happens once or, at the very least, only occurs when the user wants it to

  • Selecting persistence between restarts of a component versus persistence beyond termination/removal

  • If data is persistent beyond termination of the component, re-associating the persistent storage with a freshly launched instance of the component (be it a VM or a container)

  • If data is persistent across restarts and updates to an underlying container image, ensuring that the “old” persistent data is still . (Brent notes that our users have expectations in this area based on RPM behavior.)

5.3.3. General Approaches and Patterns - Containers

Two common patterns have emerged to address components that require one time data initialization.

  • Automatic Initialization - In this pattern, any component that requires data initialization incorporates a check into initial start up. If the check determines that persistent data is already present, it continues as normal. If persistent data is not present, it performs the required data initialization before moving on to normal start up.

  • Explicit Initialization - In this pattern users must explicitly execute an initialization step prior to running an application component. Details may differ depending on the specific container tool or framework being used.

5.3.4. Persistent Storage in Docker

Docker containers provide persistence a few different ways. Changes to the local file system of a running container persist across starts and stops but are lost if the container is ever removed. If a user requires persistence beyond removal, Docker provides the concept of "Volumes" which are available in two flavors.

  • "Data Volumes" are directories that exist outside of the container file system and whose contents persist even after a container is terminated.

  • "Bind Mounts" are host directories can also be directly mounted into a running container.

For more details on Docker storage configurations see the storage considerations section of this guide.

5.3.5. Framework Support

This is an area of active development within the various container management frameworks and there is no silver bullet.

Generally speaking, if an application component does not provide some mechanism for automatic initialization it falls to the user to identify and perform perform any expected explicit storage initialization. It is also the user’s responsibility to track the resulting persistent storage objects during the removal/termination/restart of a container or an update to the underlying container image.

Explicit Initialization - Atomic CLI

The one exception is the Atomic CLI (aka "atomic run") which provides support within its metadata format for encoding any required explicit initialization steps.

(Brief example and then reference to atomic CLI docs.)

5.4. Security and user requirements

5.5. Host and image synchronization

Some applications that run in containers require the host and container to more or less be synchronized on certain attributes so their behaviors are also similar. One such common attribute can be time. The following sections discuss best practices to keeping those attributes similar or the same.

5.5.1. Time

Consider a case where multiple containers (and the host) are running applications and logging to something like a log server. The log timestamps and information would almost be entirely useless if each container reported a different time than the host.

The best way to synchronize the time between a container and its host is through the use of bind mounts. You simply need to bind mount the host’s /etc/localtime with the container’s /etc/localtime. We use the ro flag to ensure the container can’t modify the host’s time zone by default.

In your Dockerfile, this can be accomplished by adding the following to your RUN label.

Synchronizing the timezone of the host to the container.
-v /etc/localtime:/etc/localtime:ro

5.5.2. Machine ID

The /etc/machine-id file is often used as an identifier for things like applications and logging. Depending on your application, it might be beneficial to also bind mount the machine id of the host into the container. For example. in many cases journald relies on the machine ID for identification. The sosreport application also uses it. To bind mount the machine ID of the host to the container, add something like the following to your RUN label.

Synchronizing the host machine ID with a container
-v /etc/machine-id:/etc/machine-id:ro

5.6. Considerations for images on Atomic Host and OpenShift

5.7.1. scripts

5.7.2. tar files

5.7.3. help files

5.7.4. Dockerfiles

5.8. Starting your applications within a container

At some point in the design of your Dockerfile and image, you will need to determine how to start your application. There are three prevalent methods for starting applications:

  • Call the application binary directly

  • Call a script that results in your binary starting

  • Use systemd to start the application

For the most part, there is no single right answer on which method should be used; however, there are some softer decision points that might help you decide which would be easiest for you as the Dockerfile owner.

5.8.1. Calling the binary directly

If your application is not service-oriented, calling the binary directly might be the simplest and most straight-forward method to start a container. There is no memory overhead and no additional packages are needed (like systemd and its dependencies). However, it is more difficult to deal with setting environment variables.

5.8.2. Using a script

Using a special script to start your application in a container can be a handy way to deal with slightly more complex applications. One upside is that it is generally trivial to set environment variables. This method is also good when you need to call more than a single binary to start the application correctly. One downside is that you now have to maintain the script and ensure it is always present in the image.

5.8.3. Use systemd

Using systemd to start your application is a great if your application is service-oriented (like httpd). It can benefit from leveraging well tested unit files generally delivered with the applications themselves and therefore can make complex applications that require environment variables easy to work with. One disadvantage is that systemd will increase the size of your image and there is a small amount of memory used for systemd itself.

Note
As of docker-1.10, the docker run parameter of --privileged is no longer needed to use systemd within a container.

You can implement using systemd fairly simply in your Dockerfile.

5.9. Techniques for deploying or starting images from a host

5.9.1. host systemd considerations

5.9.2. native docker (ah) unit file

example unit file - atomic create unit file

5.9.3. openshift driven

5.10. Network Considerations

5.10.1. single host

5.10.2. multi-host

5.10.3. AEP / OSE / Docker considerations

5.11. Storage Considerations

When you architect your container image, storage can certainly be a critical consideration. The power of containers is that they can mimic, replicate, or replace all kinds of applications and therein lies why you must be careful in considering how you deal with storage needs.

By nature, the storage for containers is ephemeral. This makes sense because one of the attractions of containers is that they can be easily created, deleted, replicated, and so on. If no consideration to storage is given, the container will only have access to its own filesystem. This means if the container is deleted, whatever information whether it is logs or data will be lost. For some applications, this is perfectly acceptable if not preferred.

However, if your application generates important data that should be retained or perhaps could be shared amongst multiple containers, you will need to ensure that this storage is setup for the user.

5.11.1. Persistent storage for containers: Data Volumes

Docker defines persistent storage in two ways.

  1. Data volumes

  2. Data volume containers

However at present, the use of data volumes is emerging to be the preferred storage option for users of Docker. The docker website defines a data volume as "a specially-designated directory within one or more containers that bypasses the Union File System." It has the distinct advantages that they can be shared and reused for one or more containers. Moreover, a data volume will persist even if the associated container is deleted.

Data volumes must be explicitly created and preferbaly should be named to provide it with a meaningful name. You can manually create a data volume with the docker volume create command.

Creating a data volume for persistent storage
$ docker volume create <image_name>
Note
You can also specify a driver name with the -d option
Using data volumes in a Dockerfile

For developers whose applications require persistent storage, the trick will be instantiating the data volume prior to running the image. This, however, can be achieved leveraging the LABEL metadata and applications like atomic.

We recommend that the data volume be created through the use of the INSTALL label. If you recall, the INSTALL label is meant to identify a script that should be run prior to ever running the image. In that install script, adding something like the following can be used to create the data volume.

Creating a data volume in your install script
chroot /host /usr/bin/docker volume create <image_name>

To then use the data volume, the RUN label would need to use the bind mount feature. Adding the following to your RUN label would bind mount the data volume by name:

Adding a data volume by name into your RUN label
-v <data_volume_name>:/<mount_path_inside_container>

5.11.2. Persistent storage for containers: Mounting a directory from the host

You can also leverage the host filesystem for persistent storage through the use of bind mounts. The basic idea for this is to use a directory on the host filesystem that will be bind mounted into the container at runtime. This can be simply used by adding a bind mount to your RUN label:

Bind mounting a directory from the rootfs to a running container
-v /<path_on_the_rootfs>:/<mount_path_inside_container>

One downside to this approach is that anyone with privileges to that directory on the host will be able to view and possibly alter the content.

5.11.3. OpenShift persistent storage

5.11.4. Storage backends for persistent storage

5.12. Logging

If your application logs actions, errors, and warnings to some sort of log mechanism, you will want to consider how to allows users to obtain, review, and possible retain those logs. The flexibility of a container environment can however present some challenges when it comes to logging because typically your containers are separated by namespace and cannot leverage the system logging without some explicit action by the users. There are also several solutions for logging containers like:

  • using a logging service like rsyslog or fluentd

  • setting the docker daemon’s log driver

  • logging to a file shared with the host (bind mounting)

As a developer, if your application uses logging of some manner, you should be thinking about how you will handle your log files. Each of aforementioned solutions has its pros and cons.

5.12.1. Using a logging service

Most traditional Linux systems use a logging service like rsyslog to collect and store its log files. Often the logging service will coordinate logging with journald but nevertheless it too is a service will accept log input.

If your application uses a logger and you want to take advantage of the host’s logger, you can bind mount /dev/log between the host and container as part of the RUN label like so:

-v /dev/log:/dev/log

Depending on the host distribution, log messages will now be in the host’s journald and subsequently into /var/log/messages assumming the host is using something like rsyslog.

5.12.2. Setting the log driver for docker daemon

Docker has the ability to configure a logging driver. When implemented, it will impact all containers on the system. This is only useful when you can ensure that the host will only be running your application as this might impact other containers. Therefore this method has limited usefulness unless you can ensure the final runtime environment.

5.12.3. Using shared storage with the host

The use of persistent storage can be another effective way to deal with log files whether you choose to perform a simple bind mount with the host or data volumes. Like using a logging service, it has the advantage that the logs can be preserved irregardless of the state of the container. Shared storage also reduces the potential to chew up filesystem space assigned to the container itself. You can bind mount either a file or directory between host and container using the -v flag in your RUN label.

-v <host_dir|file>:<image_dir|file>

5.12.4. Logging in OpenShift

OpenShift automatically collects logs of image builds and processes running inside containers. The recommended way to log for containers running in OpenShift is to send the logs to standard output or standard error rather than storing it in a file. This way, OpenShift can catch the logs and output them directly in the console or on the command line (oc logs).

openshift logs

For details on logs aggregation, see the OpenShift docuemntation.

5.13. Security and User considerations

5.13.1. Passing credentials and secrets

Storing sensitive data like credentials is a hot topic, especially because Docker does not provide a designated option for storing and passing secret values to containers.

Currently, a very popular way to pass credentials and secrets in a container is specifying them as environment variables at container runtime. You as a consumer of such an image don’t expose any of your sensitive data publicly.

However, this approach also has caveats:

  • If you commit such a container and push your changes to a registry, the final image will contain also all your sensitive data publicly.

  • Processes inside your container and other containers linked to your container might be able to access this information. Similarily, everything you pass as an environment variable is accessible from the host machine using docker inspect as seen in the mysql example below.

# docker run -d --name mysql_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=db -p 3306:3306 openshift/mysql-56-centos
# docker inspect openshift/mysql-56-centos

<snip>

"Env": [
            "MYSQL_USER=user",
            "MYSQL_PASSWORD=password",
            "MYSQL_DATABASE=db",

<snip>

There are other ways how to store secrets and although using environment variables might lead to leaking private data in certain corner cases, it still belongs to the safest workarounds available.

There are a couple of things to keep in mind when operating with secrets:

  • For obvious reasons, you should avoid using default passwords - users tend to forget to change the default configuration and in case a known password leaks, it can be easily misused.

  • Although squashing removes intermediate layers from the final image, secrets from those layers will still be present in the build cache.

Handling Secrets in Kubernetes

Containers running through Kubernetes can take advantage of the secret resource type to store sensitive data such as passwords or tokens. kubernetes uses tmpfs volumes for storing secrets. To learn how to create and access these, refer to the Kubernetes User Guide.

Other Projects Facilitating Secret Management

Custodia

Custodia is an open-source project that defines an API for storing and sharing secrets such as passwords and certificates in a way that keeps data secure, manageable and audiatable. Custodia uses the HTTP protocol and a RESTful API as an IPC mechanism over a local Unix Socket. Custodia is fully modular and usrs can control how authentication, authorization and API plugins are combined and exposed. You can learn more details on the project’s github repository or wiki page.

Vault

Another open-source project that aims to handle secure accessing and storing of secrets is Vault. Detailed information about the tool and use cases can be found on the Vault project website.

5.13.2. User NameSpace Mapping (docker-1.10 feature)

5.14. Preventing your image users from filling the filesystem

Most default docker deployments only set aside about 20GB of storage for each container. For many applications, that amount of storage is more than enough. But if your containerized application produces significant log output, or your deployment scenario restarts containers infrequently, file system space can become a concern.

The first step to preventing the container filesystem from filling up is to make sure your images are small and concise. This obviously will reduce how much space your image consumes from the filesystem right away. However, as a developer, the following techniques can be used by you to manage the container filesystem size.

5.14.1. Ask for a larger storage space on run

One solution to dealing with filesystem space could be to increase the amount of storage allocated to the container when your image is run. This can be achieved with the following switch to docker run.

Increase the container storage space to 60GB
--storage-opt size:60

If you are using a defined RUN label in your Dockerfile, you could add the switch to that label. However, abuse of this switch could lead to irking users and you should be prudent in using it.

5.14.2. Space considerations for logging

Logging can sometimes unknowingly consume disk space, particularily when a service or daemon has failed. If your application performs logging, or more specifically verbose logging, consider the following approaches to help keep filesystem usage down:

5.16. Deployment Considerations

Preparing applications for production distribution and deployment must carefully consider the supported deployment platforms. Production services require high uptime, injection of private or sensitive data, storage integration and configuration control. The deployment platform determines methods for load balancing, scheduling and upgrading. A platform that does not provide these services requires additional work when developing the container packaging.

5.16.1. Platform

5.16.2. Lifecycle

5.16.3. Maintenance

5.16.4. Build infrastructure

6. Creating Images

The base unit of creating an image is the Dockerfile itself. This section focuses on the instructions that make up a Dockerfile.

This chapter will not cover every Dockerfile instruction available but instead will focus on specific ones that we want to re-enforce to those who develop Dockerfiles. Docker has published a reference guide already covering each of the Dockerfile instructions. In addition, upstream docker has a nice description of best practices for Dockerfiles. It describes the various instructions that can be used to compose a Dockerfile and their best usage. Familiarize yourself with these recommendations.

6.1. Creating Base Images

6.1.1. Choosing Base Image

Images that have no parent are called base images. Docker image usually have their own root filesystem with an operating system installed. So when you want to create a new image, it either has to be based on an image that actually provides an operating system or you will need to create this layer in your image. The only difference to this are super minimal images that instead of an operating system provide only a single binary as described later in the text. There is a wide variety of base images already available on Docker Hub, so the simplest solution is to use one from there. Here are a few things that should help you determine which base image will fit your needs:

  • Linux distribution - Your personal preference and perhaps experience is a reason why to choose a certain distribution rather than another one. However, you should definitely consider whether your containerized application requires specific libraries or tools from a specific system.

  • Image size - Base images usually contain a minimal operating system with a set of tools needed for basic operations. To preserve your environment small and efficient, size should also be taken into account when picking the right base image. The size varies; you can take advantage of super small base images, such as 2MB busybox, or use a standard minimal operating system, such as Fedora or CentOS that are up to 200MB in size.

  • Updates - Not all community images are necessarily rebuilt on a regular basis or when security vulnerabilities are addressed. You should therefore consider using base images from "official repositories" on Docker Hub, and confirm their update policy in advance.

6.1.2. Creating Base Image

Once you’ve considered all options and decided to create your own base image, the process will mostly depend on the distribution you chose. Note that the major distributions have their source files available on GitHub so you still might want to consider creating an issue or opening a pull request to suggest a change in the feature set or any adjustment. Docker documentation suggests two approaches to creating a base image, using tar and building an image "FROM scratch".

Using tar

Using the tar tool is a simple way how to build a base image. As a prerequisite, you will need to set up a directory structure for chroot with all items that you wish to be part of the base image. There are various tools that might help you with this, for example debootstrap for Debian systems or supermin for RPM-based systems.

Once you have your chroot directory ready, it is as simple as running:

# tar -C <chroot_dir> -c . | docker import - <new_image_name>

Note that docker provides a set of scripts for base image creation that take advantage of tar: https://github.com/docker/docker/tree/master/contrib. Well known distributions then use their own build systems that usually also utilizes tar. For example Fedora’s koji.

FROM scratch

"scratch" is a special repository in the Docker Hub registry, created using an empty tarball. It is not meant to be pulled or run, and at any such an attempt you will most likely encounter this message: 'scratch' is a reserved name. Using scratch is ideal for creating extremely minimal images, for example for containerizing single binaries. An example is available from Docker documentation. scratch is also very handy for creating standard distribution base images. But as with tar, you’ll first need to prepare a directory structure for chroot. After that, just add the directory in your Dockerfile as follows:

FROM scratch
ADD <chroot_dir> /
CMD ["/bin/bash"]

6.2. Creating Layered Images

6.2.1. Creating Component or Application Images

6.3. Create small and concise images

It is preferable to create small and concise images whenever possible. This can be highly dependent on the application you are containerizing, but there are techniques to help you accomplish this. The following sections cover these techniques.

6.3.1. Clear packaging caches and temporary package downloads

Package managers can typically generate lots of metadata and also store downloaded content into a cache of sorts. To keep images and layers as small as possible, you should consider clearing out these caches of downloaded content. Note how the following example ends with a yum -y clean all which removes deletable yum content.

A singular RUN instruction performing multiple commands
RUN yum install -y epel-release && \
    rpmkeys --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 && \
    yum install -y --setopt=tsflags=nodocs bind-utils gettext iproute\
    v8314 mongodb24-mongodb mongodb24 && \
    yum -y clean all

There are several package managers beyond yum that should be of note: dnf, rvm, gems, cpan, pip. Most of these managers have some form of a clean up command that will handle excess cache created while performing their package management duties.

Below are examples pictured for dnf and rvm:

dnf cleanup example
RUN rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && \
    dnf -y install nodejs tar sudo git-all memcached postgresql-devel postgresql-server \
    libxml2-devel libxslt-devel patch gcc-c++ openssl-devel gnupg curl which && \
    dnf clean all && \
Ruby(rvm) cleanup example
RUN /usr/bin/curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - && \
    /usr/bin/curl -sSL https://get.rvm.io | rvm_tar_command=tar bash -s stable && \
    source /etc/profile.d/rvm.sh && \
    echo "gem: --no-ri --no-rdoc --no-document" > ~/.gemrc && \
    /bin/bash -l -c "rvm requirements && rvm install ruby 2.2.4 && rvm use 2.2.4 --default && \
    gem install bundler rake && \
    gem install nokogiri --use-system-libraries && \
    rvm cleanup all && yum clean all && rvm disk-usage all"

In the above example, notice the yum clean all called after rvm, this is because some package managers like rvm rely on others (like yum in this case) to help perform their duties. Make sure to examine your container’s layers sizes to help determine where you can eliminate excess size and keep it’s footprint size to a minimum.

Here is a listing of some package managers and the applicable cleanup commands:

Table 1. Package Managers

Package Manager

Cleanup Command

yum

yum clean all

dnf

dnf clean all

rvm

rvm cleanup all

gem

gem cleanup

cpan

rm -rf ~/.cpan/{build,sources}/*

pip

rm -rf ~/.cache/pip/*

apt-get

apt-get clean

Clearing package cache and squashing

If you squash your images after manual building or as part of an automated build process, it is not necessary to clean cache in every single relevant instruction/layer as the intermediate layers affect the previous ones in this case.

Simple example Dockerfiles below would both produce the same image if they were squashed:

Cache cleanup in a separate instruction
FROM fedora
RUN dnf install -y mariadb
RUN dnf install -y wordpress
RUN dnf clean all
Cache cleanup chained with the install command
FROM fedora
RUN dnf install -y mariadb wordpress && dnf clean all

However, without squashing, the first image would contain additional files and would be bigger than the second one.

Size comparison
# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
example             separate            54870d73715f        21 seconds ago       537.7 MB
example             chained             6a6156547888        About a minute ago   377.9 MB

Therefore, it is a good practice to write Dockerfiles in a way so that others can use it as a valid reference and are always able to reproduce the build. To ensure this, you should clean cache in every layer where applicable. In general, you should always aim to create images that are small and concise regardless of whether the final image is squashed or not.

Read more about suqashing and its repercussions in the Squashing layers section.

6.3.2. Remove unnecessary packages

In some cases, your image can end up with several packages that are not necessary to support the runtime of your application. A good example is when you actually build your application from source during the build of the image itself. Typically, when you build an application, you will pull in development (-devel) packages as well as toolchain based packages like make and gcc. Once your application is built, you may no longer need these packages for runtime depending on how your application links to libraries.

Depending on your application and which packages you added to your image, you might need to iteratively attempt to remove packages checking to make sure your application still works. One suggestion would be to remove big parts of the toolchain. And then use your package manager’s command to clean up unused packages. In the case of yum, you can remove unneeded packages like so:

Removing unnecessary packages with yum
# yum autoremove

You should run this command in an interactive shell (docker run -it --rm <image> /bin/bash) initially so you can get a feel for which packages will be removed. One upside to doing so is that you can then test run your application from the interactive shell to make sure it still works.

6.3.3. Installing Documentation

It is generally considered good practice to keep your images as small as possible. Above we have discussed that package manager caches should be cleared to reduce image sizes. You can also reduce image size by limiting the documentation being installed. If you package manager supports such a thing and then you have no expectations for users to use a shell to interact with your image, this might significantly reduce the size of your image.

Yum has an optional flag to not install documentation. The following example shows how to set the flag.

RUN yum install -y mysql --setopt=tsflags=nodocs

Note that the nodocs flag is used in some base images, for example CentOS and Fedora, and this setting gets inherited by the child layers. This can cause problems in case you want to include documentation deliberately in your layered image.

In this case, if you wish to have the documentation installed for packages from your single layer only, you have to empty the tsflags option as follows:

RUN yum -y install docker --setopt=tsflags=''

If you wish to have the documentation installed for packages from your single layer and the parent layers, you need to reinstall the packages with the empty tsflags option as follow:

RUN yum -y reinstall "*" --setopt-tsflags='' && yum install docker --setopt-tsflags=''

In case you need to have documentation included for every package from every single parent or child layer, the /etc/yum.conf file needs to be edited as follows:

RUN [ -e /etc/yum.conf ] && sed -i '/tsflags=nodocs/d' /etc/yum.conf || true
RUN yum -y reinstall "*"
RUN yum -y install <package>

6.3.4. Squashing layers

Each instruction you create in your Dockerfile results in a new image layer being created. Each layer brings additional data that are not always part of the resulting image. For example, if you add a file in one layer, but remove it in another layer later, the final image’s size will include the added file size in a form of a special "whiteout" file although you removed it. In addition, every layer contains separate metadata that add up to the overall image size as well. So what are the benefits of squashing?

  • Performance - Since all layers are copy-on-write file systems, it will take longer to build the final container from many layers. Squashing helps reduce the build time.

  • Image size - Similarly, since an image is actually a collection of other images, the final image size is the sum of the sizes of component images. With squashing, you can prevent these unwanted size additions.

  • Organization - Squashing also helps you control the structure of an image, reduce the number of layers and organize images logically.

However, Docker does not yet support squashing natively, so you will have to work around it by using alternative approaches, some of which are listed below.

docker save

You can use docker save to squash all the layers of your image into a single layer. The save command was intended for this use, so this happens to be a side effect of the process. This approach, however, is not very practical for sharing as the user will be able to only download the whole content and cannot take advantage the caching. Note that the base image layer will be included as well and might be several hundreds of megabytes in size.

Custom Tools

You will surely find a lot of utilities on the internet that facilitate layer squashing. We recommend taking advantage of Marek Goldmann’s docker-squash, which automates layer squashing and which is maintained and has been tested by the community.

Repercussions of squashing
  • When you squash an image, you will lose the history together with the metadata accompanying the layers.

  • Without the metadata, users building an image from a layered image that has been squashed are losing the idea that it happened.

  • Similarly, if you decide to include the parent layer from which your image is built into the resulting squashed image, you ultimately prevent others from seeing that this happened.

Look at the mongodb example:

# docker images openshift/mongodb-24-centos7
REPOSITORY                               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/openshift/mongodb-24-centos7   latest              d7c0c18b0ae4        16 hours ago        593.3 MB

Without squashing, you can see complete history and how each of the layers occupies space.

# docker history docker.io/openshift/mongodb-24-centos7:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d7c0c18b0ae4        About an hour ago   /bin/sh -c #(nop) CMD ["run-mongod"]            0 B
63e2ba112add        About an hour ago   /bin/sh -c #(nop) ENTRYPOINT &{["container-en   0 B
ca996db9c281        About an hour ago   /bin/sh -c #(nop) USER [184]                    0 B
8593b9473058        About an hour ago   /bin/sh -c #(nop) VOLUME [/var/lib/mongodb/da   0 B
5eca88b7872d        About an hour ago   /bin/sh -c touch /etc/mongod.conf && chown mo   0 B
9439db8f40ad        About an hour ago   /bin/sh -c #(nop) ADD dir:f38635e83f0e6943cd3   17.29 kB
12c60945cbac        About an hour ago   /bin/sh -c #(nop) ENV BASH_ENV=/usr/share/con   0 B
e6073f9a949f        About an hour ago   /bin/sh -c #(nop) ENV CONTAINER_SCRIPTS_PATH=   0 B
619bf2ae5ed8        About an hour ago   /bin/sh -c yum install -y centos-release-scl    342.6 MB
ab5deeccfe21        About an hour ago   /bin/sh -c #(nop) EXPOSE 27017/tcp              0 B
584ded9dcbca        About an hour ago   /bin/sh -c #(nop) LABEL io.k8s.description=Mo   0 B
17e3bcd28e07        About an hour ago   /bin/sh -c #(nop) ENV MONGODB_VERSION=2.6 HOM   0 B
807a1e9c5a7b        16 hours ago        /bin/sh -c #(nop) MAINTAINER SoftwareCollecti   0 B
28e524afdd05        10 days ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
044c0f15c4d9        10 days ago         /bin/sh -c #(nop) LABEL name=CentOS Base Imag   0 B
2ebc6e0c744d        10 days ago         /bin/sh -c #(nop) ADD file:6dd89087d4d418ca0c   196.7 MB
fa5be2806d4c        7 months ago        /bin/sh -c #(nop) MAINTAINER The CentOS Proje   0 B

See how the history and size changes after squashing all layers in a single one (using the script above):

# docker history docker.io/openshift/mongodb-24-centos7:squashed
IMAGE               CREATED             CREATED BY          SIZE                COMMENT
90036ed9bd1d        58 minutes ago                          522.1 MB
  • One of the biggest benefits of using layers is the posibility to reuse them. Images are usually squashed into a single big layer, which does not allow for pushing partial updates in individual layers; instead, the whole image needs to be pushed into the registry upon a change. The same applies to pulling the image from the registry.

  • Some users might rely on suqashing when it comes to sensitive data. Be cautios because squashing is not meant to "hide" content. Even though squashing removes intermediate layers from the final image, information about secrets used in those layers will stay in the build cache.

6.3.5. Chaining Commands

In general, having fewer layers improves readability. Commands that are chained together become a part of the same layer. To reduce the number of layers, chain commands together. Find a balance, though, between a large number of layers (and a great many commands), and a small number of layers (and obscurity caused by brevity).

A new layer is created for every new instruction defined. This does not necessarily mean that one instruction should be associated with only one command or definition.

Ensure transparency and provide a good overview of the content of each layer by grouping related operations together so that they together constitute a single layer. Consider this snippet:

Chained Dockerfile instruction
RUN yum install -y --setopt=tsflags=nodocs \
    httpd vim && \
    systemctl enable httpd &&
    yum clean all

Each command that is related to the installation and configuration of httpd is grouped together as a part of the same layer. This meaningful grouping of operations keeps the number of layers low while keeping the easy legibility of the layers high.

Using semi-colons (;) vs double ampersands (&&)

In the RUN instruction of Dockerfiles, it is common to string together multiple commands for efficiency. Stringing commands together in the RUN instructions are typically done with ampersands or semi-colons. However, you should consider the implications of each and their usage. The following examples illustrate the difference.

Using semi-colons as instruction conjunctions
RUN do_1; do_2

This sort of conjunction will be evaluated into do_1 and then do_2. However, using the double ampersands results in a different evaluation.

Using double ampersands as conjunctions
RUN do_1 && do_2

The ampersands change the resulting evaluation into do_1 and then do_2 only if do_1 was successful.

The use of the double ampersands as conjunctions is probably a better practice in Dockerfiles because it ensures that your instructions are completed or the build will fail. If the build were to continue and you had not closely monitored the build (or its results), then the image may not be exactly as you desired. This is particularly true with automated build systems where you will want any failure to result in the failure of the build itself.

There are certainly use cases where semi-colons might be preferred and possibly should be used. Nevertheless, the possible result of an incomplete image should be carefully considered.

6.3.6. Locales

6.4. Labels

Labels in Dockerfiles serve as a useful way to organize and document metadata used to describe an image. Some labels are only descriptive by nature, like Name whereas others, like RUN can be used to describe action-oriented metadata. Labels are often leveraged by applications, like atomic, to help the image run as the author intended. They can also for purely descriptive purposed and can viewed manually with the docker inspect <image_name> command.

The authoritative source for labels is the Container Application Generic Labels git repository.

6.4.1. When are they required?

Labels are never required per-se unless your build system or lifecycle management process requires them. However, the use of labels is highly recommended for a number of reasons:

  • As mentioned above, many container related tools can use the label metadata in meaningful ways often contributing to a better user experience.

  • The label metadata is always visible when inspecting the image. Therein, users can at least see the metadata even if their tooling does not make specific use of it. For example, the RUN label basically documents how you, as the author of the Dockerfile, expect this image to be run.

6.4.2. Descriptive labels

The descriptive labels usually are alpha-numeric strings used to describe some aspect of the image itself. Examples, might be the version and release labels which could theoretically just be integer based. The following table describes labels that are meant to be purely descriptive in nature.

Table 2. Descriptive labels
Label Description Example

changelog-url

URL of a page containing release notes for the image

TBD

name

Name of the Image

"rhel7/rsyslog"

version

Version of the image

"7.2"

release

Release number of the image

"12"

architecture

Architecture for the image

"x86_64"

build-date

Date/Time image was built as RFC 3339 date-time

"2015-12-03T10:00:44.038585Z"

vendor

Owner of the image

"Red Hat, Inc."

URL

URL with more information about the image

TBD

Summary

Brief description of the image

TBD

Description

Longer description of the image

TBD

vcs-type

The type of version control used by the container source. Generally one of git, hg, svn, bzr, cvs

"git"

vcs-url

URL of the version control repository

TBD

vcs-ref

A 'reference' within the version control repository; e.g. a git commit, or a subversion branch

"364a…​92a"

authoritative-source-url

The authoritative location in which the image is published

TBD

distribution-scope

Intended scope of distribution for image. Possible values are private, authoritive-source-only, restricted, or public

private

6.4.3. Action-oriented labels

Most action-oriented labels will be a used in the context of a docker command in order for the container to behave in a desired way. The following table describes the defined action-oriented labels.

Table 3. Action-oriented labels
Label Description Example

debug

Command to run the image with debugging turned on

tbd

help

Command to run the help command of the image

tbd

run

Command to run the image

"docker run -d --privileged --name NAME --net=host --pid=host -v /etc/pki/rsyslog:/etc/pki/rsyslog -v /etc/rsyslog.conf:/etc/rsyslog.conf -v /etc/sysconfig/rsyslog:/etc/sysconfig/rsyslog -v /etc/rsyslog.d:/etc/rsyslog.d -v /var/log:/var/log -v /var/lib/rsyslog:/var/lib/rsyslog -v /run:/run -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -e IMAGE=IMAGE -e NAME=NAME --restart=always IMAGE /bin/rsyslog.sh"

uninstall

Command to uninstall the image

"docker run --rm --privileged -v /:/host -e HOST=/host -e IMAGE=IMAGE -e NAME=NAME IMAGE /bin/uninstall.sh"

install

Command to install the image

"docker run --rm --privileged -v /:/host -e HOST=/host -e IMAGE=IMAGE -e NAME=NAME IMAGE /bin/install.sh"

stop

Command to execute before stopping container

tbd

Labels are critical to properly identifying your image and influencing how it runs. For the purposes of identification, we recommend that you at least use the following labels:

  • name

  • version

  • release

  • architecture

  • vendor

And for actionable labels, we recommend you use at least the following:

  • RUN

  • INSTALL

  • UNINSTALL

These three are the most critical for ensuring that users run the image in the manner you wish. Furthermore, tools developed to read and act upon this meta data will work correctly.

In the case that you provide a help file that does not follow the standard of a man page, then the HELP label would also be prudent.

Images that are meant to be run in OpenShift are recommended to contain a set of labels as seen in the OpenShift Origin documentation. The labels are namespaced in compliance with the Docker format; that is io.openshift for OpenShift and io.k8s for Kubernetes.

See the following example snippet from the sti-ruby image:

LABEL io.k8s.description="Platform for building and running Ruby 2.2 applications" \
      io.k8s.display-name="Ruby 2.2" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="builder,ruby,ruby22"

6.5. Template

6.6. Starting your application

Generally the CMD instruction in the Dockerfile is used by docker to start your application when the image or container is started. In the planning section, we provided some reasoning for choosing how to start your application. The following subsections will show how to implement each choice in your Dockerfile.

6.6.1. Calling the binary directly

Being the simplest of the choices, you simply need to call the binary using the CMD instruction or define an ENTRYPOINT in your Dockerfile.

CMD ["/usr/bin/some_binary"]
Using the CMD Instruction

With CMD, you can identify the default command to run from the image, along with options you want to pass to it. If there is no ENTRYPOINT in the Dockerfile, the value of CMD is the command run by default when you start the container image. If there is an ENTRYPOINT in the Dockerfile, the ENTRYPOINT value is run as the command instead, with the value of CMD used as options to the ENTRYPOINT command.

The CMD instruction can be overridden when you run the image. So, notice the different results from running mycmd in two different ways:

Any time you add an argument to the end of a docker run command, the CMD instruction inside the container is ignored. So the second example opens a bash shell instead of running the cat command. If you want to assign a command that is not overridden by options at the end of a docker run command, use the ENTRYPOINT instruction.

Using the ENTRYPOINT Instruction

Like CMD, the ENTRYPOINT instruction lets you define the command executed when you run the container image but it cannot be overridden by arguments you put at the end of a docker run line. If your Dockerfile includes an ENTRYPOINT instruction and there is also a CMD instruction, any arguments on the CMD instruction line are passed to the command defined in the ENTRYPOINT line.

This is the distinct advantage of the ENTRYPOINT instruction over the CMD instruction because the command being run is not overridden but it can be subsidized. Suppose you have an ENTRYPOINT instruction that displays two files. You could easily add an additional file to be displayed by adding it to the docker run command.

You can override the ENTRYPOINT command by defining a new entrypoint with the --entrypoint="" option on the docker command line.

6.6.2. Using a script

Using a script to start an application is very similar to calling the binary directly. Again, you use the CMD instruction but instead of pointing at the binary you point at your script that was injected into the image. The registry.access.redhat.com/rhel7/rsyslog image uses a script to start the rsyslogd application. Lets look at the two relevant instructions in its Dockerfile that make this happen.

The following instruction injects our script (rsyslog.sh) into the image in the bin dir.

ADD rsyslog.sh /bin/rsyslog.sh

The contents of the script are as follows:

#!/bin/sh
# Wrapper to start rsyslog.d with appropriate sysconfig options

echo $$ > /var/run/syslogd.pid

source /etc/sysconfig/rsyslog
exec /usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS

Notice how the script does in fact handle environment variables by sourcing the /etc/sysconfig/rsyslog file. And the CMD instruction simply calls the script.

CMD [ "/bin/rsyslog.sh" ]

6.6.3. Using systemd "inside the container"

Extending our example from starting an application with a script, the rsyslog image was started with a script. We could easily use systemd to start the application. To use systemd to start a service that has a unit file, we need to tell systemd to enable the service and then let the init process handle the rest. So instead of the ADD instruction used earlier, we would use a RUN instruction to enable the service.

RUN systemctl enable rsyslog

And then we need to change the CMD instruction to call /usr/sbin/init to let systemd take over.

RUN /usr/sbin/init

6.6.4. Using systemd to control containers

The control mechanism for most docker functions is done via the docker commands or something like the atomic application which simplifies the management of containers and images for users. But in a non-development environment, you may wish to treat your containers more like traditional services or applications. For example, you may wish to have your containers start in a specific order on boot-up. Or perhaps you wish to be able to restart (or recycle) a container because you have changed its configuration file.

There are several approaches to these sorts of function. You can make sure a specific container always starts on boot-up using the --restart switch with the docker command line when you initially run the image. There are also orchestration platforms like Kubernetes that will allow you to determine the start up order of containers even when they are distributed. But in the case where all the containers reside on a single node, systemd might just be exactly the solution. Like with traditional services, systemd is capable of making sure services both start and in the order they are specified. Moreover, any issues with startup or the container are logged like any other system service.

When using systemd to manage your containers, you are really using systemd to call docker commands (and subsequently the docker daemon) to perform the actions. Therefore, once you commit to using systemd to control a container, you will need to make sure that all start, stop, and restart actions are conducted with systemd. Failure to do so essentially decouples the docker daemon and systemd causing systemd to be out of sync.

In review, systemd is a good solution for:

  • host system services such as agents and long-running services

  • logging via journald

  • service dependant management

  • traditional service management vis systemctl

  • multi-container applications with dependencies on the same node

The configuration file below is a sample service file that can be used and edited to control your image or container. In the [Unit] section, you can declare other services needed by your image including the cases where those services are also images.

Sample template for a systemd service file
[Unit]
After=docker.service
Requires=docker.service
PartOf=docker.service
After=[cite another service]
Wants=[cite another service]

[Service]
EnvironmentFile=[path to configuration file]
ExecStartPre=-[command to execute prior to starting]
ExecStart=[command to execute for start]
ExecStartPost=/usr/bin/sleep 10
ExecStop=[command to execute for stop]
Restart=always

[Install]
WantedBy=docker.service

In the [Service] section, you can also declare the actual commands that should be run prior to start, in the case of start, and in the case of stop. These commands can either be straight base commands or docker run (or stop) commands as well. Finally, if you are using a well made image that contains labels like STOP or RUN, you could also use the atomic command. For example, a start command could simply be:

atomic run <image_name>

This works because the actual docker command to run that image is part of the image’s metadata and atomic is capable of extracting it.

The [Service] section also has an option for EnvironmentFile. In a traditional, non-containerized systemd service, this configuration file resides in /etc/sysconfig/<service_name>. In the case of a containerized application, these configuration files are not always configurable and therefore do not reside on the host’s filesystem. And in the case of where they are configurable, the EnvironmentFile is usually more important to how the service application is started. If you are starting the application within an image with systemd, then systemd will use /etc/sysconfig/<service_name> within the image itself.

For more information on writing unit files see Managing Systemd unit files.

6.7. Creating a Help file

You can now provide a man-like help file with your images that allow for users to have a deeper understanding of your image. This function now allows you to provide a:

  • more verbose description of the what the image is and does

  • understanding of how the image should be run

  • description of the security implications inherent in running the image

  • requirement if the image needs to be installed

The atomic application will allow you to display this help file trivially like so:

# atomic help <image or container name>

6.7.1. Location

The help file must be located in the images as /help.1 and in 'man' format.

6.7.2. Required headings

The following headings are strongly encouraged in the help file for an image.

NAME

Image name with short description.

DESCRIPTION

Describe in greater detail the role or purpose of the image (application, service, base image, builder image, etc.).

USAGE

Describe how to run the image as a container and what factors might influence the behavior of the image itself. Provide specific command lines that are appropriate for how the container should be run.

ENVIRONMENT VARIABLES

Explain all environment variables available to run the image in different ways without the need of rebuilding the image.

HISTORY

Similar to a Changelog of sorts which can be as detailed as the maintainer desires.

6.7.3. Optional headings

Use the following sections in your help file when applicable.

LABELS

Describe LABEL settings (from the Dockerfile that created the image) that contain pertinent information. For containers run by atomic, that could include INSTALL, RUN, UNINSTALL, and UPDATE LABELS. For more information see Container Application Generic Labels.

SECURITY IMPLICATIONS

If your image uses any privileges that you want to make the user aware of, be sure to document which ones are used and optionally why.

6.7.4. Sample template

We recommend writing the help file in the markdown language and then converting it to the man format. This is handy because github can natively display markdown so the help file can be used in multiple ways.

Sample help template in markdown format
% IMAGE_NAME (1) Container Image Pages
% MAINTAINER
% DATE

# NAME
Image_name - short description

# DESCRIPTION
Describe how image is used (user app, service, base image, builder image, etc.), the services or features it provides, and environment it is intended to run in (stand-alone docker, atomic super-privileged, oc multi-container app, etc.).

# USAGE
Describe how to run the image as a container and what factors might influence the behavior of the image itself. Provide specific command lines that are appropriate for how the container should be run. Here is an example for a container image meant to be run by the atomic command:

To pull the container and set up the host system for use by the XYZ container, run:

    # atomic install XYZimage

To run the XYZ container (after it is installed), run:

    # atomic run XYZimage

To remove the XYZ container (not the image) from your system, run:

    # atomic uninstall XYZimage

Also, describe the default configuration options (when defined): default user, exposed ports, volumes, working directory, default command, etc.

# ENVIRONMENT VARIABLES
Explain all the environment variables available to run the image in different ways without the need of rebuilding the image. Change variables on the docker command line with -e option. For example:

MYSQL_PASSWORD=mypass
                The password set for the current MySQL user.

# LABELS
Describe LABEL settings (from the Dockerfile that created the image) that contains pertinent information.
For containers run by atomic, that could include INSTALL, RUN, UNINSTALL, and UPDATE LABELS.


# SECURITY IMPLICATIONS
If you expose ports or run with privileges, note those and provide an explanation. For example:

Root privileges
    Container is running as root. Explain why is it necessary.

-p 3306:3306
    Opens container port 3306 and maps it to the same port on the Host.

--net=host --cap_add=net_admin
     Network devices of the host are visible inside the container and can be configured.

# HISTORY
Similar to a Changelog of sorts which can be as detailed as the maintainer wishes.

# SEE ALSO
Upstream repository: https://github.com/container-images/container-image-template

Does Red Hat provide MariaDB technical support on RHEL 7? https://access.redhat.com/solutions/1247193
Install and Deploy a Mariadb Container Image https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/7/single/getting-started-guide/#install_and_deploy_a_mariadb_container

6.7.5. Example help file for the rsyslog container

% RSYSLOG (1) Container Image Pages
% Stephen Tweedie
% January 27, 2016

# NAME
rsyslog \- rsyslog container image

# DESCRIPTION

The rsyslog image provides a containerized packaging of the rsyslogd daemon. The rsyslogd daemon is a
utility that supports system message logging. With the rsyslog container installed and running, you
can configure the rsyslogd service directly on the host computer as you would if the daemon were
not containerized.

You can find more information on the rsyslog project from the project Web site (http://www.rsyslog.com/doc).

The rsyslog image is designed to be run by the atomic command with one of these options:

`install`

Sets up the container to access directories and files from the host system to use for rsyslogd configuration,
logging, log rotation, and credentials.

`run`

Starts the installed container with selected privileges to the host and with logging-related files and
directories bind mounted inside the container. If the container stops, it is set to always restart.

`uninstall`

Removes the container from the system. This removes the syslog logrotate file, leave all other files
and directories associated with rsyslogd on the host system.

Because privileges are opened to the host system, the running rsyslog container can gather log messages
from the host and save them to the filesystem on the host.

The container itself consists of:
    - rhel7/rhel base image
    - rsyslog RPM package

Files added to the container during docker build include: /bin/install.sh, /bin/rsyslog.sh, and /bin/uninstall.sh.

# "USAGE"
To use the rsyslog container, you can run the atomic command with install, run, or uninstall options:

To set up the host system for use by the rsyslog container, run:

  atomic install rhel7/rsyslog

To run the rsyslog container (after it is installed), run:

  atomic run rhel7/rsyslog

To remove the rsyslog container (not the image) from your system, run:

  atomic uninstall rhel7/rsyslog

# LABELS
The rsyslog container includes the following LABEL settings:

That atomic command runs the docker command set in this label:

`INSTALL=`

  LABEL INSTALL="docker run --rm --privileged -v /:/host \
  -e HOST=/host -e IMAGE=IMAGE -e NAME=NAME \
  IMAGE /bin/install.sh"

  The contents of the INSTALL label tells an `atomic install rhel7/rsyslog` command to remove the container
  after it exits (--rm), run with root privileges open to the host, mount the root directory (/) from the hos on
  the /host directory within the container, set the location of the host file system to /host, set the name of
  the image and run the install.sh script.

`RUN=`

  LABEL RUN="docker run -d --privileged --name NAME \
  --net=host --pid=host \
  -v /etc/pki/rsyslog:/etc/pki/rsyslog \
  -v /etc/rsyslog.conf:/etc/rsyslog.conf \
  -v /etc/sysconfig/rsyslog:/etc/sysconfig/rsyslog \
  -v /etc/rsyslog.d:/etc/rsyslog.d \
  -v /var/log:/var/log \
  -v /var/lib/rsyslog:/var/lib/rsyslog \
  -v /run:/run \
  -v /etc/machine-id:/etc/machine-id:ro \
  -v /etc/localtime:/etc/localtime:ro \
  -e IMAGE=IMAGE -e NAME=NAME \
  --restart=always IMAGE /bin/rsyslog.sh"

  The contents of the RUN label tells an `atomic run rhel7/rsyslog` command to open various privileges to the host
  (described later), mount a variety of host files and directories into the container, set the name of the container,
  set the container to restart automatically if it stops, and run the rsyslog.sh script.


`UNINSTALL=`

  LABEL UNINSTALL="docker run --rm --privileged -v /:/host \
  -e HOST=/host -e IMAGE=IMAGE -e NAME=NAME \
  IMAGE /bin/uninstall.sh"

  The contents of the UNINSTALL label tells an `atomic uninstall rhel7/rsyslog` command to uninstall the rsyslog
  container. Stopping the container in this way removes the container, but not the rsyslog image from your system.
  Also, uninstalling leaves all rsyslog configuration files and log files intact on the host (only removing the
  syslog logrotate file).

`BZComponent=`

The bugzilla component for this container. For example, "BZComponent="rsyslog-docker".

`Name=`

The registry location and name of the image. For example, "Name="rhel7/rsyslog":

`Version=`

The Red Hat Enterprise Linux version from which the container was built. For example, "Version="7.2".

`Release=`

The specific release number of the container Release="12.1.a":

`Architecture=`

The machine architecture associated with the Red Hat Enterprise Linux release. For example, "Architecture="x86_64"

When the atomic command runs the rsyslog container, it reads the command line associated with the selected option
from a LABEL set within the Docker container itself. It then runs that command. The following sections detail
each option and associated LABEL:

.SH "SECURITY IMPLICATIONS"
The rsyslog container is what is referred to as a super-privileged container. It is designed to have almost complete
access to the host system as root user. The following docker command options open selected privileges to the host:

`-d`

Runs continuously as a daemon process in the background

`--privileged`

Turns off security separation, so a process running as root in the container would have the same access to the
host as it would if it were run directly on the host.

`--net=host`

Allows processes run inside the container to directly access host network interfaces

`--pid=host`

Allows processes run inside the container to see and work with all processes in the host process table

`--restart=always`

If the container should fail or otherwise stop, it would be restarted

.SH "HISTORY"
Similar to a Changelog of sorts which can be as detailed as the maintainer wishes.

.SH "AUTHORS"

Stephen Tweedie

6.7.6. Converting markdown to man format

There are several methods for converting markdown format to man format. One prevalent method is to use go-md2man supplied by the golang-github-cpuguy83-go-md2man package. To convert from markdown to man using this utility, you do as follows:

# go-md2man -in path_to_man_file -out output_file

6.8. Creating a Changelog

6.9. The Dockerfile linter

6.9.1. What is the linter?

The Dockerfile-lint is a rule based 'linter' for verifying Dockerfiles. The rules are used to check file syntax and best practice options for things such as:

  • Was yum clean up evoked after a package installation?

  • In the RUN section did the writer link commands via semicolons or double ampersands?

These are determined by the rules author and are typically defined by best practices and writer requirements. The input rules are defined via a set of yaml files.

At the time of this writing, there are a number of templates from base to automated build configurations.

6.9.2. Where do I get the linter and how do I install it?

There are two iterations of the linter.

The CLI version of the project is available here: Dockerfile_lint

The online version of the project is available here: Online linter

For the next section, we will assume that the CLI version of the linter will be installed manually via npm using the following commands:

git clone https://github.com/projectatomic/dockerfile_lint/
cd dockerfile_lint
npm install

6.9.3. Where do I get the templates?

Built in Templates

In the config directory, there are two base ruleset files. If the dockerfile_lint is executed without -r these are the base rules used.

  • dockerfile_lint/config/default_rules.yaml

  • dockerfile_lint/config/base_rules.yaml

Addition Types of templates

In the sample_rules directory there are some included templates for OpenShift and an example base template.

  • Basic rules - dockerfile_lint/sample_rules/basic_rules.yaml

This set of rules is a basic catchall of your typical Dockerfile. Things such as yum cache clean up and command execution etiquette are checked. These are the rules we will be referencing below.

  • OpenShift Template - dockerfile_lint/sample_rules/openshift.yaml

In addition to testing the semantics of the basic template from above, The OpenShift template checks for some required OpenShift labels specific to its use.

6.9.4. How do I read and customize the templates?

The filename of the basic template is included in the command above: sample_rules/basic_rules.yaml

The rules are implemented using regular expressions matched on instruction of the dockerfile. The rule file has 3 sections: a profile section, a line rule section and a required instructions section.

Profile Section

The profile section gives information about the rule file. This is the name identifier and description for the profile. This information should help users to identify an applicable template.

profile:
  name: "Default"
  description: "Default Profile. Checks basic syntax."
  includes:
    - recommended_label_rules.yaml

An excerpt from the rules shows how includes are defined:

includes:
  - recommended_label_rules.yaml

The include section allows for chaining rulesets of multiple sources. In the above example the recommended_label_rules.yaml is processed in addition to its source.

Line Rule Section

This section contains rules match on a given instruction in the dockerfile. The line rules do the bulk of the dockerfile parsing.

The example below shows rules to run against the 'FROM' instruction.

  FROM registry.access.redhat.com/rhel7:latest

The excerpt below checks for the latest flag in the 'FROM' line.

line_rules:
  FROM:
    paramSyntaxRegex: /^[a-z0-9./-]+(:[a-z0-9.]+)?$/
      rules:
        -
          label: "is_latest_tag"
          regex: /latest/
          level: "error"
          message: "base image uses 'latest' tag"
          description: "using the 'latest' tag may cause unpredictable builds. It is recommended that a specific tag is used in the FROM line or *-released which is the latest supported release."
          reference_url:
            - "https://docs.docker.com/reference/builder/"
            - "#from"

Here is another example that parses the 'RUN' line.

  RUN yum -y --disablerepo=\* --enablerepo=rhel-7-server-rpms install yum-utils && \
    yum-config-manager --disable \* && \
    yum-config-manager --enable rhel-7-server-rpms && \
    yum clean all

    RUN yum -y install file open-vm-tools perl open-vm-tools-deploypkg net-tools && \
    yum clean all

The regex below checks to see if the yum command has been issued. If it has, check to see if yum clean all has been run as well.

  RUN:
    paramSyntaxRegex: /.+/
      rules:
        -
           label: "no_yum_clean_all"  #This is a short description of the rule
           regex: /yum(?!.+clean all|.+\.repo)/g  #regex the linter is attempting to match
           level: "warn" # warn, error or info: These results will define how the linter exits
           message: "yum clean all is not used"
           description: "the yum cache will remain in this layer making the layer unnecessarily large"
           reference_url:
             - "http://docs.projectatomic.io/container-best-practices/#"
             - "_clear_packaging_caches_and_temporary_package_downloads"
            # Lastly, any best practice documentation that may be pertinent to the rule
Required Instructions Section

While the line rules section uses regex the required instructions looks for the instantiation of the instruction.

required_instructions:
  -
    instruction: "EXPOSE"
    count: 1
    level: "info"
    message: "There is no 'EXPOSE' instruction"
    description: "Without exposed ports how will the service of the container be accessed?"
    reference_url:
      - "https://docs.docker.com/reference/builder/"
      - "#expose"

6.9.5. How do I use the linter?

Execution of the CLI version of the linter may look like this:

dockerfile_lint -f /path/to/dockerfile -r sample_rules/basic_rules.yaml

Here is some sample output from the command above:

--------ERRORS---------

ERROR: Maintainer is not defined. The MAINTAINER line is useful for identifying the author in the form of MAINTAINER Joe Smith <joe.smith@example.com>.
Reference -> https://docs.docker.com/reference/builder/#maintainer

--------INFO---------

INFO: There is no 'ENTRYPOINT' instruction. None.
Reference -> https://docs.docker.com/reference/builder/#entrypoint

By default, the linter runs in strict mode (errors and/or warnings result in non-zero return code). Run the command with '-p' or '--permissive to run in permissive mode:

dockerfile_lint  -p -f /path/to/dockerfile

This allows for quick and automated testing as what is informational and what needs to be addressed immediately.

6.10. Dockerfiles

6.10.1. Location

The Dockerfiles for many of the public images are hosted in git repositories where users can view them. This also allows users to customize them as well.

It is also good practice to include the Dockerfile in the image itself. Some distributions have begun to include an image’s Dockerfile in the directory /root/buildinfo. Consider following the same approach to make sure your Dockerfiles can be easily found.

Upstream Dockerfiles should be hosted in a public GIT repository, for example GitHub. Ideally, the repository should be created under the organization relevant to a particular project. For example, Software Collections Dockerfiles are available under the GitHub sclorg organization.

6.10.2. Images

Upstream Docker images, such as CentOS and Fedora base images and layered images based on these, should be publicly available on Docker Hub.

For details on using the Docker Hub registry, see Docker User Guide.

6.10.3. Content

Docker is a platform that enables applications to be quickly assembled from components. When creating Docker images, think about the added value you can provide potential users with. The intention should always be bringing some added functionality on top of plain package installation.

As an example, take this Word Press Dockerfile. After running the image and linking it with a database image such as mysql, you will get a fully operational Word Press instance. In addition, you can also specify an external database.

This exactly is the purpose of using Docker images; instead of laborious installation and configuration of separate components, you simply pull an image from a registry, acquiring a set of tools ready to be used right out-of-the-box.

6.10.4. Enabling Necessary Repositories

TBD

6.10.5. Users

TBD

6.10.6. Working Directory

TBD

6.10.7. Exposing Ports

The EXPOSE instruction declares the ports on which a container will listen for incoming connections. You should specify ports your application commonly uses; for example, as seen in this mysql example:

EXPOSE 3306
Important
The TCP/IP port numbers below 1024 are special in that normal users are not allowed to bind on them.

Therefore, for example for Apache server, ports 8080 or 8433 (HTTP or HTTPS) should be exposed. Otherwise, only the root user will be allowed to run Apache server inside a container.

6.10.8. Logging

TBD

…​

6.11. References

Please see the following resources for more information on the Docker container technology and project-specific guidelines.

Docker Documentation — Detailed information about the Docker platform.

OpenShift Guidelines — Guidelines for creating images specific to the OpenShift project.

7. Building Applications

7.1. Simple build

7.2. Use a build service

7.3. Container Development Kit

7.3.1. OpenShift VM

7.3.2. Kubernetes VM

7.3.3. Eclipse/Docker VM

OLD - OLD - OLD -OLD - OLD - OLD -OLD OLD OLD OLD - OLD - OLD -OLD - OLD - OLD -OLD OLD OLD Building a single Docker image once is a simple matter.

sudo docker build -t <registry_URL>/some/image .

This will build the image which could then be pushed to a registry location. Done. However, this immutable image will need to be updated. And this image depends on other images which will be updated, which means this image will need to be rebuilt. If this image is part of a microservice application it is just one of several images that work together as integrated services that comprise an application. Do you really want a developer to build production services from their laptop?

Serious work with container technology should automate builds. While there are some unique challenges specific to container automation, generally following continuous integration and delivery best practices is recommended.

7.4. Build Environment

A build environment should have the following characteristics

  • is secure by limiting direct access to the build environment

  • limits access to configure and trigger builds

  • limits access to build sources

  • limits access to base images, those images referenced in the FROM line of a Dockerfile

  • provides access to build logs

  • provides some type of a pipeline or workflow, integrating with external services to trigger builds, report results, etc.

  • provides a way to test built images

  • provides a way to reproduce builds

  • provides a secure registry to store builds

  • provides a mechanism to promote tested builds

  • shares the same kernel as the target production runtime environment

A build environment that meets these requirements is difficult to create from scratch. An automation engine like Jenkins is essential to managing a complex pipeline. While a virtual machine-based solution could be created, it is recommended that a dedicated, purpose-built platform such as OpenShift be used.

8. Testing

8.1. What Should be Tested

Container images generally consist of distribution packages and some scripts that help start the container properly. For example, a container with a MariaDB database typically consists of a set of RPMs, such as mariadb-server, that provides the main functionality and some scripts that handle initialization, setup, etc. A simplified example of MariaDB Dockerfile may look like this:

FROM fedora:24

RUN dnf install -y mariadb-server && \
    dnf clean all && \
    /usr/libexec/container-setup

ADD run-mysqld /usr/bin/
ADD container-setup /usr/libexec/

VOLUME ["/var/lib/mysql/data"]
USER 27

CMD ["run-mysqld"]

If we want to test the basic functionality of the container, we do not have to test the RPM functionality. The benefit of using the distribution packaging is that we know testing has already been done during the RPM develpment process. Instead, we need to focus on testing the added scripts instead and the API of the container. The goal is to determine if it works as described. For example, a MariaDB database container, we do not run the MariaDB unit tests in the container. Instead we focus on whether the database is initialized, configured, and responds to commands properly.

8.2. Conventions for Test Scripts

It is good practice to keep the basic sanity tests for the image together with the image sources. For example test/run might be a script, that tests the image specified by the IMAGE_NAME environment variable, so users may specify an image which should be tested.

8.3. Examples of test scripts

These examples of test scripts can be found in the container images for Software Collections:

A minimal script that verifies a container image by running it as daemon and then running a script that checks the proper functionality, is shown below. It stores the IDs of the containers created during the test in a temporary directory. This makes it easy to clean up those containers after the test finishes.

#!/bin/bash
#
# General test of the image.
#
# IMAGE_NAME specifies the name of the candidate image used for testing.
# The image has to be available before this script is executed.
#

set -exo nounset
shopt -s nullglob

IMAGE_NAME=${IMAGE_NAME-default-image-name}
CIDFILE_DIR=$(mktemp --suffix=test_cidfiles -d)

# clears containers run during the test
function cleanup() {
  for cidfile in $CIDFILE_DIR/* ; do
    CONTAINER=$(cat $cidfile)

    echo "Stopping and removing container $CONTAINER..."
    docker stop $CONTAINER
    exit_status=$(docker inspect -f '{{.State.ExitCode}}' $CONTAINER)
    if [ "$exit_status" != "0" ]; then
      echo "Dumping logs for $CONTAINER"
      docker logs $CONTAINER
    fi
    docker rm $CONTAINER
    rm $cidfile
    echo "Done."
  done
  rmdir $CIDFILE_DIR
}
trap cleanup EXIT

# returns ID of specified named container
function get_cid() {
  local id="$1" ; shift || return 1
  echo $(cat "$CIDFILE_DIR/$id")
}

# returns IP of specified named container
function get_container_ip() {
  local id="$1" ; shift
  docker inspect --format='{{.NetworkSettings.IPAddress}}' $(get_cid "$id")
}

# runs command to test running container
function test_image() {
  local name=$1 ; shift
  echo "  Testing Image"
  docker run --rm $IMAGE_NAME get_status `get_container_ip $name`
  echo "  Success!"
}

# start a new container
function create_container() {
  local name=$1 ; shift
  cidfile="$CIDFILE_DIR/$name"
  # create container with a cidfile in a directory for cleanup
  docker run --cidfile $cidfile -d $IMAGE_NAME
  echo "Created container $(cat $cidfile)"
}


# Tests.

create_container test1
test_image test1

9. Maintaining

9.1. Lifecycle

9.1.1. techniques for upgrades

9.2. Garbage Collection

10. Appendix

10.1. Atomic application

The atomic application is meant to be a user-friendly way to install, manage, and run container images using container. Unlike docker commands, it leverages the ability to read and act upon LABELs defined in a Dockerfile. By its nature, it helps authors of Dockerfiles to have their images run as they intended without a significant amount of knowledge by the end users. Atomic has several subcommands:

  • diff

  • help

  • host

  • info

  • install

  • images

  • migrate

  • mount

  • push

  • scan

  • stop

  • run

  • top

  • uninstall

  • unmount

  • update

  • version

  • verify

Knowledge of these subcommands can make you a better author of Dockerfiles because you too can simplify end users' interactions with your images. Look to the following sections for more information on these relevant subcommands.

10.1.1. Displaying the help file

You can display the help file for a container using the atomic command. For example, to display the help file for a container called foobar, you would run the command:

# atomic help foobar

This is the default method of displaying the help file. In this case, the help file is formatted like any typical man page and is displayed (like man) with a pager. The help file needs to be located in the / of the image filesystem and called help.1.

If no help file is present and there is no HELP LABEL defined, atomic will simply tell you it could not find any help associated with that container or image.