Ensuring Containers Are Always Running with Docker’s Restart Policy

https://blog.codeship.com/ensuring-containers-are-always-running-with-dockers-restart-policy

Getting a notification that Docker containers are down in production is one of the worst ways to spend your night. In today’s article, we’ll discuss how to use Docker’s restart policy to automatically restart containers and avoid those late-night notifications.

“How to avoid late-night notifications with @Docker’s restart policy” via @madflojo

Click To Tweet

What Happens When an Application Crashes?

Before we get started with Docker’s restart policy, let’s understand a bit more about how Docker behaves when an application crashes. To facilitate this, we’ll create a Docker container that executes a simple bash script named crash.sh.

#/bin/bash
sleep 30
exit 1

The above script is simple; when started, it will sleep for 30 seconds, and then it will exit with an exit code of 1 indicating an error.

Building and running a custom container

In order to run this script within a container, we’ll need to build a custom Docker container which includes the crash.sh script. In order to build a custom container, we first need to create a simple Dockerfile.

$ vi Dockerfile

The Dockerfile will contain the following three lines:

FROM ubuntu:14.04
ADD crash.sh /
CMD /bin/bash /crash.sh

The above Dockerfile will build a container based on the latest ubuntu:14.04 image. It will also add the crash.sh script into the / directory of the container. The final line tells Docker to execute the crash.sh script when the container is started.
With the Dockerfile defined, we can now build our custom container using the docker build command.

$ sudo docker build -t testing_restarts ./
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
 ---> e36c55082fa6
Step 2 : ADD crash.sh /
 ---> eb6057d904ef
Removing intermediate container 5199db00ba76
Step 3 : CMD /bin/bash /crash.sh
 ---> Running in 01e6f5e12c3f
 ---> 0e2f4ac52f19
Removing intermediate container 01e6f5e12c3f
Successfully built 0e2f4ac52f19

This build command created a Docker image with a tagged name of testing_restarts. We can now start a container using the testing_restarts image by executing docker run.

$ sudo docker run -d --name testing_restarts testing_restarts
a35bb16634a029039c8b34dddba41854e9a5b95222d32e3ca5624787c4c8914a

From the above, it appears that Docker was able to start a container named testing_restarts. Let’s check the status of that container by running docker ps.

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS

The docker ps command doesn’t show any running containers. The reason for this is because docker ps by default only shows running containers. Let’s take a look at running and non-running containers by using the -a flag.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
a35bb16634a0        testing_restarts    "/bin/sh -c '/bin/bas"   9 minutes ago       Exited (1) 8 minutes ago

With the docker ps results, we can see that when an application within a Docker container exits, that container is also stopped. This means that, by default, if an application that is running within a container crashes, the container stops and that container will remain stopped until someone or something restarts it.

Changing Docker’s Default Behavior

It’s possible to automatically restart crashed containers by specifying a restart policy when initiating the container. To understand restart policies better, let’s see what happens when we use the always restart policy with this same container.

$ sudo docker run -d --name testing_restarts --restart always testing_restarts
8320e96172e4403cf6527df538fb7054accf3a55513deb12bb6a5535177c1f19

In the above command, we specified that Docker should apply the always restart policy to this container via the --restart flag. Let’s see what effect this has on our container by executing a docker ps again.

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
8320e96172e4        testing_restarts    "/bin/sh -c '/bin/bas"   About a minute ago   Up 21 seconds

This time we can see that the container is up and running but only for 21 seconds. If we run docker ps again, we will see something interesting.

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
8320e96172e4        testing_restarts    "/bin/sh -c '/bin/bas"   About a minute ago   Up 19 seconds

The second run shows the container has only been up for 19 seconds. This means that even though our application (crash.sh) continues to exit with an error, Docker is continuously restarting the container every time it exits.
Now that we understand how restart policies can be used to change Docker’s default behavior, let’s take a look at what restart policies Docker has available.

Docker’s Restart Policy(ies)

Docker currently has four restart policies:

• no
• on-failure
• unless-stopped
• always

The no policy is the default restart policy and simply does not restart a container under any circumstance.

Restarting on failure but stopping on success

The on-failure policy is a bit interesting as it allows you to tell Docker to restart a container if the exit code indicates error but not if the exit code indicates success. You can also specify a maximum number of times Docker will automatically restart the container.
Let’s try this restart policy out with our testing_restarts container and set a limit of 5 restarts.

$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts
85ff2f096bac9965a9b8cffbb73c1642bf7b64a2173bbd145961231861b95819

If we run docker ps within a minute of launching the container, we will see that the container is running and has been recently started.

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
85ff2f096bac        testing_restarts    "/bin/sh -c '/bin/bas"   About a minute ago   Up 8 seconds

The same will not be true, however, if we run the docker ps command 3 minutes after launching the container.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
85ff2f096bac        testing_restarts    "/bin/sh -c '/bin/bas"   3 minutes ago       Exited (1) 20 seconds ago

We can see from the above that after 3 minutes the container is stopped. This is due to the fact that the container has been restarted more than our max-retries setting.

With success

The benefit of on-failures is that when an application exits with a successful exit code, the container will not be restarted. Let’s see this in action by making a quick minor change to the crash.sh script.

$ vi crash.sh

The change will be to set the exit code to 0.

#/bin/bash
sleep 30
exit 0

By setting the script to exit with a 0 exit code, we will be removing the error indicator from the script. Meaning as far as Docker can tell, this script will execute successfully every time.
With the script changed, we will need to rebuild the container before we can run it again.

$ sudo docker build -t testing_restarts ./
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
 ---> e36c55082fa6
Step 2 : ADD crash.sh /
 ---> a4e7e4ad968f
Removing intermediate container 88115fe05456
Step 3 : CMD /bin/bash /crash.sh
 ---> Running in fc8bbaffd9b9
 ---> 8aaa3d99f432
Removing intermediate container fc8bbaffd9b9
Successfully built 8aaa3d99f432

With the container image rebuilt, let’s launch this container again with the same on-failures and max-retries settings.

$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts
f0052e0c509dfc1c1b112c3b3717c23bc66db980f222144ca1c9a6b51cabdc19

This time, when we perform a docker ps -a execution, we should see some different results.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
f0052e0c509d        testing_restarts    "/bin/sh -c '/bin/bas"   41 seconds ago      Exited (0) 11 seconds ago

Since the crash.sh script exited with a successful exit code (0), Docker understood this as a success and did not restart the container.

Always restart the container

If we wanted the container to be restarted regardless of the exit code, we have a couple of restart policies we could use:

• always
• unless-stopped

The always restart policy tells Docker to restart the container under every circumstance. We experimented with the always restart policy earlier, but let’s see what happens when we restart the current container with the always restart policy.

$ sudo docker run -d --name testing_restarts --restart always testing_restarts
676f12c9cd4cac7d3dd84d8b70734119ef956b3e5100b2449197c2352f3c4a55

If we wait for a few minutes and run docker ps -a again, we should see that the container has been restarted even with the exit code showing success.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
9afad0ccd068        testing_restarts    "/bin/sh -c '/bin/bas"   4 minutes ago       Up 22 seconds

What’s great about the always restart policy is that even if our Docker host was to crash on boot, the Docker service will restart our container. Let’s see this in action to fully appreciate why this is useful.

$ sudo reboot

By default or even with on-failures, our container would not be running on reboot. Which, depending on what task the container performs, may be problematic.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
676f12c9cd4c        testing_restarts    "/bin/sh -c '/bin/bas"   9 minutes ago       Up 2 seconds

With the always restart policy, that is not the case. The always restart policy will always restart the container. This is true even if the container has been stopped before the reboot. Let’s look at that scenario in action.

$ sudo docker stop testing_restarts
testing_restarts
$ sudo reboot

Before rebooting our system, we simply stopped the container. This means the container is still there, just not running. Once the system is back up after our reboot however, the container will be running.

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
676f12c9cd4c        testing_restarts    "/bin/sh -c '/bin/bas"   11 minutes ago      Up 24 seconds

The reason our container is running after a reboot is because of the always policy. Whenever the Docker service is restarted, containers using the always policy will be restarted regardless of whether they were running or now.
The problem is that restarting a container that has been previously stopped after a reboot can be a bit problematic. What if our container was stopped for a valid reason, or worse, what if the container is out of date?
The solution for this is the unless-stopped restart policy.

Only stop when Docker is stopped

The unless-stopped restart policy behaves the same as always with one exception. When a container is stopped and the server is rebooted or the Docker service is restarted, the container will not be restarted.
Let’s see this in action by starting the container with the unless-stopped policy and repeating our last example.

$ sudo docker run -d --name testing_restarts --restart unless-stopped testing_restarts
fec5be52b9559b4f6421b10fe41c9c1dc3a16ff838c25d74238c5892f2b0b36

With the container running, let’s stop it and reboot the system again.

$ sudo docker stop testing_restarts
testing_restarts
$ sudo reboot

This time when the system restarts, we should see the container is in a stopped state.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                            PORTS               NAMES
fec5be52b955        testing_restarts    "/bin/sh -c '/bin/bas"   2 minutes ago       Exited (137) About a minute ago

One important item with unless-stopped is that if the container was running before the reboot, the container would be restarted once the system restarted. We can see this in action by restarting our container and rebooting the system again.

$ sudo docker start testing_restarts
testing_restarts
$ sudo reboot

After this reboot, the container should be running.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
fec5be52b955        testing_restarts    "/bin/sh -c '/bin/bas"   5 minutes ago       Up 13 seconds                           testing_restarts

The difference between always and unless-stopped may be small, but in some environments this small difference may be a critical decision.

Selecting the Best Restart Policy

When selecting the best restart policy, it’s important to keep in mind what type of workload the container is performing.
A Redis instance, for example, may be a critical component in your environment which should have an always or unless-stopped policy. On the other hand, a batch-processing application may need to be restarted until the process successfully completes. In this case, it would make sense to use the on-failures policy.
Either way, with Docker’s restart policy you can now rest assured that next time a Docker host mysteriously reboots at 3 a.m., your containers will be restarted.