These days, more and more of our customers are running their applications in Docker containers. From a monitoring point of view, this adds some new challenges. Classic monitoring tools usually struggle to meet the requirements, since they were built for monolithic applications. That’s why at CoScale we built a monitoring solution that can better deal with ephemeral infrastructures and the lightweight and distributed services that run in Docker containers.
In this blog post I will focus specifically on how to monitor Docker containers and the services that run inside. Traditional monitoring tools require an agent that runs on the same OS as the application. However, in a microservices environment, it is not desirable to run a monitoring agent inside a container. Not only because it adds a lot of overhead to the lightweight structures that containers are. But more importantly: because it breaks the concept of process isolation, where each container runs only a single well-bounded process.
In forthcoming blogposts, I will also discuss how to gather additional data and metadata from orchestration platforms such as Kubernetes and Swarm, to get better visibility of resource usage at a services and cluster level. But let’s first focus on the containers themselves.
Installation of the CoScale agent
To get started, I first install a CoScale agent on my host. There are two options available for installing the CoScale agent:
- The first way is by installing a package (.deb, .rpm) directly on the host.
- A second, alternative way, is by pulling the CoScale agent image from the DockerHub. This image runs the agent in its own privileged container.
This means that we only run a single agent per host, and this agent is very lightweight both in file size (25 MB), memory usage, and CPU usage (less than 1%). This makes it ideally suited for large production environments.
Setting up the Docker plugin
After I have an agent running on my machine, I can add the Docker plugin to it. I can easily do this via the CoScale user interface, or via configuration files to integrate the agent setup with my configuration management or orchestration platform.
The Docker plugin will monitor the amount of Docker containers running on my host as well as resource metrics (CPU, memory, I/O, network) per container. This is what most monitoring tools do, leveraging the Docker Stats API. But I want to know more than just general resource metrics of my containers. I also want to know the performance of the applications that run inside my containers.
Monitoring applications inside the containers
Let's start an application that I want to monitor in Docker. For this demo I will run Elasticsearch. Here is the Docker run command:
$ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch:2.1.1
Now that the container is running, I can simply configure the Docker plugin to automatically start monitoring all containers that come online that match our image:version, elasticsearch:2.1.1. This is done by enabling the Elasticsearch plugin inside the Docker plugin, as illustrated below.
This plugin connects to the Elasticsearch server running inside each container matching the image. It collects application-specific metrics about processes and threads, as well as network and JVM statistics. Our application plugins typically leverage sources such as status pages, performance counters/commands, APIs and log files to gather metrics from specific application components. The list of supported applications grows with every release, but if you feel like there are any plugins missing, let us know!
Scaling your monitoring with your application
The screenshot below shows that the Docker plugin now monitors 10 hosts that I set up, each with a couple of containers running on them
In the dashboard below, you can see that when the number of containers on the hosts increases, the running instances of Elasticsearch are automatically monitored. This is very useful because your containers will be monitored as you scale your applications, without any additional configuration needed. All you need to do is install one agent per host and then any new containers that get started on that machine will also be monitored. We will also automatically group or tag containers running the same image, so that you can easily monitor performance metrics across containers running the same image.
This was just an initial taste of the things you can do with CoScale, using our Docker plugin for general container metrics, as well as our application plugins to automatically monitor the applications running inside the containers. In a next blog post, I will discuss how monitoring your Kubernetes cluster or Docker Swarm can give you even better visibility.
Meanwhile, if you would like to get a more in depth look into our Docker monitoring solution, you can sign up for our 30 day free trial.