Rowanto Luo


Just another blog. or log.


Docker and (Console) Logs Problem

Anyone who used docker before and care about quality will have to deal with the logs problem. As we all know, docker runs in a container, and if we don't handle the logs in a special way, the logs will be lost when the container is restarted.

In order to deal with this, usually we will map the logs folder of our application to the parent host of the container, so that the logs don't get lost when the container is restarted or changed.

# example of volume mapping for log    
docker run -it -v /srv/log:/srv/log --rm rowanto/someapplication

What people often forget to deal about is the console log of the docker application. The docker container, usually only runs one command, and we could view the console log of the container (or of that one command) using:

# viewing the console log of a container
docker logs $container_id

What people often forgot is, how is docker able to view the console log again and again? Well, it actually saves the console log to a disk file, and depending on how much you log to the console, you could have a big problem.

Docker actually saves the log file per default as json in the docker folder under:

/var/lib/docker/containers/${container_id]/${container_id}-json.log

Unfortunately, that log file doesn't roll or have a limit on size like what we could usually find in a framework. So if you log into your console too much, you might run out of disk space soon.

If you never use your console log, you could just solve this the old Linux way, which is by redirecting the console logs into /dev/null on your CMD command in your docker file. Most of the time, no one actually uses the "docker logs" command to view the log. Console log is usually also logged by the framework to its output log folder, and there we have the rolling logs from the framework.

But, if you actually want to look at the console log from time to time using the "docker logs" command, you could also use the logging options which is built into docker. The documentation could be found here. Here's an example about how to use it:

# example of running container with the log options
# this will make the log rolls when it reaches 20mb, and with a max file of 5
docker run -it -v /srv/log:/srv/log --log-opt max-size=20m --log-opt max-file=5 --rm rowanto/someapplication

This problem happened to us, when we are running the Solr-Cloud server in docker containers. In our case, Solr actually logged 14GBs of console log. We were confused at first. What could be using so many spaces in the OS harddrive? The indexes and logs were saved and mapped to a separate hard drive so it should not be a problem. The OS only has 15 GBs hard drive since it's only for running docker, but now something is producing a 14GBs file. After a few "du" commands, we found out that it's the json file which is under the /var/lib/docker folder.