openstack/kolla-ansible
Code Issues Proposed changes

Constrain the size of Docker logs

Browse Source 
Even though Kolla services are configured to log output to file rather than
stdout, some stdout still occurs when for example the container re(starts).
Since the Docker logs are not constrained in size, they can fill up the
docker volumes drive and bring down the host. One example of when this is
particularly problematic is when Fluentd cannot parse a log message. The
warning output is written to the Docker log and in production we have seen
it eat 100GB of disk space in less than a day. We could configure Fluentd
not to do this, but the problem may still occur via another mechanism.

Change-Id: Ia6d3935263a5909c71750b34eb69e72e6e558b7a
Closes-Bug: #1794249
This commit is contained in:
Doug Szumski 2018-10-11 10:12:41 +01:00
parent 8045b3fbd0
commit bd54b99132
3 changed files with 13 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
ansible/group_vars/all.yml
View File

@ -92,6 +92,10 @@ docker_namespace: "kolla"
docker_registry_username:
docker_registry_insecure: "{{ 'yes' if docker_registry else 'no' }}"
# Retention settings for Docker logs
docker_log_max_file: 5
docker_log_max_size: 50m
# Valid options are [ never, on-failure, always, unless-stopped ]
docker_restart_policy: "unless-stopped"

2
ansible/roles/baremetal/templates/docker_systemd_service.j2
View File

@ -1,4 +1,4 @@
[Service]
MountFlags=shared
ExecStart=
ExecStart=/usr/bin/{{ docker_binary_name|default("docker daemon", true) }}{% if docker_registry_insecure | bool %} --insecure-registry {{ docker_registry }}{% endif %}{% if docker_storage_driver %} --storage-driver {{ docker_storage_driver }}{% endif %}{% if docker_runtime_directory %} --graph {{ docker_runtime_directory }}{% endif %}{% if docker_custom_option %} {{ docker_custom_option }}{% endif %}
ExecStart=/usr/bin/{{ docker_binary_name|default("docker daemon", true) }}{% if docker_registry_insecure | bool %} --insecure-registry {{ docker_registry }}{% endif %}{% if docker_storage_driver %} --storage-driver {{ docker_storage_driver }}{% endif %}{% if docker_runtime_directory %} --graph {{ docker_runtime_directory }}{% endif %}{% if docker_custom_option %} {{ docker_custom_option }}{% endif %} --log-opt max-file={{ docker_log_max_file }} --log-opt max-size={{ docker_log_max_size }}

8
releasenotes/notes/limit-docker-log-size-33133da03b232ece.yaml Normal file
View File

@ -0,0 +1,8 @@
---
features:
- Docker logs are no longer allowed to grow unbounded and have
been limited to a fixed size per container. Two new variables
have been added, `docker_log_max_file` and `docker_log_max_size`
which default to 5 and 50MB respectively. This means that for
each container, there should be no more than 250MB of Docker
logs.