kolla-ansible/releasenotes/notes/reduce-rabbit-busy-waiting-085433c822165eab.yaml
John Garbutt 70f6f8e4c0 Reduce RabbitMQ busy waiting, lowering CPU load
On machines with many cores, we were seeing excessive CPU load on systems
that were not very busy. With the following Erlang VM argument we saw
RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
40 hyperthreads.

    +S 2:2

By default RabbitMQ starts N schedulers where N is the number of CPU
cores, including hyper-threaded cores. This is fine when you assume all
your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
Kolla Ansible setup. Here we go for two scheduler threads.
More details can be found here:
https://www.rabbitmq.com/runtime.html#scheduling
and here:
https://erlang.org/doc/man/erl.html#emulator-flags

    +sbwt none

This stops busy waiting of the scheduler, for more details see:
https://www.rabbitmq.com/runtime.html#busy-waiting
Newer versions of rabbit may need additional flags:
"+sbwt none +sbwtdcpu none +sbwtdio none"
But this patch should be back portable to older versions of RabbitMQ
used in Train and Stein.

Note that information on this tuning was found by looking at data from:
rabbitmq-diagnostics runtime_thread_stats
More details on that can be found here:
https://www.rabbitmq.com/runtime.html#thread-stats

Related-Bug: #1846467

Change-Id: Iced014acee7e590c10848e73feca166f48b622dc
2021-06-07 13:18:39 +01:00

14 lines
598 B
YAML

---
fixes:
- |
Fixes an issue where RabbitMQ consumes a large amount of CPU, particularly
on multi-core systems. The default RabbitMQ tuning assumes that RabbitMQ
is running on a dedicated host, which is the opposite of a typical Kolla
Ansible container setup. For more details on tuning RabbitMQ in your
environment, please see: https://www.rabbitmq.com/runtime.html#busy-waiting
https://www.rabbitmq.com/runtime.html#scheduling
upgrade:
- |
Modifies the default value of ``rabbitmq_server_additional_erl_args`` from
an empty string to ``+S 2:2 +sbwt none``.