788d3c4969
In past days there were discussions about various issues
with memcached connections [1][2][3].
After investigation it looks like common root cause for above
problems is keystonemiddleware. More precisely said the way
how keystonemiddleware is caching tokens.
Currently it's using some home-made CachePool with direct
usage of memcached library, moreover it looks like its
approach is not eventlet-safe.
Discussion can be mainly found in [4].
Fortunately keystonemiddleware can use "advanced cache pool",
which is oslo.cache's implementation and was added long time ago [5],
but it is turned on only if memcache_use_advanced_pool=True.
This patch is switching to more elaborated oslo.cache CachePool
and adding deprecation warning about eventlet-unsafe variant
of keystonemiddleware's memcache pool.
How to reproduce ?
with memcache_use_advanced_pool=False
1. Build clean ENV of openstack
2. Deploy core projects (keystone,glance,nova,placement...)
3. Run while true; do COMMAND FOR SERVICE; done
- several bashes, in parallel (5-7)
COMMAND FOR SERVICE:
- openstack network list
- openstack volume list
- openstack server list
- openstack image list
4. Check memcached connections (which will grow up):
- ss | grep 11211 | wc -l every second
How to fix and test it ?
Repeat above, to fix:
- with memcache_use_advanced_pool=True
OR
- apply this patch
Compare measurements in graph.
[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852
[2] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659
[4] https://review.opendev.org/c/openstack/oslo.cache/+/742193
[5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664
Closes-Bug: #1883659
Closes-Bug: #1892852
Closes-Bug: #1888394
Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac
23 lines
1.1 KiB
YAML
23 lines
1.1 KiB
YAML
---
|
|
deprecations:
|
|
- |
|
|
We no longer recommend using the eventlet unsafe keystonemiddleware's
|
|
memcacheclientpool. This implementation may result in growing connections
|
|
to memcached.
|
|
|
|
It is recommended that the ``memcache_use_advanced_pool`` option
|
|
is set to ``True`` in the ``keystone_authtoken`` configuration section of
|
|
the various services (e.g. nova, glance, ...) when memcached is used for
|
|
token cache.
|
|
upgrade:
|
|
- |
|
|
[`bug 1892852 <https://bugs.launchpad.net/keystonemiddleware/+bug/1892852>`_]
|
|
[`bug 1888394 <https://bugs.launchpad.net/oslo.cache/+bug/1888394>`_]
|
|
[`bug 1883659 <https://bugs.launchpad.net/keystonemiddleware/+bug/1883659>`_]
|
|
Keystonemiddleware now using eventlet-safe implementation of
|
|
``MemcacheClientPool`` from oslo.cache's library by default.
|
|
The ``keystonemiddleware`` implementation is now deprecated. For backwards
|
|
compatibility, the ``[keystone_authtoken] memcache_use_advanced_pool``
|
|
option can be set to ``False`` config files of the various services (e.g.
|
|
nova, glance, ...) when memcached is used for token cache.
|