In past days there were discussions about various issues
with memcached connections [1][2][3].
After investigation it looks like common root cause for above
problems is keystonemiddleware. More precisely said the way
how keystonemiddleware is caching tokens.
Currently it's using some home-made CachePool with direct
usage of memcached library, moreover it looks like its
approach is not eventlet-safe.
Discussion can be mainly found in [4].
Fortunately keystonemiddleware can use "advanced cache pool",
which is oslo.cache's implementation and was added long time ago [5],
but it is turned on only if memcache_use_advanced_pool=True.
This patch is switching to more elaborated oslo.cache CachePool
and adding deprecation warning about eventlet-unsafe variant
of keystonemiddleware's memcache pool.
How to reproduce ?
with memcache_use_advanced_pool=False
1. Build clean ENV of openstack
2. Deploy core projects (keystone,glance,nova,placement...)
3. Run while true; do COMMAND FOR SERVICE; done
- several bashes, in parallel (5-7)
COMMAND FOR SERVICE:
- openstack network list
- openstack volume list
- openstack server list
- openstack image list
4. Check memcached connections (which will grow up):
- ss | grep 11211 | wc -l every second
How to fix and test it ?
Repeat above, to fix:
- with memcache_use_advanced_pool=True
OR
- apply this patch
Compare measurements in graph.
[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852
[2] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659
[4] https://review.opendev.org/c/openstack/oslo.cache/+/742193
[5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664
Closes-Bug: #1883659
Closes-Bug: #1892852
Closes-Bug: #1888394
Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac
788d3c4969