Document issues in scale testing of fuel-ccp
This document provides details of issues found in scale testing of fuel-ccp tool and OpenStack containerized control plane installed by it. It is split into two sections, one dedicated to Kubernetes issues, and the other to OpenStack-specific problems. Change-Id: Ided3c47f7425d5e8b5fe2bdea9e794fbb4d550ec
This commit is contained in:
Oleg Gelbukh
parent
b37458301a
commit
fb8a2f1e35
@ -24,6 +24,7 @@ Contents
|
|||||||
labs/index.rst
|
labs/index.rst
|
||||||
test_plans/index
|
test_plans/index
|
||||||
test_results/index
|
test_results/index
|
||||||
|
issues/index
|
||||||
|
|
||||||
.. raw:: pdf
|
.. raw:: pdf
|
||||||
|
|
||||||
|
|||||||
12
doc/source/issues/index.rst
Normal file
12
doc/source/issues/index.rst
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
.. raw:: pdf
|
||||||
|
|
||||||
|
PageBreak oneColumn
|
||||||
|
|
||||||
|
=======================
|
||||||
|
Issues Analysis
|
||||||
|
=======================
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 2
|
||||||
|
|
||||||
|
scale_testing_issues
|
||||||
881
doc/source/issues/scale_testing_issues.rst
Normal file
881
doc/source/issues/scale_testing_issues.rst
Normal file
@ -0,0 +1,881 @@
|
|||||||
|
.. _scale_testing_issues:
|
||||||
|
|
||||||
|
======================================
|
||||||
|
Kubernetes Issues At Scale 900 Minions
|
||||||
|
======================================
|
||||||
|
|
||||||
|
Glossary
|
||||||
|
========
|
||||||
|
|
||||||
|
- **Kubernetes** is an open-source system for automating deployment,
|
||||||
|
scaling, and management of containerized applications.
|
||||||
|
|
||||||
|
- **fuel-ccp**: CCP stands for “Containerized Control Plane”. The goal
|
||||||
|
of this project is to make building, running and managing
|
||||||
|
production-ready OpenStack containers on top of Kubernetes an
|
||||||
|
easy task for operators.
|
||||||
|
|
||||||
|
- **OpenStack** is a cloud operating system that controls large pools
|
||||||
|
of compute, storage, and networking resources throughout a
|
||||||
|
datacenter, all managed through a dashboard that gives
|
||||||
|
administrators control while empowering their users to provision
|
||||||
|
resources through a web interface.
|
||||||
|
|
||||||
|
- **Heat** is an OpenStack service to orchestrate composite cloud
|
||||||
|
applications using a declarative template format through an
|
||||||
|
OpenStack-native REST API.
|
||||||
|
|
||||||
|
- **Slice** is a set of 6 VMs:
|
||||||
|
|
||||||
|
- 1x Yahoo! Benchmark (ycsb)
|
||||||
|
|
||||||
|
- 1x Cassandra
|
||||||
|
|
||||||
|
- 1x Magento
|
||||||
|
|
||||||
|
- 1x Wordpress
|
||||||
|
|
||||||
|
- 2x Idle VM
|
||||||
|
|
||||||
|
|
||||||
|
Setup
|
||||||
|
=====
|
||||||
|
|
||||||
|
We had about 181 bare metal machines, 3 of them were used for Kubernetes
|
||||||
|
control plane services placement (API servers, ETCD, Kubernetes
|
||||||
|
scheduler, etc.), others had 5 virtual machines on each node, where
|
||||||
|
every VM was used as a Kubernetes minion node.
|
||||||
|
|
||||||
|
Each bare metal node has the following specifications:
|
||||||
|
|
||||||
|
- HP ProLiant DL380 Gen9
|
||||||
|
|
||||||
|
- **CPU** - 2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|
||||||
|
|
||||||
|
- **RAM** - 264G
|
||||||
|
|
||||||
|
- **Storage** - 3.0T on RAID on HP Smart Array P840 Controller, HDD -
|
||||||
|
12 x HP EH0600JDYTL
|
||||||
|
|
||||||
|
- **Network** - 2x Intel Corporation Ethernet 10G 2P X710
|
||||||
|
|
||||||
|
Running OpenStack cluster (from Kubernetes point of view) is represented
|
||||||
|
with the following numbers:
|
||||||
|
|
||||||
|
1. OpenStack control plane services are running within ~80 pods on 6
|
||||||
|
nodes
|
||||||
|
|
||||||
|
2. ~4500 pods are spread across all remaining nodes, 5 pods on each.
|
||||||
|
|
||||||
|
Kubernetes architecture analysis obstacles
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
During the 900 nodes tests we used `Prometheus <https://prometheus.io/>`__
|
||||||
|
monitoring tool for the
|
||||||
|
verification of the resources consumption and the load put on core
|
||||||
|
system, Kubernetes and OpenStack levels services. During one of the
|
||||||
|
Prometheus configuration optimisations old data from the Prometheus
|
||||||
|
storage was deleted to improve Prometheus API speed, and this old data
|
||||||
|
included 900 nodes cluster information, therefore we have only partial
|
||||||
|
data being available for the post run investigation. This fact,
|
||||||
|
although, does not influence overall reference architecture analysis, as
|
||||||
|
all issues, that were observed during the containerized OpenStack setup
|
||||||
|
testing, were thoughtfully documented and debugged.
|
||||||
|
|
||||||
|
To prevent monitoring data loss in future (Q1 2017 timeframe and
|
||||||
|
further) we need to proceed with the following improvements of the
|
||||||
|
monitoring setup:
|
||||||
|
|
||||||
|
1. Prometheus by default is more optimized to be used as real time
|
||||||
|
monitoring / alerting system, and there is an official
|
||||||
|
recommendation from Prometheus developers team to keep monitoring
|
||||||
|
data retention for about 15 days to keep tool working in quick
|
||||||
|
and responsive manner. To keep old data for the post-usage
|
||||||
|
analytics purposes external store requires to be configured.
|
||||||
|
|
||||||
|
2. We need to reconfigure monitoring tool (Prometheus) to include data
|
||||||
|
backup to one of the persistent time series databases (e.g.
|
||||||
|
InfluxDB / Cassandra / OpenTSDB) that’s supported as an external
|
||||||
|
persistent data store by Prometheus. This will allow us to store
|
||||||
|
old data for extended amount of time for post-processing needs.
|
||||||
|
|
||||||
|
Observed issues
|
||||||
|
===============
|
||||||
|
|
||||||
|
Huge load on kube-apiserver
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Both API servers, running in Kubernetes cluster, were utilising up to
|
||||||
|
2000% of CPU (up to 45% of total node compute performance capacity)
|
||||||
|
after we migrated them to hardware nodes. Initial setup with all nodes
|
||||||
|
(including Kubernetes control plane nodes) running on virtualized
|
||||||
|
environment was showing not workable API servers at all.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
All services that are placed not on Kubernetes masters (``kubelet``,
|
||||||
|
``kube-proxy`` on all minions) access ``kube-apiserver`` via local
|
||||||
|
``ngnix`` proxy.
|
||||||
|
|
||||||
|
Most of those requests are watch requests that stay mostly idle after
|
||||||
|
they are initiated (most timeouts on them are defined to be about 5-10
|
||||||
|
minutes). ``nginx`` was configured to cut idle connections in 3 seconds,
|
||||||
|
which makes all clients to reconnect and (the worst) restart aborted SSL
|
||||||
|
session. On the server side it makes ``kube-apiserver`` consume up to 2000%
|
||||||
|
CPU resources and other requests become very slow.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Set ``proxy_timeout`` parameter to 10 minutes in ``nginx.conf`` config
|
||||||
|
file, which should be more than enough not to cut SSL connections before
|
||||||
|
requests time out by themselves. After this fix was applied, one
|
||||||
|
api-server became to consume 100% of CPU (about 2% of total node compute
|
||||||
|
performance capacity), the second one about 200% (about 4% of total node
|
||||||
|
compute performance capacity) of CPU (with average response time 200-400
|
||||||
|
ms).
|
||||||
|
|
||||||
|
Upstream issue (fixed)
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Make Kargo deployment tool set ``proxy_timeout`` to 10 minutes:
|
||||||
|
`issue <https://github.com/kubernetes-incubator/kargo/issues/655>`__
|
||||||
|
fixed with `pull request <https://github.com/kubernetes-incubator/kargo/pull/656>`__
|
||||||
|
by Fuel CCP team.
|
||||||
|
|
||||||
|
KubeDNS cannot handle big cluster load with default settings
|
||||||
|
------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
When deploying OpenStack cluster on this scale, ``kubedns`` becomes
|
||||||
|
unresponsive because of high load. This end up with very often error
|
||||||
|
appearing in logs of ``dnsmasq`` container in ``kubedns`` pod::
|
||||||
|
|
||||||
|
Maximum number of concurrent DNS queries reached.
|
||||||
|
|
||||||
|
Also ``dnsmasq`` containers sometimes get restarted due to hitting high
|
||||||
|
memory limit.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
First of all, ``kubedns`` seems to fail often on high load (or even without
|
||||||
|
load), during the experiment we observed continuous kubedns container
|
||||||
|
restarts even on empty (but big enough) Kubernetes cluster. Restarts
|
||||||
|
are caused by liveness check failing, although nothing notable is
|
||||||
|
observed in any logs.
|
||||||
|
|
||||||
|
Second, ``dnsmasq`` should have taken load off ``kubedns``, but it needs some
|
||||||
|
tuning to behave as expected for big load, otherwise it is useless.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
This requires several levels of fixing:
|
||||||
|
|
||||||
|
1. Set higher limits for ``dnsmasq`` containers: they take on most of the
|
||||||
|
load.
|
||||||
|
|
||||||
|
2. Add more replicas to ``kubedns`` replication controller (we decided to
|
||||||
|
stop on 6 replicas, as it solved the observed issue - for bigger
|
||||||
|
clusters it might be needed to increase this number even more).
|
||||||
|
|
||||||
|
3. Increase number of parallel connections ``dnsmasq`` should handle (we
|
||||||
|
used ``--dns-forward-max=1000`` which is recommendaed parameter setup
|
||||||
|
in ``dnsmasq`` manuals)
|
||||||
|
|
||||||
|
4. Increase size of cache in ``dnsmasq``: it has hard limit of 10000 cache
|
||||||
|
entries which seems to be reasonable amount.
|
||||||
|
|
||||||
|
5. Fix ``kubedns`` to handle this behaviour in proper way.
|
||||||
|
|
||||||
|
Upstream issues (partially fixed)
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
#1 and #2 are fixed by making them configurable in Kargo by Kubernetes
|
||||||
|
team:
|
||||||
|
`issue <https://github.com/kubernetes-incubator/kargo/issues/643>`__,
|
||||||
|
`pull request <https://github.com/kubernetes-incubator/kargo/pull/652>`__.
|
||||||
|
|
||||||
|
Other fixes are still being implemented as of time of this publication.
|
||||||
|
|
||||||
|
Kubernetes scheduler is ineffective with pod antiaffinity
|
||||||
|
---------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
It takes significant amount of time for scheduler to process pods with
|
||||||
|
pod antiaffinity rules specified on them. It is spending about **2-3
|
||||||
|
seconds** on each pod which makes time needed to deploy OpenStack
|
||||||
|
cluster on 900 nodes unexpectedly long (about 3h for just scheduling).
|
||||||
|
Antiaffinity rules are required to be used for OpenStack deployment to
|
||||||
|
prevent several OpenStack compute nodes to be mixed and messed to one
|
||||||
|
Kubernetes Minion node.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
According to profiling results, most of the time is spent on creating
|
||||||
|
new Selectors to match existing pods against them, which triggers
|
||||||
|
validation step. Basically we have O(N^2) unnecessary validation steps
|
||||||
|
(N - number of pods), even if we have just 5 deployments entities
|
||||||
|
covering most of the nodes.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Specific optimization that speeds up scheduling time up to about 300
|
||||||
|
ms/pod was required in this case. It’s still slow in terms of common
|
||||||
|
sense (about 30m spent just on pods scheduling for 900 nodes OpenStack
|
||||||
|
cluster), but is close to be reasonable. This solution lowers number of
|
||||||
|
very expensive operations to O(N), which is better, but still depends on
|
||||||
|
number of pods instead of deployments, so there is space for future
|
||||||
|
improvement.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Optimization merged into master: `pull
|
||||||
|
request <https://github.com/kubernetes/kubernetes/pull/37691>`__;
|
||||||
|
backported to 1.5 branch (will release in 1.5.2 release): `pull
|
||||||
|
request <https://github.com/kubernetes/kubernetes/pull/38693>`__.
|
||||||
|
|
||||||
|
Kubernetes scheduler needs to be deployed on separate node
|
||||||
|
----------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
During huge OpenStack cluster deployment against pre-deployed
|
||||||
|
Kubernetes ``scheduler``, ``controller-manager`` and ``apiserver`` start
|
||||||
|
competing for CPU cycles as all of them get big load. Scheduler is more
|
||||||
|
resource-hungry (see next problem), so we need a way to deploy it
|
||||||
|
separately.
|
||||||
|
|
||||||
|
Root Cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
The same problem with Kubsernetes scheduler efficiency at scale of about
|
||||||
|
1000 nodes as in the issue above.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Kubernetes scheduler was moved to a separate node manually, all other
|
||||||
|
schedulers were manually killed to prevent them from moving to other
|
||||||
|
nodes.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
`Issue <https://github.com/kubernetes-incubator/kargo/issues/834>`__
|
||||||
|
created in Kargo installer Github repository.
|
||||||
|
|
||||||
|
kube-apiserver have low default rate limit
|
||||||
|
------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Different services start receiving “429 Rate Limit Exceeded” HTTP error
|
||||||
|
even though ``kube-apiservers`` can take more load. It is linked to a
|
||||||
|
scheduler bug (see below).
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Raise rate limit for ``kube-apiserver process`` via ``--max-requests-inflight``
|
||||||
|
option. It defaults to 400, in our case it became workable at 2000. This
|
||||||
|
number should be configurable in Kargo deployment tool, as for bigger
|
||||||
|
deployments it might be required to increase it accordingly.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Upstream issue or pull request was not created for this issue.
|
||||||
|
|
||||||
|
Kubernetes scheduler can schedule wrongly
|
||||||
|
-----------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
When many pods are being created (~4500 in our case of OpenStack
|
||||||
|
deployment) and faced with 429 error from ``kube-apiserver`` (see above),
|
||||||
|
the scheduler can schedule several pods of the same deployment on one node
|
||||||
|
in violation of pod antiaffinity rule on them.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
This issue arises due to scheduler cache being evicted before the pod
|
||||||
|
actually processed.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
`Pull
|
||||||
|
request <https://github.com/kubernetes/kubernetes/pull/38503>`__ accepted
|
||||||
|
in Kubernetes upstream.
|
||||||
|
|
||||||
|
Docker become unresponsive at random
|
||||||
|
------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Docker process sometimes hangs on several nodes, which results in
|
||||||
|
timeouts in ``kubelet`` logs and pods cannot be spawned or terminated
|
||||||
|
successfully on the affected minion node. Although bunch of similar
|
||||||
|
issues has been fixed in Docker since 1.11, we still are observing those
|
||||||
|
symptoms.
|
||||||
|
|
||||||
|
Workaround
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Docker daemon logs does not contain any notable information, so we had
|
||||||
|
to restart docker service on the affected node (during those experiments
|
||||||
|
we used Docker 1.12.3, but we have observed similar symptoms in 1.13
|
||||||
|
as well).
|
||||||
|
|
||||||
|
Calico start up time is too long
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
If we have to kill a Kubernetes node, Calico requires ~5 minutes to
|
||||||
|
reestablish all mesh connections.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Calico uses BGP, so without route reflector it has to do full-mesh
|
||||||
|
between all nodes in cluster.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
We need to switch to using route reflectors in our clusters. Then every
|
||||||
|
node needs only to establish connections to all reflectors.
|
||||||
|
|
||||||
|
Upstream Issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
None. For production use, architecutre of Calico network should be
|
||||||
|
adjusted to use route reflectors set up on selected nodes or on
|
||||||
|
switching fabric hardware. This will reduce the number of BGP
|
||||||
|
connections per node and speed up the Calico startup.
|
||||||
|
|
||||||
|
===========================================
|
||||||
|
OpenStack Issues At Scale 200 Compute Nodes
|
||||||
|
===========================================
|
||||||
|
|
||||||
|
Workloads testing approach
|
||||||
|
==========================
|
||||||
|
|
||||||
|
The goal of this test was to investigate OpenStack and Kubernetes
|
||||||
|
behavior under load. This load needs to emulate end-user traffic running
|
||||||
|
on OpenStack servers (guest systems) with different types of
|
||||||
|
applications running against cloud system. We were interested in the
|
||||||
|
following metrics: CPU usage, Disk statistics, Network load, Memory used
|
||||||
|
and IO stats on each controller nodes and chosen set of the compute
|
||||||
|
nodes.
|
||||||
|
|
||||||
|
Testing preparation steps
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
To preare for testing, several steps should be made:
|
||||||
|
|
||||||
|
- [0] Pre-deploy Kubernetes + OpenStack Newton environment (Kargo +
|
||||||
|
fuel-ccp tools)
|
||||||
|
|
||||||
|
- [1] Establish advanced monitoring of the testing environment (on
|
||||||
|
all three layers - system, Kubernetes and OpenStack)
|
||||||
|
|
||||||
|
- [2] Prepare to automatically deploy slices against OpenStack and
|
||||||
|
configure applications running within them
|
||||||
|
|
||||||
|
- [3] Prepare to run workloads against applications running within
|
||||||
|
slices
|
||||||
|
|
||||||
|
[1] was achieved through configuring Prometheus monitoring and alerting
|
||||||
|
tool with Collectd and Telegraf collectors. Separated monitoring
|
||||||
|
document is under preparation to present significant effort spent on
|
||||||
|
this task.
|
||||||
|
|
||||||
|
[2] was automated by using Heat OpenStack orchestration service
|
||||||
|
`templates <https://github.com/ayasakov/hot-ansible-templates>`__.
|
||||||
|
|
||||||
|
[3] was automated mostly through generating HTTP trafic native for the
|
||||||
|
applications under test using Locust.IO tool
|
||||||
|
`templates <https://github.com/ayasakov/locustio-workloads>`__.
|
||||||
|
Cassandra VM workload was automated through Yahoo! Benchmark (ycsb)
|
||||||
|
running on neighbour VM of the same slice.
|
||||||
|
|
||||||
|
Step [1] was tested against 900 nodes Kubernetes cluster described in
|
||||||
|
section above and later against all test environments we had, steps [2] and
|
||||||
|
[3] were verified against small (20 nodes) testing environment in
|
||||||
|
parallel with finalizing step [1] workability. During this verification
|
||||||
|
several issues with recently introduced Heat support to fuel-ccp were
|
||||||
|
observed and fixed. Later all of those steps were assumed to be run
|
||||||
|
against 200 bare metal nodes lab, but bunch of issues was observed
|
||||||
|
during step [2] that blocked finishing the testing. All of those issues
|
||||||
|
(including those found during small environment verification) are listed
|
||||||
|
below.
|
||||||
|
|
||||||
|
Heat/fuel-ccp integration issues
|
||||||
|
================================
|
||||||
|
|
||||||
|
Lack of Heat domain configuration
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Authentication errors during Heat stacks (representing workloads testing
|
||||||
|
slices on each compute node) creation.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
During OpenStack Newton timeframe Orchestration has started to require
|
||||||
|
additional information in the Identity service to manage stacks - in
|
||||||
|
particular, configuration of heat domain that would contain stacks
|
||||||
|
projects and users, creating of ``heat_domain_admin`` user to manage
|
||||||
|
projects and users in the heat domain and adding the admin role to the
|
||||||
|
``heat_domain_admin`` user in the heat domain to enable administrative
|
||||||
|
stack management privileges by the ``heat_domain_admin`` user. Please take
|
||||||
|
a look on `OpenStack Newton configuration
|
||||||
|
guide <http://docs.openstack.org/project-install-guide/orchestration/newton/install-ubuntu.html>`__
|
||||||
|
for more information.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Set up needed configuration steps in ``fuel-ccp``:
|
||||||
|
|
||||||
|
- `Patch #1 <https://review.openstack.org/#/c/400846/>`__
|
||||||
|
|
||||||
|
- `Patch #2 <https://review.openstack.org/#/c/402045/>`__
|
||||||
|
|
||||||
|
Lack of heat-api-cfn service configuration in fuel-ccp
|
||||||
|
------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Applications configured in Heat templates are not receiving their
|
||||||
|
configurations and data required for the succeeded applications
|
||||||
|
workability.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Initial Heat support in ``fuel-ccp`` did not include ``heat-api-cfn``
|
||||||
|
service, which is used by Heat for some config transports. This service is
|
||||||
|
necessary to support default ``user_data_format`` (``HEAT_CFNTOOLS``), used
|
||||||
|
for most applications-related Heat templates.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Add new ``heat-api-cfn`` image, which will be used to create new service and
|
||||||
|
configure all necessary endpoints for it in ``fuel-ccp`` tool. Patches to
|
||||||
|
``fuel-ccp``:
|
||||||
|
|
||||||
|
- `Patch #1 <https://review.openstack.org/#/c/401138/>`__
|
||||||
|
|
||||||
|
- `Patch #2 <https://review.openstack.org/#/c/401174/>`__
|
||||||
|
|
||||||
|
Heat endpoint not reachable from virtual machines
|
||||||
|
-------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Heat API and Heat API CFN are not reachable from OpenStack VM, even
|
||||||
|
though some of the applications configurations require such access.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Fuel CCP deploys Heat services with default configuration and changes
|
||||||
|
``endpoint_type`` from ``publicURL`` to ``internalURL``. However,
|
||||||
|
such configuration in Kubernetes cluster is not enough for several types
|
||||||
|
of Heat stack resources like ``OS::Heat::Waitcondition`` and
|
||||||
|
``OS::Heat::SoftwareDeployment``, which require callbacks to Heat API
|
||||||
|
or Heat API CFN. Due to Kubernetes architecture, it's not possible to
|
||||||
|
do such callback on the default port value (for ``heat-api`` it's - 8004
|
||||||
|
and 8000 for ``heat-api-cfn``).
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
There are two ways to fix described above issues:
|
||||||
|
|
||||||
|
- Out of the box, which requires just adding some data to .ccp.yaml
|
||||||
|
configuration file. This workaround can be used prior OpenStack
|
||||||
|
cluster deployment during future OpenStack cluster description.
|
||||||
|
|
||||||
|
- Second which requires some manual actions and can be processed when
|
||||||
|
Openstack is already deployed and cloud administrator can change
|
||||||
|
only one component configuration.
|
||||||
|
|
||||||
|
Both of those solutions are described in the `patch to Fuel CCP
|
||||||
|
Documentation <https://review.openstack.org/#/c/404114>`__. Please
|
||||||
|
notice that additionally `patch to Fuel CCP
|
||||||
|
codebase <https://review.openstack.org/#/c/405263/>`__ need to be
|
||||||
|
applied to make some of the Kubernetes options configurable.
|
||||||
|
|
||||||
|
Glance+Heat authentication misconfiguration
|
||||||
|
-------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
During Heat stacks (representing workloads testing slices) creation,
|
||||||
|
random Glance authentication errors were observed.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Service-specific users should have admin role in "service" project and
|
||||||
|
should not belong to user-facing admin project. Initially Fuel-CCP
|
||||||
|
contained Glance and Heat services misconfigured and several patches
|
||||||
|
were required to fix it.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
- `Patch #1 <https://review.openstack.org/#/c/409033/>`__
|
||||||
|
|
||||||
|
- `Patch #2 <https://review.openstack.org/#/c/409037/>`__
|
||||||
|
|
||||||
|
Workloads Testing Issues
|
||||||
|
========================
|
||||||
|
|
||||||
|
Random loss of connection to MySQL
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
During Heat stacks (representing slices) creation some of the stacks are
|
||||||
|
moved to ERROR state with the following traceback being found in the
|
||||||
|
Heat logs::
|
||||||
|
|
||||||
|
2016-12-11 16:59:22.220 1165 ERROR nova.api.openstack.extensions
|
||||||
|
DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't
|
||||||
|
connect to MySQL server on 'galera.ccp.svc.cluster.local' ([Errno
|
||||||
|
-2] Name or service not known)")
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
This turned out to be exactly the same issue with KubeDNS being unable
|
||||||
|
to handle high loads as described in Kubernetes Issues section above.
|
||||||
|
|
||||||
|
First of all, ``kubedns`` seems to fail often on high load (or even without
|
||||||
|
load), during the experiment we observed continuous kubedns container
|
||||||
|
restarts even on empty (but big enough) Kubernetes cluster. Restarts
|
||||||
|
are caused by liveness check failing, although nothing notable is
|
||||||
|
observed in any logs.
|
||||||
|
|
||||||
|
Second, ``dnsmasq`` should have taken load off ``kubedns``, but it needs some
|
||||||
|
tuning to behave as expected for big load, otherwise it is useless.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
See above in Kubernetes Issues section.
|
||||||
|
|
||||||
|
Slow Heat API requests (`Bug 1653088 <https://bugs.launchpad.net/fuel-ccp/+bug/1653088>`__)
|
||||||
|
-----------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Requests to the Heat API for regular requests took more than a minute of
|
||||||
|
time. Example of such request can be presented with `this
|
||||||
|
traceback <http://paste.openstack.org/show/593132/>`__ showing time
|
||||||
|
needed for listing details of the specific stack
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Fuel-ccp team proposed that it might be a hidden race condition between
|
||||||
|
multiple Heat workers, and it's not yet clear where is this race
|
||||||
|
happening. This is still under debug
|
||||||
|
|
||||||
|
Workaround
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Set up number of Heat workers to 1 until real cause of this issue will
|
||||||
|
be found.
|
||||||
|
|
||||||
|
OpenStack VMs cannot fetch required metadata
|
||||||
|
--------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
OpenStack servers do not receive metadata with applications-specific
|
||||||
|
information. There is a following
|
||||||
|
`trace <http://paste.openstack.org/show/593337/>`__ in Heat logs.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Prior pushing metadata to OpenStack VMs, Heat is storing information
|
||||||
|
about stack under creation to its own database. Starting with OpenStack
|
||||||
|
Newton Heat works in so-called “convergence” mode by default, making
|
||||||
|
sure that Heat engine process several requests at one time. This
|
||||||
|
parallel task processing might end up with race conditions during DB
|
||||||
|
access.
|
||||||
|
|
||||||
|
Workaround
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Turn off Heat convergence engine::
|
||||||
|
|
||||||
|
convergence_engine=False
|
||||||
|
|
||||||
|
RPC timeouts during Heat resources validation
|
||||||
|
---------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
During Heat resources validation, this process is failed with the
|
||||||
|
following `trace <http://paste.openstack.org/show/593112/>`__ being
|
||||||
|
caught.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Initial assumption was that there might be too small
|
||||||
|
``rpc_response_timeout`` parameter being set up in Heat configuration
|
||||||
|
file. This parameter was set up to 10 minutes exactly like that was used
|
||||||
|
in MOS::
|
||||||
|
|
||||||
|
rpc_response_timeout = 600
|
||||||
|
|
||||||
|
After that was done, no more RPC timeouts were observed.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Set up ``rpc_response_timeout = 600`` as that’s tested value for
|
||||||
|
generations of MOS releases.
|
||||||
|
|
||||||
|
Overloaded Memcache service (`Bug 1653089 <https://bugs.launchpad.net/fuel-ccp/+bug/1653089>`__)
|
||||||
|
----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
On 200 nodes with workloads deploying OpenStack is actively using cache
|
||||||
|
(``memcached``). At some point of time requests to ``memcached`` begin to be
|
||||||
|
processed really slow.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Memcache size is 256 MB by default in Fuel CCP, which is really small
|
||||||
|
size. That was a reason for great amount of retransmissions being
|
||||||
|
processed.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Increase cache size up to 30G in ``fuel-ccp`` configuration file for huge or
|
||||||
|
loaded deployments::
|
||||||
|
|
||||||
|
configs:
|
||||||
|
memcached:
|
||||||
|
address: 0.0.0.0
|
||||||
|
port:
|
||||||
|
cont: 11211
|
||||||
|
ram: 30720
|
||||||
|
|
||||||
|
Incorrect status information for Neutron agents
|
||||||
|
-----------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
While asking for Neutron agents status through OpenStack client, all of
|
||||||
|
them are displayed in down state, although due to the environment
|
||||||
|
behaviour it does not seem to be true.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
The root cause of this issue is hidden in OpenStackSDK refactoring that
|
||||||
|
caused various OSC networking commands to fail. The full discussion
|
||||||
|
regarding this issue can be found in `upstream
|
||||||
|
bug <https://bugs.launchpad.net/python-openstackclient/+bug/1652317>`__.
|
||||||
|
|
||||||
|
Workaround
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Use Neutron client directly to gather Neutron services statuses until
|
||||||
|
`bug <https://bugs.launchpad.net/python-openstackclient/+bug/1652317>`__
|
||||||
|
will be fixed.
|
||||||
|
|
||||||
|
Nova client not working (`Bug 1653075 <https://bugs.launchpad.net/fuel-ccp/+bug/1653075>`__)
|
||||||
|
------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
All commands running through Nova client end up with the `stack
|
||||||
|
trace <http://paste.openstack.org/show/593531/>`__.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
In debug mode it’s seen that client tries to use HTTP protocol for the
|
||||||
|
Nova requests, e.g. http://compute.ccp.external:8443/v2.1/, although
|
||||||
|
HTTPS protocol is required for this client-server conversation.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Add the following lines to
|
||||||
|
``ccp-repos/fuel-ccp-nova/service/files/nova.conf.j2`` file::
|
||||||
|
|
||||||
|
[DEFAULT]
|
||||||
|
|
||||||
|
secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO
|
||||||
|
|
||||||
|
Neutron server timeouts (`Bug 1653077 <https://bugs.launchpad.net/fuel-ccp/+bug/1653077>`__)
|
||||||
|
------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Neutron server is not able to process requests to it with the following
|
||||||
|
error being caught in the Neutron logs:
|
||||||
|
`trace <http://paste.openstack.org/show/593483/>`__
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Default values (that are used in ``fuel-ccp``) for the Neutron database pool
|
||||||
|
size, as well as max overflow parameter are not enough for the
|
||||||
|
environment with big enough load on Neutron API.
|
||||||
|
|
||||||
|
Solution
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Modify ``ccp-repos/fuel-ccp-neutron/service/files/neutron.conf.j2`` file to
|
||||||
|
contain the following configuration parameters::
|
||||||
|
|
||||||
|
[database]
|
||||||
|
|
||||||
|
max_pool_size = 30
|
||||||
|
|
||||||
|
max_overflow = 60
|
||||||
|
|
||||||
|
No access to OpenStack VM from tenant network
|
||||||
|
---------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Some of the VMs representing the slices are not reachable via tenant
|
||||||
|
network. For example::
|
||||||
|
|
||||||
|
| 93b95c73-f849-4ffb-9108-63cf262d3a9f | cassandra_vm_0 |
|
||||||
|
ACTIVE | slice0-node162-net=11.62.0.8, 10.144.1.35 |
|
||||||
|
ubuntu-software-config-last |
|
||||||
|
|
||||||
|
root@node1:~# ssh -i .ssh/slace ubuntu@10.144.1.35
|
||||||
|
Connection closed by 10.144.1.35 port 22
|
||||||
|
|
||||||
|
It is unreachable from tenant network as well. For example from instance
|
||||||
|
``b1946719-b401-447d-8103-cc43b03b1481`` which has been spawned by the same
|
||||||
|
Heat stack on the same compute node (``node162``):
|
||||||
|
`http://paste.openstack.org/show/593486/ <http://paste.openstack.org/show/593486/>`__
|
||||||
|
|
||||||
|
Root cause and solution
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Still under investigation. Root cause not clear yet. **This issue is blocking
|
||||||
|
running workloads against deployed slices.**
|
||||||
|
|
||||||
|
OpenStack services don’t handle PXC pseudo-deadlocks
|
||||||
|
----------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
When run in parallel, create operations of lots of resources were
|
||||||
|
failing with DBError saying that Percona Xtradb Cluster identified a
|
||||||
|
deadlock and transaction should be restarted.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
oslo.db is responsible for wrapping errors received from DB into proper
|
||||||
|
classes so that services can restart transactions if similar errors
|
||||||
|
occur, but it didn’t expect error in format that is being sent by
|
||||||
|
Percona. After we fixed this, we still experienced similar errors
|
||||||
|
because not all transactions that could be restarted were properly
|
||||||
|
decorated in Nova code.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
`Bug <https://bugs.launchpad.net/oslo.db/+bug/1648818>`__ has been
|
||||||
|
fixed by Roman Podolyaka’s
|
||||||
|
`CR <https://review.openstack.org/409194>`__ and
|
||||||
|
`backported <https://review.openstack.org/409679>`__ to Newton. It
|
||||||
|
fixes Percona deadlock error detection, but there’s at least one place
|
||||||
|
in Nova to be fixed (TBD)
|
||||||
|
|
||||||
|
Live migration failed with live_migration_uri configuration
|
||||||
|
-------------------------------------------------------------
|
||||||
|
|
||||||
|
Symptoms
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
With ``live_migration_uri`` configuration, live migrations fails because
|
||||||
|
one compute host can’t connect to a libvirt on another host.
|
||||||
|
|
||||||
|
Root cause
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
We can’t specify which IP address to use in template in
|
||||||
|
``live_migration_uri``, so it was trying to use address from first
|
||||||
|
interface which happened to be in PXE network while libvirt listens in
|
||||||
|
private network. We couldn’t use ``live_migration_inbound_addr`` which
|
||||||
|
would solve this problem because of a problem in upstream Nova.
|
||||||
|
|
||||||
|
Upstream issues
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
A `bug <https://bugs.launchpad.net/nova/+bug/1638625>`__ in Nova has
|
||||||
|
been `fixed <https://review.openstack.org/398956>`__ and
|
||||||
|
`backported <https://review.openstack.org/404810>`__ to Newton. We
|
||||||
|
`switched <https://review.openstack.org/407708>`__ to using
|
||||||
|
``live_migration_inbound_addr`` after that.
|
||||||
|
|
||||||
|
Contributors
|
||||||
|
============
|
||||||
|
|
||||||
|
The following people have credits for contributing to this
|
||||||
|
document:
|
||||||
|
|
||||||
|
* Dina Belova <dbelova@mirantis.com>
|
||||||
|
|
||||||
|
* Yuriy Taraday <ytaraday@mirantis.com>
|
||||||
Loading…
Reference in New Issue
Block a user