274 lines
11 KiB
ReStructuredText
274 lines
11 KiB
ReStructuredText
|
|
Methodology for testing Hyper-Scale
|
|
===================================
|
|
|
|
What is meant by Hyper-Scale?
|
|
-----------------------------
|
|
|
|
For this section, hyper-scale is defined as being able to achieve 100K of
|
|
something in an OpenStack cloud. Initially this may be VMs, but could be
|
|
hypervisors, projects, etc. Because of the open-endnesses of this definition,
|
|
it is necessary to identify all places where operations show linear (or
|
|
even worse...) execution time as the number of items that the operation deals
|
|
with scales up. For example, if execution time of an operation is order of
|
|
10 microseconds per item that operation is dealing with, that means that at
|
|
100K, a second has been added to agent execution time.
|
|
|
|
How is linearity defined in this context?
|
|
-----------------------------------------
|
|
|
|
To determine linearity, one measures the execution time of each operation
|
|
over a sample the has increasing scale and then performance a least squares
|
|
fit [#]_ to the raw data. Such a fit looks to map a straight line (y=mx+b) to
|
|
the sample data, where
|
|
|
|
* x == the number of items being scaled,
|
|
* m == the slope of the fit,
|
|
* b == the vertical intercept, and
|
|
* y == the operation's execution time
|
|
|
|
In addition to finding m and b, the fit will also generate a correlation
|
|
coefficient (r) which provides a measurement of how random the sample data was.
|
|
As an initial filter, fits with a correlation coefficient above a certain
|
|
threshold can be considered as candidates for improvement. Once an operation
|
|
is considered a candidate for improvement, then the execution time of all of
|
|
its sub-operations are measured and a least squares fit performed on them.
|
|
When looking at sub-operations, of interest are the slope and the ratio of
|
|
the vertical intercept to the slope (the "doubling scale" of the operation).
|
|
These together determine the "long pole in the test" (the sub-operation that
|
|
should be analyzed first), with the slope determining the order and the
|
|
"doubling scale" acting as a tie breaker.
|
|
|
|
How Should Code be Instrumented When Looking for Hyper-Scale?
|
|
-------------------------------------------------------------
|
|
|
|
Today, instrumentation is a very manual process: after running devstack,
|
|
debug LOG statements are added into the code base to indicate the entrance
|
|
and exit of each routine that should be timed. These debug log statements
|
|
can be tagged with strings like "instrument:" or "instrument2:" to allow
|
|
multiple levels of instrumentation to be applied during a single test run.
|
|
Then restart the agent that is being instrumented via the screen that is
|
|
created by devstack.
|
|
|
|
Note: as this ends up to be an iterative process as one looks to isolate
|
|
the code that shows linear execution time, for subsequent restacks, it
|
|
usually makes sense to not reclone (unless you want to repeat the by hand
|
|
editing afterwards).
|
|
|
|
It is hoped that the osprofiler sub-project of oslo [#]_ will be extended to
|
|
allow for method decoration to replace the use of by hand editing.
|
|
|
|
Setup Considerations
|
|
--------------------
|
|
|
|
There are two ways to go about looking for potential hyper-scale issues.
|
|
The first is to use a brute force method where one actually builds a
|
|
hyper-scale cloud and tests. Another route is to build a cloud using the
|
|
slowest, smallest, simplest components one has on hand. The idea behind this
|
|
approach is that by slowing down the hardware, the slope and intercept of
|
|
the least squares fit are both scaled up, making linearity issues easier to
|
|
find with lower numbers of instances. However, this makes comparing raw values
|
|
from the least squares fits between different setups difficult - one can
|
|
never be 100% sure that the comparison is "apples-to-apples" and that
|
|
hardware effects are completely isolated. The choice above of using the
|
|
correlation coefficient and "doubling scale" above are intended to use values
|
|
that are more independent of the test cloud design.
|
|
|
|
An Example: Neutron L3 Scheduling
|
|
---------------------------------
|
|
|
|
The rest of this section presents an example of this methodology in action
|
|
(In fact, this example was what led to the methodology being developed).
|
|
This example uses a four node cloud, configured as a controller, a network
|
|
node, and two compute nodes.
|
|
|
|
To determine what parts of L3 scheduling needed scaling improvements the
|
|
following experiment was used. This test can certainly be automated via
|
|
Rally and should be (once instrumentation no longer requires manual decoration
|
|
and/or code manipulation):
|
|
|
|
#. Create the external network and subnet with:
|
|
|
|
.. code-block:: bash
|
|
|
|
neutron net-create test-external --router:external --provider:physical_network default --provider:network_type flat
|
|
neutron subnet-create --name test-external-subnet --disable-dhcp test-external 172.18.128.0/20
|
|
|
|
#. Create X number of projects
|
|
|
|
#. For each project, perform the following steps (in fhe following,
|
|
$i is the tenant index and $id is the tenant UUID, $net is the UUID of
|
|
the tenant network created in the first step below, and $port is the
|
|
interface port UUID reported by nova):
|
|
|
|
.. code-block:: bash
|
|
|
|
neutron net-create network-$i --tenant-id $id
|
|
neutron subnet-create network-$i 192.168.18.0/24 --name subnet-$i --tenant-id $id
|
|
neutron router-create router-$i --tenant-id $id
|
|
neutron router-interface-add router-$i subnet-$i
|
|
neutron router-gateway-set router-$i test-external
|
|
keystone user-role-add --user=admin --tenant=tenant-$i --role=admin
|
|
nova --os-tenant-name tenant-$i boot --flavor m1.nano --image cirros-0.3.4-x86_64-uec --nic net-id=$net instance-tenant-$i
|
|
(once the instance's interface port is shown as active by nova):
|
|
neutron floatingip-create --tenant-id $id --port-id $port test-external
|
|
|
|
#. For each project, perform the following steps:
|
|
|
|
.. code-block:: bash
|
|
|
|
neutron floatingip-delete <floatingip-uuid>
|
|
nova --os-tenant-name tenant-$i stop instance-tenant-$i
|
|
neutron router-gateway-clear <router-uuid> test-external
|
|
neutron router-interface-delete router-$i subnet-$i
|
|
neutron router-delete router-$i
|
|
neutron subnet-delete subnet-$i
|
|
neutron net-delete network-$i
|
|
|
|
This provides testing of both the paths that create and delete Neutron routers
|
|
and the attributes that make them usuable.
|
|
|
|
Extracting Execution Time
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Once one has instrumented logs (or instrumentation records), extracting
|
|
execution time becomes a matter of filtering via one's favorite language.
|
|
One approach is to filter to a comma separate value (csv) file and then
|
|
import that into a spreadsheet program that has the least squares function
|
|
built in to find the fit and correlation coefficient.
|
|
|
|
Legacy Results Using Neutron Master as of 11/6/15 - Adding Routers/FIPs
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
+-----------------+------------+---------------+-----------------+
|
|
| Function | Corelation | Slope | Doubling |
|
|
| | Coeficient | (msec/router) | Scale (routers) |
|
|
+=================+============+===============+=================+
|
|
| Interface Add | 0.302 | 1.56 | 1211 |
|
|
+-----------------+------------+---------------+-----------------+
|
|
| Set Gateway | 0.260 | | |
|
|
+-----------------+------------+---------------+-----------------+
|
|
| Add FIP | 2.46E-05 | | |
|
|
+-----------------+------------+---------------+-----------------+
|
|
|
|
Thus, the interface add operation is a candidate for further evaluation.
|
|
Looking at the largest contributers to the slope, we find the following
|
|
methods at successively deeper levels of code:
|
|
|
|
* _process_router_if_compatible (1.34 msec/router)
|
|
|
|
* _process_added_router (1.31 msec/router)
|
|
|
|
* _router_added (0.226 msec/router)
|
|
|
|
* _process (0.998 msec/router)
|
|
|
|
* process_internal_ports (0.888 msec/router)
|
|
|
|
* ovs add-port (0.505 msec/router)
|
|
|
|
This last time is outside of Neutron: it needs to be addressed from
|
|
within the OVS code itself (this is being tracked by [#]_)
|
|
|
|
Legacy Results Using Neutron Master as of 11/6/15 - Removing Routers/FIPs
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
+------------------+------------+---------------+-----------------+
|
|
| Function | Corelation | Slope | Doubling |
|
|
| | Coeficient | (msec/router) | Scale (routers) |
|
|
+==================+============+===============+=================+
|
|
| Clear FIP | 0.389 | 1.503 | 401.5 |
|
|
+------------------+------------+---------------+-----------------+
|
|
| Clear Gateway | 0.535 | 1.733 | 289.3 |
|
|
+------------------+------------+---------------+-----------------+
|
|
| Remove Interface | 0.390 | 1.425 | 302.6 |
|
|
+------------------+------------+---------------+-----------------+
|
|
| Remove Router | 0.526 | 1.5 | 252.8 |
|
|
+------------------+------------+---------------+-----------------+
|
|
|
|
Each of these operations is a candidate for improvement. Looking more
|
|
deeply at each operation, we find the following contributors to the slope:
|
|
|
|
* Clear FIP
|
|
|
|
* get_routers (9.667 usec/router)
|
|
|
|
* _process_routers_if_compatible (1.496 msec/router)
|
|
|
|
* _process_updated_router (1.455 msec/router)
|
|
|
|
* process (1.454 msec/router)
|
|
|
|
* process_internal_ports (151 usec/router)
|
|
|
|
* process_external (1.304 msec/router)
|
|
|
|
* iptables apply time: slope (0.513 msec/router)
|
|
|
|
* process_external_gateway (0.207 msec/router)
|
|
|
|
* FIP cleanup (0.551 msec/router)
|
|
|
|
* Clear Gateway
|
|
|
|
* get_routers (25.8 usec/router)
|
|
|
|
* _process_routers_if_compatible (1.714 msec/router)
|
|
|
|
* _process_update_router (1.700 msec/router)
|
|
|
|
* process (1.700 msec/router)
|
|
|
|
* process_internal_ports (92 usec/router)
|
|
|
|
* process_external (0.99 msec/router)
|
|
|
|
* process_external_gateway (0.99 msec/router)
|
|
|
|
* unplug (0.857 msec/router)
|
|
|
|
* iptables apply time: slope (0.614 msec/router)
|
|
|
|
* Remove Interface
|
|
|
|
* _process_routers_if_compatible (1.398 msec/router)
|
|
|
|
* _process_update_router (1.372 msec/router)
|
|
|
|
* process (1.372 msec/router)
|
|
|
|
* process_internal_ports (0.685 sec/router)
|
|
|
|
* old_ports_loop (0.586 msec/router)
|
|
|
|
* iptables apply time: slope (0.538 msec/router)
|
|
|
|
* Remove Router
|
|
|
|
* _safe_router_removed (1.5 msec/router)
|
|
|
|
* before_delete_callback (0.482 msec/router)
|
|
|
|
* delete (1.055 msec/router)
|
|
|
|
* process (0.865 msec/router)
|
|
|
|
* process_internal_ports (0.168 msec/router)
|
|
|
|
* process_external (0.200 msec/router)
|
|
|
|
* iptables apply time: slope (0.496 msec/router)
|
|
|
|
* namespace delete (0.195 msec/router)
|
|
|
|
As can be seen, improving execution time for iptables will impact a lot of
|
|
operations (and this is a known issue). In addition there are several other
|
|
functions to be dug into further to see what improvements can be made.
|
|
|
|
References
|
|
----------
|
|
|
|
.. [#] http://mathworld.wolfram.com/LeastSquaresFitting.html
|
|
.. [#] https://review.openstack.org/#/c/103825/
|
|
.. [#] https://bugs.launchpad.net/neutron/+bug/1494959
|