Add testplans for Marathon and Mesos
Change-Id: I18d47fab3e976dd7e5ea06fc01a702f7e1c96108
This commit is contained in:
Sergey Galkin
parent
7ed7769b70
commit
439f56579c
797
doc/source/test_plans/container_cluster_systems/plan.rst
Normal file
797
doc/source/test_plans/container_cluster_systems/plan.rst
Normal file
@ -0,0 +1,797 @@
|
||||
|
||||
.. _Measuring_performance_of_container_cluster_systems:
|
||||
|
||||
**************************************************
|
||||
Measuring performance of container cluster systems
|
||||
**************************************************
|
||||
|
||||
:status: **draft**
|
||||
:version: 0
|
||||
|
||||
:Abstract:
|
||||
|
||||
A complete microservices architecture has the ability to manage a large
|
||||
number of small applications which together form a set of services.
|
||||
Container technology provides a way to package and run applications
|
||||
independent of the underlying operating system.
|
||||
Microservice and container technologies are very complimentary and
|
||||
usually used in conjunction.
|
||||
There are several container management systems, such as Apache Mesos,
|
||||
Kubernetes, and Docker Swarm,
|
||||
which provide various ways to manage groups of containers.
|
||||
|
||||
:Conventions:
|
||||
|
||||
|
||||
|
||||
Test Plan
|
||||
=========
|
||||
Test Environment
|
||||
----------------
|
||||
Preparation
|
||||
^^^^^^^^^^^
|
||||
Typical container frameworks consist of several components:
|
||||
* Configuration and Synchronization (Zookeeper)
|
||||
* Scheduler (Mesos)
|
||||
* Application manager (Marathon)
|
||||
|
||||
Environment description
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
In this document only performance and scaling aspects of the different
|
||||
frameworks will be covered.
|
||||
It is expected that existing solutions will be used and that they have had
|
||||
sufficient functional testing.
|
||||
Performance aspects should cover elapsed times for operations used in the
|
||||
framework.
|
||||
|
||||
Test Case #1: A new application creation time
|
||||
---------------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the create
|
||||
application operation.
|
||||
This metric includes the time of application submission processing,
|
||||
time of scheduling,
|
||||
and time of actual all containers spawning on the slaves hosts.
|
||||
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
Test Case #2: A application CPU update time
|
||||
-------------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the update CPU
|
||||
application operation. This metric includes the time of application submission
|
||||
processing, time of scheduling, and time of actual all containers updated on
|
||||
the slaves hosts.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Update applications CPU up to 2.
|
||||
4.
|
||||
Wait util all application go to state Running
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
|
||||
Test Case #3: A application memory update time
|
||||
----------------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the update memory
|
||||
application operation. This metric includes the time of application submission
|
||||
processing, time of scheduling, and time of actual all containers updated on
|
||||
the slaves hosts.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Update applications memory up to 512M.
|
||||
4.
|
||||
Wait util all application go to state Running
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
|
||||
Test Case #4: A application disk update time
|
||||
--------------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the update disk
|
||||
application operation. This metric includes the time of application submission
|
||||
processing, time of scheduling,
|
||||
and time of actual all containers updated on the slaves hosts.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Update applications disk up to 100G.
|
||||
4.
|
||||
Wait util all application go to state Running
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
|
||||
Test Case #5: A application instances update time
|
||||
-------------------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the update number of
|
||||
instances application operation.
|
||||
This metric includes the time of application submission processing, time of
|
||||
scheduling, and time of actual all containers updated on the slaves hosts.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Update instances up to twice in the each application.
|
||||
4.
|
||||
Wait util all application go to state Running
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
Test Case #6: A application restart time
|
||||
----------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the restart
|
||||
application operation. This metric includes the time of application submission
|
||||
processing, time of scheduling, and time of actual all containers restarted on
|
||||
the slaves hosts.
|
||||
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Restart all applications.
|
||||
4.
|
||||
Wait util all application go to state Running
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
Test Case #7: A application delete time
|
||||
---------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at measuring the total elapsed time of the delete
|
||||
application operation. This metric includes the time of application submission
|
||||
processing, time of scheduling, and time of actual all containers deleted on
|
||||
the slaves hosts.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+==============================+========================================+
|
||||
| APPLICATION_OPERATION_MAX |Maximum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MIN |Minimum execution time of one operation |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEAN |Mean execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
| APPLICATION_OPERATION_MEDIAN |Median execution time of all operations |
|
||||
+------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| NODES_NUMBER_PER_APPLICATION |50, 100, 500 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| CONCURRENCY |1, 2, 4, 8, 16 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring values
|
||||
^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Start to create simultaneously the number of applications equal CONCURRENCY
|
||||
with number of instances equal NODES_NUMBER_PER_APPLICATION in the each
|
||||
application through Marathon API. Every instances is docker container with
|
||||
nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
|
||||
2.
|
||||
Wait util all application go to state Running
|
||||
3.
|
||||
Delete all applications.
|
||||
4.
|
||||
Wait util list of application return empty list.
|
||||
5.
|
||||
As a result of the previous step you should be able to provide the amount of
|
||||
graphs and tables with the dependencies on an concurrency and instances
|
||||
number of a operation time. One table for all results and graphs for every
|
||||
concurrency and every nodes number. On this step you need to
|
||||
calculate minimum, maxima, average and median of operation time. You need
|
||||
to fill the following table with calculated values:
|
||||
|
||||
.. table:: results
|
||||
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
| CONCURRENCY | NODES_NUMBER_PER_APPLICATION | APPLICATION_OPERATION (sec) |
|
||||
| | +--------+--------+---------+--------+
|
||||
| | |minimum |maximum | average | median |
|
||||
+=============+==============================+========+========+=========+========+
|
||||
|1 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|1 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|2 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|4 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|8 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 50 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 100 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|16 | 500 | | | | |
|
||||
+-------------+------------------------------+--------+--------+---------+--------+
|
||||
|
||||
|
||||
Test Case #8: Health check Performance
|
||||
--------------------------------------
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
This test is aimed at validating the response time of a health check when many
|
||||
health checks are configured.
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. table:: list of test metrics to be collected during this test
|
||||
|
||||
+---------------------------------------------------+----------------------------------------+
|
||||
| Parameter | Description |
|
||||
+===================================================+========================================+
|
||||
| HEALTH_CHECK_INTERVAL_DEVIATION(CONTAINERS_COUNT) | | Difference between real health check |
|
||||
| | | interval and configured value |
|
||||
+---------------------------------------------------+----------------------------------------+
|
||||
|
||||
.. table:: list of test metrics to be persistent during this test:
|
||||
|
||||
+---------------------------------+--------------------------------+
|
||||
| Parameter | Value |
|
||||
+=================================+================================+
|
||||
| HEALTH_CHECK_TEST_DURATION | 20 |
|
||||
+---------------------------------+--------------------------------+
|
||||
| HEALTH_CHECK_INTERVAL | 30 |
|
||||
+---------------------------------+--------------------------------+
|
||||
|
||||
Measuring REAL_INTERVAL(TASK_COUNT) values
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
1.
|
||||
Deploy Marathon cluster from scratch.
|
||||
2.
|
||||
Run CONTAINERS_COUNT containers in container cluster system. Setup health
|
||||
check for those containers in container cluster system with health check
|
||||
interval is set to HEALTH_CHECK_INTERVAL seconds.
|
||||
3.
|
||||
Gather HEALTH_CHECK_INTERVAL_DEVIATION from containers during
|
||||
HEALTH_CHECK_TEST_DURATION minutes.
|
||||
4.
|
||||
You need to perform step 2 and step 3 per each CONTAINERS_COUNT value from
|
||||
the following list:
|
||||
|
||||
* CONTAINERS_COUNT=2500
|
||||
* CONTAINERS_COUNT=5000
|
||||
* CONTAINERS_COUNT=7500
|
||||
* CONTAINERS_COUNT=10000
|
||||
* CONTAINERS_COUNT=12500
|
||||
* CONTAINERS_COUNT=15000
|
||||
* CONTAINERS_COUNT=17500
|
||||
* CONTAINERS_COUNT=20000
|
||||
|
||||
3.
|
||||
As a result of the previous step you should be able to provide table with
|
||||
the dependences on an health check interval deviation of a containers
|
||||
count. You need to calculate minimuml, maximal, average and 95% percental
|
||||
of HEALTH_CHECK_INTERVAL_DEVIATION parameter per each CONTAINERS_COUNT
|
||||
value. You need to fill the following table with calculated values:
|
||||
|
||||
.. table:: PUSH_TIME(CONCURRENCY)
|
||||
|
||||
+------------------+---------+---------+---------+-----+
|
||||
| CONTAINERS_COUNT | HEALTH_CHECK_INTERVAL_DEVIATION |
|
||||
| +---------+---------+---------+-----+
|
||||
| | minimum | maximal | average | 95% |
|
||||
+==================+=========+=========+=========+=====+
|
||||
| | | | | |
|
||||
+------------------+---------+---------+---------+-----+
|
||||
|
||||
Applications
|
||||
============
|
||||
list of container platforms
|
||||
---------------------------
|
||||
|
||||
+-----------------------------+---------+
|
||||
| Name of container platform | Version |
|
||||
+=============================+=========+
|
||||
| Apache Mesos | |
|
||||
+-----------------------------+---------+
|
||||
| Marathon | |
|
||||
+-----------------------------+---------+
|
||||
| Docker | |
|
||||
+-----------------------------+---------+
|
||||
@ -16,3 +16,5 @@ Test Plans
|
||||
tenant_networking/plan
|
||||
container_repositories/plan
|
||||
keystone/plan
|
||||
container_cluster_systems/plan
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user