diff --git a/doc/source/test_plans/container_cluster_systems/API_latency.rst b/doc/source/test_plans/container_cluster_systems/API_latency.rst index d13d683..5cfe131 100644 --- a/doc/source/test_plans/container_cluster_systems/API_latency.rst +++ b/doc/source/test_plans/container_cluster_systems/API_latency.rst @@ -148,6 +148,33 @@ Measuring values of API latencies of requests which make actions on containers | POST | | +---------------------------------+-----------------------------------------+ +Test Case #1.5: Container creation startup latency extended measurement +----------------------------------------------------------------------- + +Description +^^^^^^^^^^^ + +The is a specific interest in understanding not only when container cluster +system is reporting that the requested container is up and running, but when +this container really becomes operable in data plane terms. This can be +measured via scheduling and spawning containers, that will report about their +status to some centralized manager. This data needs to be collected and then +carefully analyzed, especially keeping attention to possible regressions (if +huge number of containers is created - either at once or one by one). + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. table:: list of test metrics to be collected during this test + + +-----------------------------+---------------------------------------------+ + | Parameter |Description | + +=============================+=============================================+ + | CONTAINER_STARTUP_LATENCY | | The time which a container management | + | | | system needs to spawn workable and fully | + | | | operable container. | + +-----------------------------+---------------------------------------------+ + Test Case #2: API latencies of any other supported requests ----------------------------------------------------------- Description diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/15000_containers.png b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/15000_containers.png new file mode 100644 index 0000000..abdb382 Binary files /dev/null and b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/15000_containers.png differ diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_first_run.png b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_first_run.png new file mode 100644 index 0000000..7b8913c Binary files /dev/null and b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_first_run.png differ diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_second_run.png b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_second_run.png new file mode 100644 index 0000000..5e60d17 Binary files /dev/null and b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/MMM-tool/500_containers_second_run.png differ diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/index.rst b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/index.rst index 2a8af65..e53953d 100644 --- a/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/index.rst +++ b/doc/source/test_results/container_cluster_systems/kubernetes/API_testing/index.rst @@ -8,7 +8,7 @@ Results of measuring of API performance of Kubernetes This document includes performance test results of Kubernetes API. All tests have been performed regarding - :ref:`Measuring_of_API_performance_of_Kubernetes` + :ref:`Measuring_of_API_performance_of_container_cluster_system` Environment description @@ -81,7 +81,7 @@ bond0 interface of a server: spanning-tree bpduguard enable no snmp trap link-status -Software configuration of kubernetes service +Software configuration of Kubernetes service -------------------------------------------- Setting up Kubernetes ^^^^^^^^^^^^^^^^^^^^^ @@ -196,7 +196,7 @@ You can see more from the `load.py`_ code. +----------------------------+------------------------------------------+ | Ubuntu | Ubuntu 14.04 LTS | +----------------------------+------------------------------------------+ - | e2e-test (kubernetes repo) | v1.3.5 | + | e2e-test (Kubernetes repo) | v1.3.5 | +----------------------------+------------------------------------------+ | Docker | 1.11.2, build b9f10c9 | +----------------------------+------------------------------------------+ @@ -212,7 +212,7 @@ Preparation ----------- 1. Kubernetes was set up on top of 10 nodes as described in - `Setting up kubernetes`_ section. + `Setting up Kubernetes`_ section. 2. e2e-test container was running on top of infrastructure one-node Kubernetes @@ -292,6 +292,44 @@ Please note, that numbers of pods and other items depend on numbers of nodes. | :scale: 50 | :scale: 50 | +---------------------------------------------+-------------------------------+ +Kubernetes pod startup latency measurement +------------------------------------------ + +For this testing purposes `MMM(MySQL/Master/Minions) testing suite`_ was used +(more information in `Pod startup time measurement toolkit`_ section). + +This toolkit was run against 150 nodes Kubernetes environment installed via +`Kargo`_ deployment tool. The most basic configuration (1 replication +controller, N pods, each pod containing 1 container) was run against the +environment. Additional configurations will be tested and results published +in terms of further researches. + +The first run includes information about 500 pods being run on fresh Kubernetes +environment (no tests have been run on it before): + +.. image:: MMM-tool/500_containers_first_run.png + :alt: Containers startup time (500 containers, first run) + :width: 650px + +This weird timings pattern is related to the fact that first 500 containers +pack was run against not warmed up environment (minions images were not +pre-loaded on Kubernetes worker nodes, that means that during first run Docker +registry/repo/etc was really stressed). + +The same scenario run against the warmed-up environment will have linear +pattern (with ~50 milliseconds per container startup): + +.. image:: MMM-tool/500_containers_second_run.png + :alt: Containers startup time (500 containers, second run) + :width: 650px + +This pattern will remain the same with bigger number of containers (15000 +containers, the same ~50 milliseconds per container startup): + +.. image:: MMM-tool/15000_containers.png + :alt: Containers startup time (15000 containers) + :width: 650px + Applications ============ Files and scripts to build Docker container with e2e-test tool @@ -320,9 +358,35 @@ e2e-tests/create_rst_table_from_k8s_e2e_log.py: .. literalinclude:: e2e-tests/create_rst_table_from_k8s_e2e_log.py :language: python +Pod startup time measurement toolkit +------------------------------------ + +for `Kubernetes pod startup latency measurement`_ test case +`MMM(MySQL/Master/Minions) testing suite`_ was used. + +This is a client/server set for testing speed of k8s/docker/networking +scheduling capabilities speed. + +Architecture is simple and consist of the following: + +* MariaDB/MySQL service (replication controller with only one replica) +* Master service, a simple Python application based on `Flask`_ framework with + multiple threads and producer/consumer queue for SQL inserts +* Minion replication controller - a simple bash script which registers minions + on master service. + +This approach guarantees that container will report about its status itself, +so any issues (e.g. too slow startup or unsuccessful at all attempt to create +a container will be observed in the testing results). + +For more details please proceed to the +`MMM(MySQL/Master/Minions) testing suite`_ documentation. + .. references: .. _Kargo: https://github.com/kubespray/kargo .. _e2e-tests: https://github.com/kubernetes/kubernetes/blob/release-1.4/docs/devel/e2e-tests.md -.. _load.py: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/load.go \ No newline at end of file +.. _load.py: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/load.go +.. _MMM(MySQL/Master/Minions) testing suite: https://github.com/AleksandrNull/MMM +.. _Flask: http://flask.pocoo.org/ \ No newline at end of file