Kubernetes pod recovery test plan and report

Change-Id: I8db42b5ef403b191755c6a2c4771df6927c89e57

doc/source/test_plans/kubernetes_pod_recovery/code/Dockerfile

@@ -0,0 +1,7 @@
+FROM debian:latest
+ENV DEBIAN_FRONTEND noninteractive
+
+RUN apt-get update && apt-get -y upgrade
+RUN apt-get -y --no-install-recommends install python
+
+ENTRYPOINT ["python", "-m", "SimpleHTTPServer"]

doc/source/test_plans/kubernetes_pod_recovery/code/svc-dpl.yaml

@@ -0,0 +1,20 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: svc
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: svc
+      annotations:
+        scheduler.alpha.kubernetes.io/affinity: '{"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution":
+          {"nodeSelectorTerms": [{"matchExpressions": [{"key": "kubernetes.io/hostname",
+          "operator": "In", "values": ["node4", "node5", "node6"]}]}]}}}'
+    spec:
+      containers:
+      - name: svc
+        image: 127.0.0.1:31500/qa/svc
+        ports:
+        - containerPort: 8000

doc/source/test_plans/kubernetes_pod_recovery/code/svc-svc.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: svc
+spec:
+  ports:
+  - nodePort: 32333
+    port: 8000
+    protocol: TCP
+    targetPort: 8000
+  selector:
+    app: svc
+  type: NodePort

doc/source/test_plans/kubernetes_pod_recovery/plan.rst

@@ -0,0 +1,80 @@
+.. _Kubernetes_pod_recovery_test_plan:
+
+*********************************
+Kubernetes POD recovery test plan
+*********************************
+
+:status: **ready**
+:version: 1.0
+
+:Abstract:
+
+  This test plan covers scenarios for Kuberneter POD recovery testing.
+
+Test Plan
+=========
+
+In Kubernetes "classic" HTTP-based services can be represented with help of
+replication controller and service. Replication controller is responsible
+for keeping number of PODs constant and in restarts POD in case of failure.
+If the POD is deleted, then replication controller creates a new one.
+Depending on scheduler settings a new instance of POD can be scheduled
+to a different node.
+
+Under service performance we mean the amount of work produced by a service,
+for HTTP service this can be measured as number of requests per second.
+
+Test Environment
+----------------
+
+Preparation
+^^^^^^^^^^^
+
+The test plan is executed against Kubernetes deployed on bare-metal nodes.
+
+Environment description
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The environment description includes hardware specification of servers,
+network parameters, operation system and OpenStack deployment characteristics.
+
+
+Test Case #1: POD restart
+-------------------------
+
+Description
+^^^^^^^^^^^
+
+In this test case we investigate how POD rescheduling affects service
+performance.
+
+Steps:
+ * Build Docker image and push it into local registry:
+   ``docker build -t localhost:31500/qa/svc code/``
+   ``docker push localhost:31500/qa/svc
+ * Create Kubernetes Service and Deployments based on files
+   :download:`code/svc-dpl.yaml` and :download:`code/svc-svc.yaml`
+ * Measure service performance during 20 seconds:
+   ``wrk -c 20 -d 10 -t 20 -H "Connection: close" --latency --timeout 10s http://svc.ccp:8000/``
+ * While measurement is running delete the pod via
+   ``kubectl delete pod -l app=svc``
+
+List of performance metrics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. table:: list of test metrics to be collected during this test
+
+  +-------------------------+---------------------------------------------+
+  | Parameter               | Description                                 |
+  +=========================+=============================================+
+  | RPS_BASE                | Base RPS value                              |
+  +-------------------------+---------------------------------------------+
+  | RPS_WHILE_RESCHEDULING  | RPS measured during rescheduling            |
+  +-------------------------+---------------------------------------------+
+
+
+Reports
+=======
+
+Test plan execution reports:
+ * :ref:`Kubernetes_pod_recovery_test_report`

doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/index.rst

@@ -0,0 +1,56 @@
+.. _Kubernetes_pod_recovery_test_report:
+
+******************************
+Kubernetes density test report
+******************************
+
+:Abstract:
+
+  This document is the report for :ref:`Kubernetes_pod_recovery_test_plan`
+
+
+Environment description
+=======================
+
+This report is collected on the hardware described in
+:ref:`intel_mirantis_performance_lab_1`.
+
+
+Software
+~~~~~~~~
+
+Kubernetes is installed using :ref:`Kargo` deployment tool on Ubuntu 16.04.1.
+
+Node roles:
+ - node1: minion+master+etcd
+ - node2: minion+master+etcd
+ - node3: minion+etcd
+ - node4: minion
+
+Software versions:
+ - OS: Ubuntu 16.04.1 LTS (Xenial Xerus)
+ - Kernel: 4.4.0-47
+ - Docker: 1.12.1
+ - Kubernetes: 1.4.3
+
+Reports
+=======
+
+Test Case #1: POD restart
+-------------------------
+
+With Kubernetes support of gracefully POD restart the service downtime is
+minimized and may result in several requests to fail. The following chart is
+based on 11 measurements: column 1 - no POD restart, 2 - with POD restart:
+
+.. image:: chart.png
+
+.. list-table:: POD stats
+    :header-rows: 1
+
+    *
+      - RPS_BASE
+      - RPS_WHILE_RESCHEDULING
+    *
+      - 1136 (stddev 162)
+      - 818 (stddev 158)