From e987dd57bc7c409abb31e12e4da152318a1168f7 Mon Sep 17 00:00:00 2001 From: Andrey Kurilin Date: Fri, 20 Jan 2017 15:00:51 +0200 Subject: [PATCH] [docs] Move several docs Change-Id: Idc6ce01f779847f28302f8ef9e8da6b277c949ac --- doc/source/cli_reference.rst | 11 + doc/source/index.rst | 6 +- doc/source/task/index.rst | 380 ++++++++++++++++++++++++++++++ doc/source/verification/index.rst | 27 +++ 4 files changed, 421 insertions(+), 3 deletions(-) create mode 100644 doc/source/cli_reference.rst create mode 100644 doc/source/task/index.rst create mode 100644 doc/source/verification/index.rst diff --git a/doc/source/cli_reference.rst b/doc/source/cli_reference.rst new file mode 100644 index 00000000..23fb7220 --- /dev/null +++ b/doc/source/cli_reference.rst @@ -0,0 +1,11 @@ +.. _cli-reference: + + +Command Line Interface +====================== + +.. contents:: + :depth: 1 + :local: + +.. make_cli_reference:: diff --git a/doc/source/index.rst b/doc/source/index.rst index 59583684..32ff7489 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -39,9 +39,9 @@ Contents overview/index install_and_upgrade/index quick_start/index - cli/cli_reference - components/task - components/verification + cli_reference + task/index + verification/index plugins/index contribute feature_requests diff --git a/doc/source/task/index.rst b/doc/source/task/index.rst new file mode 100644 index 00000000..fdf0993f --- /dev/null +++ b/doc/source/task/index.rst @@ -0,0 +1,380 @@ +.. _task-component: + +============== +Task Component +============== + +This section describes Rally Task Component (including feature presented since +Rally v0.5.0, allowing to analyze statistics trends for the given tasks). + +.. contents:: + :depth: 2 + :local: + +HTML Reports +============ + +HTML reports provide comprehensive analysis. +Data is structured and displayed interactively, with charts and tables. + +Task Report +----------- + +Get the whole information about task workloads results, in pretty +and convenient format! + +.. image:: ../images/Report-Collage.png + +Generate report for single task, using task UUID +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Having a finished task, generate report with command: + +.. code-block:: shell + + $ rally task report --out + +Example: + +.. code-block:: shell + + $ rally task report 6f63d9ec-eecd-4696-8e9c-2ba065c68535 --out report.html + +Generate report for single task, using JSON file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Report can be generated from a task results JSON file. +This file can be generated with command *rally task results*: + +.. code-block:: shell + + $ rally task results 6f63d9ec-eecd-4696-8e9c-2ba065c68535 > results.json + $ rally task report results.json --out report.html + +Generate report for many tasks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Report can be generated from many tasks. All workloads from specified +tasks results will be composed into an entire report. +To generate report, use *--tasks* argument with specified list of tasks UUIDs +and/or tasks results JSON files. + +Example: + +.. code-block:: shell + + $ rally task report --tasks 6f63d9ec-eecd-4696-8e9c-2ba065c68535 20ae7e95-7395-4be4-aec2-b89220adee60 a5737eba-a204-43d6-a262-d5ea4b0065da results.json another_results.json --out report.html + +Task Overview +~~~~~~~~~~~~~ + +This is a table with brief summary of all workloads results. +All columns are sortable and clickable. + +.. image:: ../images/Report-Task-Overview.png + +Load duration ++++++++++++++ + +Time from first iteration start to last iteration end. +In other words, this is a time of all workload iterations execution. + +Full duration ++++++++++++++ + +This time includes iterations time (`Load duration <#load-duration>`_) +plus time taken by another actions related to the task, mostly Contexts +execution time. + +Iterations +++++++++++ + +How many times the workload has run. This comes from the value of +*runner.times* in task input file. + +Failures +++++++++ + +Number of failed iterations. Failure means that there was an Exception raised. + +Success (SLA) ++++++++++++++ + +This is a boolean result of workload SLA. See +`Service-level agreement explanation <#id2>`_ below. + +Input file +~~~~~~~~~~ + +This shows JSON which can be used to run a task with exactly the same workloads +list and configuration. This is not an exact copy (neither concatenation) of +actually used input files (in command *rally task start*), however this is +exactly what is needed to run workloads given in the report. + +.. image:: ../images/Report-Task-Input-file.png + +Tab «Overview» +~~~~~~~~~~~~~~ + +Service-level agreement ++++++++++++++++++++++++ + +`SLA`_ results appear in task report only if *"sla"* section is defined in task +input file. + +For example, having this in task input file: + +.. code-block:: json + + "sla": { + "performance_degradation": { + "max_degradation": 50 + }, + "max_seconds_per_iteration": 1.0, + "failure_rate": { + "max": 0 + }, + "outliers": { + "max": 1, + "min_iterations": 10, + "sigmas": 10 + }, + "max_avg_duration": 0.5 + } + +will result SLA section similar to the following: + +.. image:: ../images/Report-Task-SLA.png + + +What if workload has no "sla" configuration in input file? +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +If *"sla"* section is missed in input file, then block *Service-level +agreement* is not displayed and its result is assumed to be always passed +(no matter how many failures occurred). + +Total durations ++++++++++++++++ + +There is a durations analysis, which is represented by statistics table and +duration StackedArea chart. + +.. image:: ../images/Report-Task-Total-durations.png + +Table with statistics data +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +**Action** + Name of the workload metric that has some duration saved. + This is either an atomic action name or *Total* which points to workload +`load duration <#load-duration>`_. + +**Min (sec)** + `Minimal`_ duration value + +**Median (sec)** + `Median`_ duration value + +**90%ile (sec)** + `Percentile`_ for 90% durations + +**95%ile (sec)** + `Percentile`_ for 95% durations + +**Max (sec)** + `Maximal`_ duration value + +**Avg (sec)** + `Average`_ duration value + +**Success** + Percent of successful runs. This is how many percent of this action runs + (number of runs is given in *Count* column) were successful. + +**Count** + Number of actually run atomic actions. This can differ from + `iterations count <#iterations>`_ because some atomic actions do not start if + some exception is raised before in the workload runtime (for example in + previous atomic action). + +StackedArea with durations per iteration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This chart shows `load_duration <#load-duration>`_ and `idle_duration <#id5>`_ +values per iteration. If there is only one iteration, then chart is useless so +it is hidden. + +Idle duration +^^^^^^^^^^^^^ + +Sometimes workload does nothing for some reason (waiting for something or just +making a dummy load). This is achieved by calling *time.sleep()* and spent time +is called *idle duration*. + +Load Profile +++++++++++++ + +`Load profile`_ chart shows number of iterations running in parallel for each +workload moment: + +.. image:: ../images/Report-Task-Load-profile.png + +Distribution +++++++++++++ + +Pie chart shows percent of successful and failed `iterations <#iterations>`_. + +Histogram shows durations distribution with the following `methods`_ (selected +in dropdown list): **Square Root Choice**, **Sturges Formula**, **Rise Rule** + +.. image:: ../images/Report-Task-Distribution.png + +Tab «Details» +~~~~~~~~~~~~~ + +Atomic Action Durations ++++++++++++++++++++++++ + +There is a StackedArea chart that shows atomic actions durations per iteration. +If there is only one iteration, then chart is useless so it is hidden. + +.. image:: ../images/Report-Task-Actions-durations.png + +Distribution +++++++++++++ + +`Distribution <#distribution>`_ for atomic actions durations + +Tab «Scenario Data» +~~~~~~~~~~~~~~~~~~~ + +This tab only appears if workload provides some custom output via method +*Scenario.add_output()*. + +Aggregated +++++++++++ + +This shows charts with data aggregated from all iterations. +This means that each X axis point represents an iteration, so each iteration +provided some values that are aggregated into charts or tables. + +.. image:: ../images/Report-Task-Scenario-Data-Aggregated.png + +Per iteration ++++++++++++++ + +Each iteration can create its own, complete charts and tables. + +.. image:: ../images/Report-Task-Scenario-Data-Per-iteration.png + +Tab «Failures» +++++++++++++++ + +Complete information about exceptions raised during the workload run + +**Iteration** + Number of iteration where exception is occurred + +**Exception type** + Type of raised Exception subclass + +**Exception message** + Message delivered by the exception + +Click on a row expands it with exception traceback. + +.. image:: ../images/Report-Task-Failures.png + +Tab «Input Task» +~~~~~~~~~~~~~~~~ + +This shows JSON for input file which can be used to run current workload. + +.. image:: ../images/Report-Task-Subtask-configuration.png + +Trends Report +------------- + +If same workload is run several times, some results of these runs can be +compared. Compared metrics are ssuccess rate (percent of successful iterations) +and statistics for durations. + +How to generate trends report +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use command *rally task trends* with given tasks UUIDs and/or tasks results +JSON files and the name of desired output file. + +Example: + +.. code-block:: shell + + $ rally task trends --tasks 6f63d9ec-eecd-4696-8e9c-2ba065c68535 a5737eba-a204-43d6-a262-d5ea4b0065da --out trends.html + +What is an order of workload runs? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Workload run number in shown on charts X axis, the order of runs is exactly as +it comes from tasks data in the moment of report generation. + +Trends overview +~~~~~~~~~~~~~~~ + +.. image:: ../images/Report-Trends-Overview.png + +If workload has been actually run only once +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +That is obvious that it is not possible to have trend for a single value. +There should be at least two workload runs to make results comparison possible. +So in this case there is only a help message displayed. + +.. image:: ../images/Report-Trends-single-run.png + +Tab «Total» +~~~~~~~~~~~ + +Total durations ++++++++++++++++ + +Shows workload `load_duration <#load-duration>`_ statistics trends. + +Total success rate +++++++++++++++++++ + +Shows trends for percent of successful iterations + +.. image:: ../images/Report-Trends-Total.png + +Tab «Atomic actions» +++++++++++++++++++++ + +Statistics trends for atomic actions durations. +Charts are same as for total durations. + +.. image:: ../images/Report-Trends-Atomic-actions.png + +Tab «Configuration» ++++++++++++++++++++ + +Here is a configuration JSON for current workload. + +.. image:: ../images/Report-Trends-Configuration.png + +CLI References +============== + +For more information regarding Rally Task Component CLI please proceed +to `CLI reference <../cli/cli_reference.html#category-task>`_ + +.. references: + +.. _SLA: https://en.wikipedia.org/wiki/Service-level_agreement +.. _Minimal: https://en.wikipedia.org/wiki/Maxima_and_minima +.. _Median: https://en.wikipedia.org/wiki/Median +.. _Percentile: https://en.wikipedia.org/wiki/Percentile +.. _Maximal: https://en.wikipedia.org/wiki/Maxima_and_minima +.. _Average: https://en.wikipedia.org/wiki/Average +.. _Load profile: https://en.wikipedia.org/wiki/Load_profile +.. _methods: https://en.wikipedia.org/wiki/Histogram diff --git a/doc/source/verification/index.rst b/doc/source/verification/index.rst new file mode 100644 index 00000000..27d32e45 --- /dev/null +++ b/doc/source/verification/index.rst @@ -0,0 +1,27 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +====================== +Verification Component +====================== + +Functional testing is a first step to ensuring that your product works as +expected and API covers all use-cases. Rally Verification Component is all +about this. It is not designed to generate a real big load (for this job we +have :ref:`task-component`), but it should be enough to check that your +environment works by different tools (we call them +:ref:`glossary-verification`). + +.. toctree:: + :maxdepth: 3 + :glob: