openstack/rally-openstack
Code Issues Proposed changes

Propose release notes for Rally 0.10.0

Browse Source 
Change-Id: Idd1a72f17756f275ba453959b35fadf6d650fe55
This commit is contained in:
Andrey Kurilin 2017-07-05 16:18:39 +03:00
parent 04382bc6c9
commit e97e312fc7
6 changed files with 1341 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

520
doc/release_notes/archive/v0.10.0.rst Normal file
View File

@ -0,0 +1,520 @@
=============
Rally v0.10.0
=============
Overview
--------
+------------------+-----------------------+
| Release date | **10/20/2017** |
+------------------+-----------------------+
* Ability to use OpenStack deployments without admin credentials
* Better validation process of tasks
* New task format (with an ability to set description for workloads)
* New JSON report
* ElasticSearch exporter
* `OSProfiler support
<https://rally.readthedocs.io/en/0.10.0/quick_start/tutorial/step_11_profiling_openstack_internals.html>`_
* Restructure of configuration options.
Details
-------
Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~
* Introduce `rally task import
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-task-import>`_
command for importing task results into database.
* Extend tags support for tasks. Now you can specify several tags for a single
task using `--tag argument
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#task-start-tag>`_.
Also filtering tasks by tags is now available.
* Move DB category from ``rally-manage db`` to `rally db
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#category-db>`_ and
introduce `rally db show
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-db-show>`_
command for printing the used connection string.
Deployments
~~~~~~~~~~~
This release we started a huge work related to simplification of deployment
component of Rally. There is a good progress which includes several nice
features:
* The format.
"ExistingCloud" deployment type covers 99.99% cases and is used as a base for
all new things. Also, it will be extended to support different platforms
soon. The new format looks like (for OpenStack case):
.. code-block:: json
{
"openstack": {
"admin": {
"username": "admin",
"password": "changeme",
"tenant_name": "foo",
},
"auth_url": "https://example.com",
}
}
* admin user is optional in case of setting existing users.
From the beginning, setting admin credentials was a required section of Rally
deployment configuration. Even with introducing existing users feature, this
behaviour left.
Finally, we finished a big code refactoring and admin credential become
optional section. If a set of plugins for particular workload doesn't require
admin user, you can launch this task at deployment without setting it.
The information about the requirements of plugins you can find at
`Plugins Reference page
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html>`_ (see
``Requires platform(s):`` section at the bottom of each plugin).
* Originally, Rally project was designed to check performance of OpenStack and
we succeeded in building awesome tool. We do not plan to stop and just want
to inform about our future plans to expand a number of supported platforms.
Subscribe to our `GitHub organization
<https://github.com/xrally>`_ to not miss new plugins.
Task component
~~~~~~~~~~~~~~
* The new task format is introduced. It includes a bunch of improvements,
unification, etc. All the docs and samples will be updated soon.
As for now, you can check `a spec
<https://github.com/openstack/rally/blob/0.10.0/doc/specs/implemented/new_rally_input_task_format.rst>`_
for this big change.
* SLA failure_rate max=0 become a default if nothing else is specified.
* Totally reworked atomic actions. The atomic actions now supports nested
actions which allows to measure durations inside the scenario even more
precise. You can find them in HTML report or in our new json report
(see ``rally task report --json``).
* Generation of names for new resources takes care about particular workload
id, so it helps to provide a better cleanup and prepare for new feature -
disaster cleanup.
Plugins
~~~~~~~
We started supporting discovering plugins by entry-points, so you can easily
deliver your custom plugins as a simple python package.
To make you package after-discoverable, you need to specify the proper
entry-point at your setup.cfg file:
.. code-block::
rally_plugins =
path=package_name
**Deployment Engines**:
Remove serverproviders & rarely used deployers
Unfortunately, seems like nobody is using deployers for deploying
their clouds and mostly people would like just to execute their code.
1) Remove server provides
2) Remove engines that uses server providers
**OpenStack clients**:
* Deprecate EC2 client. It wasn't used in any of plugins and doesn't support
keystone v3
* Move ``rally.osclients`` module to ``rally.plugins.openstack.oscliens``
**Scenarios**:
The old way to describe scenario plugin via method is finally removed.
Initially Rally scenario plugins were methods of special class, like below:
.. code-block:: python
from rally.task import scenario
class SomeBasicClass(scenario.Scenario):
@scenario.configure()
def some_scenario(self, arg1):
"""An implementation of SomeBasicClass.foo scenario."""
@scenario.configure()
def another_scenario(self):
"""Implementation of another scenario, SomeBasicClass.bar."""
However to unify scenarios with other plugins we moved to model where
plugin is class. It was done long time ago.
.. code-block:: python
from rally.task import scenario
@scenario.configure(name="CustomName")
class Some(scenario.Scenario):
def run(self, arg1):
"""An implementation of the scenario."""
We had a bunch of code that was used for transition and backward compatibility
that we finally removed.
* *NEW!!*
- `CinderQos.create_and_get_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-get-qos-scenario>`_
- `CinderQos.create_and_list_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-list-qos-scenario>`_
- `CinderQos.create_and_set_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-set-qos-scenario>`_
- `CinderQos.create_qos_associate_and_disassociate_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-qos-associate-and-disassociate-type-scenario>`_
- `CinderVolumeTypes.create_and_get_volume_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-get-volume-type-scenario>`_
- `CinderVolumeTypes.create_and_list_volume_types
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-list-volume-types-scenario>`_
- `CinderVolumeTypes.create_and_update_encryption_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-encryption-type-scenario>`_
- `CinderVolumeTypes.create_and_update_volume_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-volume-type-scenario>`_
- `CinderVolumeTypes.create_get_and_delete_encryption_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-get-and-delete-encryption-type-scenario>`_
- `CinderVolumeTypes.create_volume_type_add_and_list_type_access
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-volume-type-add-and-list-type-access-scenario>`_
- `Dummy.openstack
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#dummy-openstack-scenario>`_
- `GlanceImages.create_and_deactivate_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-deactivate-image-scenario>`_
- `GlanceImages.create_and_download_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-download-image-scenario>`_
- `GlanceImages.create_and_get_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-get-image-scenario>`_
- `GlanceImages.create_and_update_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-update-image-scenario>`_
- `K8sPods.create_pods
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-pods-scenario>`_
- `K8sPods.create_rcs
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-rcs-scenario>`_
- `K8sPods.list_pods
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-list-pods-scenario>`_
- `ManilaShares.create_and_extend_share
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-extend-share-scenario>`_
- `ManilaShares.create_and_shrink_share
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-shrink-share-scenario>`_
- `ManilaShares.create_share_then_allow_and_deny_access
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-share-then-allow-and-deny-access-scenario>`_
- `NeutronBGPVPN.create_and_delete_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-delete-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_and_list_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_and_list_networks_associations
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-networks-associations-scenario>`_
- `NeutronBGPVPN.create_and_list_routers_associations
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-routers-associations-scenario>`_
- `NeutronBGPVPN.create_and_update_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-update-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_networks
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-networks-scenario>`_
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_routers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-routers-scenario>`_
- `NeutronNetworks.create_and_show_ports
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-ports-scenario>`_
- `NeutronNetworks.create_and_show_routers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-routers-scenario>`_
- `NeutronNetworks.create_and_show_subnets
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-subnets-scenario>`_
- `NeutronNetworks.set_and_clear_router_gateway
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-set-and-clear-router-gateway-scenario>`_
- `NeutronSecurityGroup.create_and_delete_security_group_rule
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-delete-security-group-rule-scenario>`_
- `NeutronSecurityGroup.create_and_list_security_group_rules
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-list-security-group-rules-scenario>`_
- `NeutronSecurityGroup.create_and_show_security_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-scenario>`_
- `NeutronSecurityGroup.create_and_show_security_group_rule
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-rule-scenario>`_
- `NovaServerGroups.create_and_delete_server_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-delete-server-group-scenario>`_
- `NovaServerGroups.create_and_get_server_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-get-server-group-scenario>`_
- `NovaServers.boot_and_get_console_url
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-and-get-console-url-scenario>`_
- `NovaServers.boot_server_and_attach_interface
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-attach-interface-scenario>`_
- `NovaServers.boot_server_and_list_interfaces
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-list-interfaces-scenario>`_
- `NovaServers.boot_server_attach_volume_and_list_attachments
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-attach-volume-and-list-attachments-scenario>`_
* *UPDATED!!*
- The new argument ``properties`` is added to scenario
`IronicNodes.create_and_list_node
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
* *DELETED*
Fuel and Nova-Network are not alive any more. So we removed those scenarios.
If any of those scenarios a critical for you, please contact us.
- `FuelEnvironments.create_and_delete_environment
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-delete-environment-scenario>`_
- `FuelEnvironments.create_and_list_environments
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-list-environments-scenario>`_
- `FuelNodes.add_and_remove_node
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelnodes-add-and-remove-node-scenario>`_
- `NovaFloatingIpsBulk.create_and_delete_floating_ips_bulk
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-delete-floating-ips-bulk-scenario>`_
- `NovaFloatingIpsBulk.create_and_list_floating_ips_bulk
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-list-floating-ips-bulk-scenario>`_
- `NovaNetworks.create_and_delete_network
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-delete-network-scenario>`_
- `NovaNetworks.create_and_list_networks
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-list-networks-scenario>`_
- `NovaSecGroup.boot_and_delete_server_with_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-and-delete-server-with-secgroups-scenario>`_
- `NovaSecGroup.boot_server_and_add_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-server-and-add-secgroups-scenario>`_
- `NovaSecGroup.create_and_delete_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-delete-secgroups-scenario>`_
- `NovaSecGroup.create_and_list_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-list-secgroups-scenario>`_
- `NovaSecGroup.create_and_update_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-update-secgroups-scenario>`_
**Validators**:
The validators refactoring was a long-term task which blocked us to abandon
alignment to only OpenStack platform and requirements of setting admin
credentials. In this release, we made a great progress and fixed a lot of
issues and blockers which made possible to refactor validators.
Now validation step is equal for all types of plugins (Scenario, SLA, Context,
Hooks, Runners, etc).
The old way to add validator for scenario is deprecated. The new unified way
looks like:
.. code-block:: python
import time
from rally.common import validation
from rally.task import scenario
@validation.add("number", param_name="timeout", minval=0)
@scenario.configure(name="Foo.bar")
class FooScenario(scenario.Scenario):
def run(self, timeout):
time.sleep()
The old validators from ``rally.task.validators`` module is deprecated too, see
equivalents which can be used with ``add`` decorator:
- required_openstack --> `required_platform
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-platform-validator>`_
with setting platform argument to "openstack"
- external_network_exists ->`external_network_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#external-network-exists-validator>`_
- file_exists ->`file_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#file-exists-validator>`_
- flavor_exists ->`flavor_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#flavor-exists-validator>`_
- image_exists ->`image_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-exists-validator>`_
- image_valid_on_flavor ->`image_valid_on_flavor
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-valid-on-flavor-validator>`_
- number ->`number
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#number-validator>`_
- required_api_versions ->`required_api_versions
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-api-versions-validator>`_
- required_cinder_services ->`required_cinder_services
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-cinder-services-validator>`_
- required_clients ->`required_clients
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-clients-validator>`_
- required_contexts ->`required_contexts
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-contexts-validator>`_
- required_neutron_extensions ->`required_neutron_extensions
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-neutron-extensions-validator>`_
- required_param_or_context ->`required_param_or_context
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-param-or-context-validator>`_
- required_services ->`required_services
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-services-validator>`_
- restricted_parameters ->`restricted_parameters
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#restricted-parameters-validator>`_
- validate_heat_template ->`validate_heat_template
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#validate-heat-template-validator>`_
- volume_type_exists ->`volume_type_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#volume-type-exists-validator>`_
- workbook_contains_workflow -> `workbook_contains_workflow
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#workbook-contains-workflow-validator>`_
- network_exists is removed, since we do not find any customers for it.
Please contact us if it was useful for you.
- validate_share_proto is removed in favor of enum validator
Fixed bugs
~~~~~~~~~~
* [plugins] JSON schema of `servers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#servers-context>`_
context allows to transmit a list of nics in two formats. First one is a
format that novaclient expects to see (each network should be represented
like ``{"nic-id": "the id of the network"}``). The second one is more
user-friendly - just list of strings (each network should be represented
just by id of the network). Unfortunately, the second case was not covered
by our code base.
Also, the first described format works in limited cases due to bad
serialization.
`Launchpad bug-report #1695245
<https://bugs.launchpad.net/rally/+bug/1695245>`_
* [deployment] ~/rally/.openrc not working for keystone v3
`Launchpad bug-report #1683820
<https://bugs.launchpad.net/rally/+bug/1683820>`_
* [plugins] Failed to list volumes in case of missed name in the object.
* [backported into 0.9.1][deployment] Credentials is not updated as soon as
deployment is recreated. Need to call recreate request twice.
`Launchpad bug-report #1675271
<https://bugs.launchpad.net/rally/+bug/1675271>`_
* [backported into 0.9.1][plugins] Scenario `IronicNodes.create_and_list_node
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
had a wrong check that list of all nodes contains newly created one.
* [backported into 0.9.1][task][cleanup] Do not remove quotas in case of
existing users
* [backported into 0.9.1][task][cleanup] Various traces of neutron resources
* [backported into 0.9.1][core] Keystone v3, authentication error for Rally
users if the value of project_domain_name of admin user isn't equal "default"
`Launchpad bug-report #1680837
<https://bugs.launchpad.net/rally/+bug/1680837>`_
* [backported into 0.9.1][task] Scenario `NovaHosts.list_and_get_hosts
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
obtains hostname for all hosts. But it fails in some environments if host is
not compute.
`Launchpad bug-report #1675254
<https://bugs.launchpad.net/rally/+bug/1675254>`_
* [backported into 0.9.1][verification] Rally fails to run on systems on which
python-virtualenv is not installed
`Launchpad bug-report #1678047
<https://bugs.launchpad.net/rally/+bug/1678047>`_
* [backported into 0.9.1][verification] CLI `rally verify rerun
<https://rally.readthedocs.io/en/0.9.1/verification/cli_reference.html#rally-verify-rerun>`_
fails with TypeError due to wrong integration with Rally API.
* [plugins] Rally fails while creating neutron router on the clouds where
ext-gw-mode extension is not installed.
* [plugins] Scenario `CinderVolumes.create_nested_snapshots_and_attach_volume
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumes-create-nested-snapshots-and-attach-volume-scenario>`_
fails on a big load due to the fact that one server is used for several
iterations. In such case we are facing 2 issues: the maximum number of
volumes per VM is 26 (which is a number of available names for volumes);
detaching volume of one iteration can block attaching of other iterations.
`Launchpad bug-report #1708160
<https://bugs.launchpad.net/rally/+bug/1708160>`_
Thanks
~~~~~~
2 Everybody!

2
doc/release_notes/latest.rst
View File

@ -1 +1 @@
./archive/v0.9.2.rst
./archive/v0.10.0.rst

157
doc/specs/implemented/improve_atomic_actions_format.rst Normal file
View File

@ -0,0 +1,157 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
=============================================
New Atomic actions format in workload results
=============================================
Currently atomic actions data in workload results is insufficient,
therefore some new features can not be implemented.
Problem description
===================
The main problem is that current format does not support nested
atomic actions.
Also, atomic actions data does not include timestamps for each action
start and end time. Having this data will allow us to inspect atomic
actions runtime better and generate detailed reports.
Since word "atomic" means something that can not be split into parts
and we introduce nested atomic actions, we should use different term
instead of "atomic actions".
Proposed change
===============
Term "atomic actions" should be renamed to just "actions".
Change actions results schema from type "object" to "array"
and extend it with timestamps and nested actions.
Nested actions will be represented by "children" key and have
unlimited nesting.
With timestamps, there is no need to save durations anymore,
so get rid of this value.
Since this change is not backward compatible, we need to create
a database migration script. The migration will use iteration start
timestamp as start timestamp for first action and then calculate
further timestamps based on actions order and their durations.
Benefits of new format
----------------------
Nested actions will make actions measurement more detailed and flexible
since we could have data what sub-actions were run during specific action
runtime, without complicated changes at code.
Start and end timestamps will provide us with accurate information
about action runtime within the whole iteration and ability to create
`Gantt charts <https://en.wikipedia.org/wiki/Gantt_chart>`_.
Schema modification
-------------------
Schema location is *rally.common.objects.task.TASK_RESULT_SCHEMA
["properties"]["result"]["properties"]["atomic_actions"]*
should be moved to *rally.common.objects.task.TASK_RESULT_SCHEMA
["properties"]["result"]["properties"]["actions"]*
and changed:
AS IS:
.. code-block:: python
{
"type": "object"
}
Here keys are actions names, and values are their durations.
Actions data is actually represented by collections.OrderedDict,
so we have real order saved.
Example:
.. code-block:: python
OrderedDict([("keystone.create_tenant", 0.1234),
("keystone.create_users", 1.234)])
TO BE:
.. code-block:: python
{
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"}, # name of action
"started_at": {"type": "number"}, # float UNIX timestamp
"finished_at": {"type": "number"}, # float UNIX timestamp
"children": {"$ref": "#/"},
},
"required": ["name", "started_at", "finished_at", "children"],
"additionalProperties": False
},
"minItems": 0
}
Example how this data can be represented:
.. code-block:: python
[{"name": "keystone.create_tenant",
"started_at": 1455281370.288397,
"finished_at": 1455281372.672342,
"children": []},
{"name": "keystone.create_users",
"started_at": 1455281372.931324,
"finished_at": 1455281373.375184,
"children": []}]
Alternatives
------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Alexander Maretskiy <amaretskiy@mirantis.com>
Work Items
----------
- Rename atomic actions into actions
- Improve actions results format
- Create a DB migartion that transforms results to new format
Dependencies
============
None

348
doc/specs/implemented/new_rally_input_task_format.rst Normal file
View File

@ -0,0 +1,348 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
====================================
Make the new Rally input task format
====================================
Current Rally format is not flexible enough to cover all use cases that are
required. Let's change it!
Problem description
===================
Why do we need such fundamental change?
- Multi scenarios load generation support.
This is very important, because it will allow to use Rally for more
real life load generation. Like making load on different components
and HA testing (where one scenario tries for example to authenticate
another is disabling controller)
- Ability to add require meta information like (title and descriptions)
That are required to generate clear reports
- Fixing UX issues. Previous format is very hard for understanding and
end users have issues with understanding how it works exactly.
Proposed change
===============
Make a new format that address all issues.
Old format JSON schema:
.. code-block:: python
{
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema",
"patternProperties": {
".*": {
"type": "array",
"items": {
"type": "object",
"properties": {
"args": {
"type": "object"
},
"runner": {
"type": "object",
"properties": {
"type": {"type": "string"}
},
"required": ["type"]
},
"context": {
"type": "object"
},
"sla": {
"type": "object",
},
},
"additionalProperties": False
}
}
}
}
Old format sample:
.. code-block:: yaml
---
<ScenarioName>:
-
args: <dict_with_scenario_args>
runner: <dict_with_runner_type_and_args>
context:
<context_name>: <dict_with_context_args>
...
sla:
<sla_name>: <sla_arguments>
-
-//-
-
-//-
<AnotherScenarioName>:
-//-
Every element of list corresponding to <ScenarioName> is separated task,
that generates environment according to context, generates load using
specified runner that runs multiple times <ScenarioName> with it's args.
New format JSON schema:
.. code-block:: python
{
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema",
"properties": {
"version": {"type": "number"},
"title": {"type": "string"},
"description": {"type": "string"},
"tags": {
"type": "array",
"items": {"type": "string"}
},
"subtasks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"description": {"type": "string"},
"tags": {
"type": "array",
"items": {"type": "string"}
},
"run_in_parallel": {"type": "boolean"},
"workloads": {
"type": "array",
"items": {
"type": "object",
"properties": {
"scenario": {"type": "object"},
"runner": {"type": "object"}
"sla": {"type": "object"},
"contexts": {"type": "object"}
},
"required": ["scenario", "runner"]
}
},
"contexts": {"type": "object"}
},
"required": ["title", "workloads"]
}
}
},
"required": ["title", "tasks"]
}
New format sample:
.. code-block:: yaml
---
# Having Dictionary on top level allows us in future to add any new keys.
# Keeping the schema of format more or less same for end users.
# Version of format
version: 2
# Allows to set title of report. Which allows end users to understand
# what they can find in task report.
title: "New Input Task format"
# Description allows us to put all required information to explain end
# users what kind of results they can find in reports.
description: "This task allows you to certify that your cloud works"
# Explicit usage "rally task start --tag" --tag attribute
tags: ["periodic", "nova", "cinder", "ha"]
subtasks:
# Note every task is executed serially (one by one)
#
# Using list for describing what subtasks to run is much better idea then
# using dictionary. It resolves at least 3 big issues:
#
# 1) Bad user experience
# 1.1) Users do not realize that Rally can run N subtask
# 1.2) Keys of Dictionary were Scenario names (reasonable question why?!)
# 1.3) Users tried to put N times same k-v (to run one subtask N times)
# 2) No way to specify order of scenarios execution, especially in case
# where we need to do chain like: ScenarioA -> SecnearioB -> ScenarioA
# 3) No way to support multi scenario load, because we used scenario name
# as a identifier of single task
-
# title field is required because in case of multi scenario load
# we can't use scenario name for it's value.
title: "First task to execute"
description: "We will stress Nova" # optional
# Tags are going to be used in various rally task reports for filtering
# and grouping.
tags: ["nova", "my_favorite_task", "do it"]
# The way to execute scenarios (one by one or all in parallel)
run_in_parallel: False
# Single scenario load can be generated by specifying only one element
# in "workloads" section.
workloads:
-
scenario:
NovaServers.boot_and_delete:
image:
name: "^cirros$"
flavors:
name: "m1.small"
runner:
constant:
times: 100
concurrency: 10
# Subtask success of criteria based on results
sla:
# Every key means SLA plugin name, values are config of plugin
# Only if all criteria pass task is marked as passed
failure_rate:
max: 0
# Specification of context that creates env for scenarios
# E.g. it creates users, tenants, sets quotas, uploads images...
contexts:
# Each key is the name of context plugin
# This context creates temporary users and tenants
users:
# These k-v will be passed as arguments to this `users` plugin
tenants: 2
users_per_tenant: 10
# This context set's quotas for created by `users` context tenants
quotas:
nova:
cpu: -1
-
title: "Second task to execute"
description: "Multi Scenario load generation with common context"
run_in_parallel: True
# If we put 2 or more scenarios to `scenarios` section we will run
# all of them simultaneously which allows us to generate more real life
# load
workloads:
-
scenario:
CinderVolumes.create_and_delete:
size: 10
runner:
constant:
times: 100
concurrency: 10
sla:
failure_rate:
max: 0
-
scenario:
KeystoneBasic.create_and_delete_users:
name_length: 20
runner:
rps:
rps: 1
times: 1000
sla:
max_seconds_per_iteration: 10
-
scenario:
PhysicalNode.restart:
ip: "..."
user: "..."
password: "..."
runner:
rps:
rps: 10
times: 10
sla:
max_seconds_per_iteration: 100
# This scenario is called in own independent and isolated context
contexts: {}
# Global context that is used if scenario doesn't specify own
contexts:
users:
tenants: 2
users_per_tenant: 10
Alternatives
------------
No way
Implementation
==============
Assignee(s)
-----------
Primary assignee:
boris-42 aka Boris Pavlovic
Work Items
----------
- Implement OLD -> NEW format converter
- Switch task engine to use new format. This should affect only task engine
- Implement new DB schema format, that will allow to store multi-scenario
output data
- Add support for multi scenario results processing in rally task
detailed|sla_check|report
- Add timestamps to task, scenarios and atomics
- Add support for usage multi-runner instance in single task with
common context
- Add support for scenario's own context
- Add ability to use new format in rally task start.
- Deprecate OLD format
Dependencies
============
None

206
doc/specs/implemented/pluggable_validators.rst Normal file
View File

@ -0,0 +1,206 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
=================================
Rally Task Validation Refactoring
=================================
Problem description
===================
* Current validator system is pluggable - but it doesn't use our plugin
mechanism which creates problems (e.g. validators are imported directly and
used in code, instead of using their names, which doesn't allow to rename
them or move without breaking backward compatibility).
* Current mechanism of validation leads to a lot of OpenStack related code in
the Rally task engine.
* It's hard to use the same validators for different types of plugins, current
approach is used only for scenarios.
Proposed change
===============
To create unified validation mechanism that can be used for all types of
future deployments and type of plugins in the same way. So we will be able
to remove `OpenStack related code <https://github
.com/openstack/rally/blob/be8cd7bff6de9b3e83dd31005ae5d07ca1c86b9e/rally
/task/engine.py#L188-L278>`_ from the task engine, and create a bunch of
common validators (e.g. jsonschema) that can be used by any
plugin.
As a bonus of refactoring, it allows us to switch to common mechanism of
plugins.
Alternatives
------------
No way
Implementation
==============
Here is an example of base class for all pluggable validators.
.. code-block:: python
import abc
import six
from rally.common.plugin import plugin
from rally.task import validation
def configure(name, platform="default"):
return plugin.configure(name=name, platform=platform)
@six.add_metaclass(abc.ABCMeta)
@configure(name="base_validator")
class Validator(plugin.Plugin):
def validate(self, cache, deployment, cfg, plugin_cfg):
"""
Method that validates something.
:param cache: this is cross validator cache where different
validators could store information about
environment like initialized OpenStack clients,
images, etc and share it through validators.
E.g. if your custom validators need to perform 200
OpenStack checks and each validator plugin need to
initialize client, Rally will take extra 2 minutes
for validation step. As well, its not efficient to
fetch all image each time if we have image related
validators.
:param deployment: Deployment object, deployment which would be
used for validation
:param cfg: dict, configuration of subtask
:param plugin_cfg: dict, with exact configuration of the plugin
"""
pass
def add(name, **kwargs):
"""
Add validator instance to the validator plugin class meta.
Get validator class by name. Initialize an instance. Add validator
instance to validators list stored in the Validator meta by
'validator_v2' key. This would be used to iterate and execute through
all validators used during execution of subtask.
:param kwargs: dict, arguments used to initialize validator class
instance
:param name: str, name of the validator plugin
"""
validator = Validator.get(name)(**kwargs)
def wrapper(p):
p._meta_setdefault("validators_v2", [])
p._meta_get("validators_v2").append(validator)
return p
return wrapper
@abc.abstractmethod
def validate(plugin, deployment, cfg, plugin_cfg):
"""
Execute all validate() method of all validators stored in meta of
Validator.
Iterate during all validators stored in the meta of Validator and
execute proper validate() method and add validation result to the
list.
:param plugin: is plugin class instance that has validators and should
be validated
:param deployment: Deployment object, deployment which would be
used for validation
:param cfg: dict, configuration of subtask
:param plugin_cfg: dict, with exact configuration of the plugin
"""
results = []
cache = {}
for v in plugin._meta_get("validators_v2"):
try:
v.validate(cache, deployment, cfg, plugin_cfg)
except Exception as e:
results.append(validation.ValidationResult(is_valid=False,
msg=e))
return results
New design allows us to use the same validator and same validation mechanism
for different types of plugins (context, sla, runner, scenarios) which was not
possible before. For example, we could implement jsonschema validation as a
plugin.
.. code-block:: python
import jsonschema
@configure(name="jsonschema")
class JsonSchemaValidator(Validator):
def __init__(self, schema=None):
super(JsonSchemaValidator, self).__init__()
self.schema = schema or {}
def validate(self, cache, deployment, cfg, plugin_cfg):
jsonschema.validate(plugin_cfg, self.schema)
@validator.add("jsonschema", schema="<here_json_schema>")
class SomeContext(base.Context):
pass
class SomeScenario(base.Scenario):
@validator.add("jsonschema", schema="<here_json_schema>")
def some_function(self):
pass
Assignee(s)
-----------
Primary assignee:
- boris-42 <bpavlovic@mirantis.com>
- rvasilets <rvasilets@mirantis.com>
Work Items
----------
- Create validation module with base plugin and method of adding validators
- Add support to task engine of new validation mechanism
- Port all old validators to new mechanism
- Deprecate old validation mechanism
- Remove deprecated in new release
Dependencies
============
None

109
doc/specs/implemented/task_and_verification_export.rst Normal file
View File

@ -0,0 +1,109 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
====================================================
Export task and verifications into external services
====================================================
Currently Rally stores all information about executed tasks and verifications
in its database and it is also able to provide this data in JSON format or
in the form of HTML reports. There is a request for Rally to export this data
into external services (like test management system or Google Docs)
via its API.
Problem description
===================
There are many, including a lot of proprietary, test management systems
in the market available as SaaS and/or On-Premises, like TestRail, TestLink,
TestLodge etc, which objective is to manage, organize and track all testing
efforts.
Most of the systems provide an API for importing test data. The systems also
possess data model somewhat similar to Rally's one.
It usually includes (among others) models for project, test suite test case,
test plan and test execution results.
It is suggested to provide Rally users an ability to export information about
testing their environments into such test management systems in order
to integrate testing via Rally into rest of their testing activities.
Since different test management systems have alike yet different API
for the purpose it is reasonable to implement this export functionality via
plugins.
Proposed change
===============
1. Implement a base class Exporter for an export plugin at
*rally/task/exporter.py*.
..code-block:: python
class Exporter(plugin.Plugin):
def export(self, task, connection_string):
...
2. Implement a CLI command of the form
..code-block:: shell
rally task export <UUID> <CONNECTION_STRING>
3. Implement a base class VerifyExporter for an export plugin at
*rally/verify/exporter.py*.
..code-block:: python
class VerifyExporter(plugin.Plugin):
def export(self, verification, connection_string):
...
4. Implement a CLI command of the form
..code-block:: shell
rally verify export <UUID> <CONNECTION_STRING>
Alternatives
------------
No way
Implementation
==============
Assignee(s)
-----------
Primary assignee:
rvasilets <rvasilets@mirantis.com>
Work Items
----------
- Implement plugin base class
- Implement CLI command
- Implement plugin for TestRail
Dependencies
============
None