openstack/rally-openstack
Code Issues Proposed changes

Release notes for 0.11.0

Browse Source 
Change-Id: I0d26a082dd50a59dff281ea6cbe42115d75363ad
This commit is contained in:
Andrey Kurilin 2018-02-07 15:37:32 +02:00
parent 349ec91f43
commit 8974baa6ef
4 changed files with 642 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

263
doc/release_notes/archive/v0.11.0.rst Normal file
View File

@ -0,0 +1,263 @@
=============
Rally v0.11.0
=============
Overview
--------
+------------------+-----------------------+
| Release date | **02/15/2018** |
+------------------+-----------------------+
* Stabilize Rally releases (see requirements section)
* Publishing docker images is returned!
* Environment introduction (see Deployment section)
Details
-------
Requirements
~~~~~~~~~~~~
As for long time we tried to make our releases stable and workable even after
a year from release date. For this purpose, upper limits for all our
requirements were used. Such solution helped to make releases more stable, but
it did not work well and created more pain than a profit.
The main issue was related to new releases of not-direct rally
dependencies which can be incompatible which each other.
From Rally 0.11.0 we are stopping adding upper version limits for our
`requirements
<https://github.com/openstack/rally/blob/0.11.0/requirements.txt>`_ (this does
not apply to cases when we sure that some new releases of dependency broke us
for sure).
As alternative solution, the `upper-constraints file
<https://github.com/openstack/rally/blob/0.11.0/upper-constraints.txt>`_ is
selected. It is a file with a list of packages and their versions which we used
in our CI while testing. Versions from this list are suggested to use in
production.
Please note, that it also contains not direct Rally dependencies (dependencies
of rally dependencies and dependencies of dependencies of rally dependencies
as well) and packages which possibly doesn't relate to Rally at all.
The example ou usage:
.. console:: bash
$ git clone https://github.com/openstack/rally && cd rally
$ pip install . --constraint upper-constraints.txt
Logging
~~~~~~~
The default value of ``use_stderr`` option of Rally config is changed to
**True**. It is done to ensure that json output of rally commands will not be
messed with logging and integration with third-party tools and scripts should
become simpler.
Docker
~~~~~~
Unfortunately, we lost access to `rallyforge organization at DockerHub
<https://hub.docker.com/r/rallyforge/>`_, so our docker images were not
published anywhere officially for some time. Docker Support did not help us a
lot. At least, the original repo is removed now and we can start new
organization at DockerHub from the scratch.
The new docker images for Rally+OpenStack plugins with an updated HowTo you
can find at `xrally/xrally-openstack repo
<https://hub.docker.com/r/xrally/xrally-openstack>`_. It contains all Rally
releases + latest tag which maps to master branch.
Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~
* Introduce `rally db ensure
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#rally-db-ensure>`_
command. It is going to create Rally DB if it doesn't exist, otherwise it
does nothing.
* Various improvements for `rally db
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-db>`_.
Such as printing results of operations as well as connection string to DB,
hiding credentials by default, etc.
* Various changes in returning error codes. Error codes become different for
different exceptions. Also, `rally task start
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#rally-task-start>`_
began to return 1 in case of any runtime issues and 2 if any workload doesn't
pass it's SLA.
* The new CLI for new component (see ``Environments & Deployments`` section
for more details) - `rally env
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-env>`_
Environments & Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~
Deployment Component is fundamental part of Rally which is used by Task and
Verification components. Whereas Task and Verification components experienced
redesigns (some parts were redesigned even several times), Deployment component
was only extended over the time.
Currently, we are at the point when further extending requires much work and
user experience become worth. It is a hard decision, as like we had done with
Verification component in Rally 0.8, the Deployment component is re-written
from the scratch. To be clear, the new version of the component has not only
the new code, but the new name as well - it is the Environment.
We are at the stage when Rally is suitable not only for OpenStack but for
various platforms and systems. The Environment is some entity against which
Rally should perform load. The specific Environment can include credentials for
OpenStack or for Kubernetes or for both. The Environment is a way to describe
the complex system/cloud which all of us have. As well it can be used for
simple systems as easy as for complex.
If you are regular Rally user which tests OpenStack clusters, nothing is
changing for you at this moment. `rally deployment
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-deployment>`_
CLI still works and it is not even deprecated. It will be kept for a while.
But you can start looking at our new CLI `rally env
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-env>`_ .
As for writing plugins for external platforms - we are working on updating our
documentary.
Task component
~~~~~~~~~~~~~~
* The json results are extended with context execution stats (i.e when and
which context was executed, how much time setup and cleanup methods took,
etc). Also, `volumes@openstack
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#volumes-context>`_
and
`volume_types@openstack
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#volume-types-context>`_
are ported to include detailed stats of executed actions. In further
releases, all existing contexts will be adopted to the similar behavior.
* Better OSProfiler integration
Rally environment&deployment config began accept
``profiler_conn_str`` property which is used to generate OSProfiler native
html-report and to embed it to Rally's html-report.
* HTML report is extended to include a timestamp of failures.
Plugins
~~~~~~~
As it mentioned above, the Deployment Component will be replaced soon. Such
decision led to abandoning all deployment-related plugins (engines, providers,
etc).
**Scenarios**:
* *NEW!!*
`NovaServers.boot_server_attach_volume_and_list_attachments
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-attach-volume-and-list-attachments-scenario>`_
* *UPDATED!!*
- Extend several Nova&Neutron related scenarios with
``create_floating_ip_args`` parameter
`NovaServers.boot_and_associate_floating_ip
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-and-associate-floating-ip-scenario>`_
`NovaServers.boot_server_associate_and_dissociate_floating_ip
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-server-associate-and-dissociate-floating-ip-scenario>`_
- Modify Bgpvpn scenarios to test true bgpvpn
All Bgpvpn scenarios began to boot a server to add active port in the
network associated to the bgpvpn which allows to test not only the record in
the database, but true bgpvpn
**Contexts**:
*UPDATED!!*
`network@openstack
<https://rally.readthedocs.io/en/latest/plugins/plugin_reference.html#network-context>`_
context is extended with ability to specify external router information.
Fixed bugs
~~~~~~~~~~
* [backported into 0.10.1][deployment] Suppress deprecation warning about an
old format in case of using `--fromenv option of rally deployment create
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-create-fromenv>`_
* [backported into 0.10.1][deployment] Failure `rally deployment show
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-show>`_
while displaying the information about deployment with a config in an old
format.
* [backported into 0.10.1][task] New json report processed the hook results in
a wrong way
`Launchpad bug-report #1734336
<https://bugs.launchpad.net/rally/+bug/1734336>`_
* [backported into 0.10.1][task] Failure while generating trends reports in
case of failures in setup method of any context
`Launchpad bug-report #1732193
<https://bugs.launchpad.net/rally/+bug/1732193>`_
* [backported into 0.10.1][task] Failure to export results in ElasticSearch 5.x
cluster in case of extra ``/`` in the end of destination url.
* [deployment] OpenStack deployment creation with `--fromenv
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-create-fromenv>`_
option used old deprecated format.
* [verify] Rally did not support creating verifiers from Gerrit/SSH source.
`Launchpad bug-report #1737529
<https://bugs.launchpad.net/rally/+bug/1737529>`_
* [task][openstack] Removing default security group in users@openstack context
did not take into account that neutron can return multiple resources in some
configuration instead of one security group which relates to requested
tenant.
* [task][openstack] Existing openstack users get their roles un-assigned if
they are equal to what roles@openstack context is configured to assign.
`Launchpad bug-report #1720270
<https://bugs.launchpad.net/rally/+bug/1720270>`_
* [task][openstack] Validation step ignores roles@openstack context and
marks as "invalid" valid cases
Some actions in openstack can be performed only if the user has specific
role. Since these roles can be different in different OpenStack installations
Rally has `roles@openstack context
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#roles-context>`_
context which can assign roles to the users.
Validation step did not check for specified roles in workload config and made
wrong assumption about accessibility of resources
`Launchpad bug-report #1539878
<https://bugs.launchpad.net/rally/+bug/1539878>`_
* [task][openstack] Wrong identifiers were used for filtering Mistral resources
while cleanup step.
* [task][openstack] `NovaServers.boot_and_live_migrate_server
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-and-live-migrate-server-scenario>`_
does wrong target host selection
`Launchpad bug-report #1734914
<https://bugs.launchpad.net/rally/+bug/1734914>`_
Thanks
~~~~~~
2 Everybody!

2
doc/release_notes/latest.rst
View File

@ -1 +1 @@
./archive/v0.10.1.rst archive/v0.11.0.rst

216
doc/specs/implemented/deployment_type.rst Normal file
View File

@ -0,0 +1,216 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Rally Deployment Unification
============================
Make Rally be able to examine any software through the API,
unbound it from OpenStack.
Problem description
===================
Rally is able to examine only system that use Keystone as a authentication
services, which limits sphere where Rally is suitable.
At the moment to run Rally Task or Rally Verify you must specify OpenStack
deployment which contains credentials of it. These credentials are used in
Rally Task & Verify for different setups and validations.
Rally is not able to store more than one credential for one deployment, so
it is impossible to support multi-scenario runs related to different systems.
Proposed change
===============
* Modify 'Deployment' database model to be able to store credentials
of many different systems, adding type of system.
Now we have model Deployment with admin and users columns,
which are credentials for Keystone (tight coupled with OpenStack).
There is next model now:
.. code-block:: python
class Deployment(BASE, RallyBase):
...
admin = sa.Column(types.PickleType, nullable=True)
users = sa.Column(types.PickleType, default=[], nullable=False)
...
and values of columns in DB something like that:
``admin = {admin_creds} or None``
``users = [{user_creds1}, {user_creds2}, ...] or []``
We need to decouple deployment from OpenStack and
make credentials more flexible, we describe it in one column named
``credentials``, where we can store special structure containing credentials
for many different systems, including type of credentials for each.
.. code-block:: python
class Deployment(BASE, RallyBase):
...
credentials = sa.Column(types.PickleType, default=[], nullable=False)
...
So, for current OpenStack credentials we will have next data
in credentials column in DB after migration:
.. code-block:: python
credentials = [
[
"openstack",
{admin: {admin_creds} or None,
users: [{user_creds1}, {user_creds2}, ...] or []}
],
]
and for multi-credentials deployment:
.. code-block:: python
credentials = [
[
"openstack",
{admin: {admin_creds} or None,
users: [{user_creds1}, {user_creds2}, ...] or []}
],
[
"zabbix",
{"url": "example.com", "login": "admin", "password": "admin"}
]
]
Future summarized schema in DB:
``credentials = [[<type>, <Credentials>], ... ]``
To implement this point we need to write db migration, tests for it
and write adapters for credentials get/create/update methods,
mostly for support backward compatibility in ``rally.api`` module methods.
* Get rid of ``rally.common.objects.credential.Credential`` class
and fix it usages mostly in ``rally.osclients`` if needed.
Refactor all usages of passing ``rally.common.objects.credential.Credential``
to ``rally.osclients.OSClient``, make possible to take dict as credentials
for ``rally.osclients.OSClient`` class, initialise
``rally.plugins.openstack.credentials.OpenStackCredentials`` class
in ``OSClient`` ``__init__`` method.
Base class for credentials will be inherited from plugins.Plugin
and must implement validation method,
it will be placed in ``rally.plugins.common.credentials``:
.. code-block:: python
@six.add_metaclass(abc.ABCMeta)
@plugin.configure(name="base_credentials", schema="{...}")
class Credentials(plugin.Plugin):
def __init__(self, credentials):
self.validate(credentials)
super(Credentials, self).__setattr__("credentials", credentials)
def __getattr__(self, item):
if item in self.__dict__:
return self.__dict__[item]
return self.credentials[item]
def __setattr__(self, key, value):
self.credentials[key] = value
def to_dict(self):
return self.credentials.copy()
def validate(self, obj):
jsonschema.validate(obj, self._meta_get("schema"))
and we need to add child for openstack credentials,
it will be placed in ``rally.plugins.openstack.credentials``:
.. code-block:: python
openstack_credentials_schema = {
"type": "object",
"properties": {
"auth_url": {"type": "string"},
"username": {"type": "string"},
"password": {"type": "string"},
},
"required": ["auth_url", "username", "password"]
}
@plugin.configure(name="openstack_credentials",
schema=openstack_credentials_schema)
class OpenStackCredentials(Credentials):
pass
Replace usage of ``rally.common.objects.credential.Credential`` to
``rally.plugins.openstack.credentials.OpenStackCredentials``
in ``rally.osclients``
* Update cli to show deployment type in output of 'rally deployment list'.
Make possible to show deployments list in case of multi-scenario as:
.. code-block:: shell
> rally deployment list # (in case of many deployments)
uuid | name | created_at | type | credential
-------+--------+------------+-----------+---------------------------------
<uuid> | <name> | 21-02-2016 | openstack | {"admin": {...}, "users": [...]}
| zabbix | {"login": "login", "psw": "..."}
Alternatives
------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
rpromyshlennikov aka Rodion Promyshlennikov (rpromyshlennikov@mirantis.com)
Work Items
----------
- Change Deployment db model class
- Write migrations
- Make adapters for credentials get/create/update methods for temporary
support changed data format
- Remove all usages of passing ``rally.common.objects.credential.Credential``
to ``rally.osclients.OSClient``
- Create new plugin based class for credentials
- Write subclass of rally.plugins.common.credentials.Credential
for OpenStack credentials with proper validation of them
- Migrate to new credentials class
- Remove ``rally.common.objects.credential.Credential`` class
- Improve CLI-client to make possible show multi-credentials deployments.
- Feature refactoring: remove adapters after
"Multi Scenario support" implementation.
Dependencies
============
None

162
doc/specs/implemented/osprofiler.rst Normal file
View File

@ -0,0 +1,162 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://www.sphinx-doc.org/en/stable/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
=================================
OSProfiler integration into Rally
=================================
The OSProfiler is a distributed trace toolkit library. It provides pythonic
helpers to do trace generation to avoid repeated code to trace WSGI, RPC, DB,
and other important places...
Integration OSProfiler into Rally can help to dig into concurrency problems of
OpenStack which is a huge ecosystem of cooperative services.
Problem description
===================
Rally Framework provides a powerful interface to generate real, big, load for
the deployment. Such load can kill the cloud, specifically OpenStack. There is
no way to identify reasons and bottlenecks without parsing timestamps and logs.
To fix that issue embedding profiling into each of workload iteration can help
to display wide picture of where we were in that particular moment when
something went wrong.
Proposed change
===============
Two facts about OSProfiler which are the start point for the proposed changes:
* HMAC key is used as a secret identifier while profiling
* Initialization of profiling is made in thread safe mode. Profiling of one
iteration should not influence profiling another one
Storing secret key
------------------
The HMAC key is not something that will be changed from one task to another.
It is specific thing for the deployment, like authentication url or other
credentials. That is why Rally deployment config is the best place to store
such information.
Since OSProfiler is OpenStack specific tool, we need to extend OpenStack
credentials model in Rally to support new argument. It should be done in two
places: validation (by modifying jsonschema [0]_) and the place where
credentials are store (specific class [1]_ [2]_).
Initialization profiling
------------------------
As it was mentioned before, we need to initialize OSProfiler per iteration.
OSProfiler is made in thread safe mode [3]_, so we should not have problem from
that side.
Initialization of OSProfiler is quite simple.
.. code-block:: python
from osprofiler import profiler
profiler.init(HMAC_KEY)
As for the place where to initialize OSProfiler in Rally, constructor of
scenario is a good choice. First of all, we have a separate class for OpenStack
scenarios [4]_ which means that integration with OSProfiler there will not
affect all other platforms. Another reason for using constructor is that we
initialize new instance of scenario class for each iterations.
Storing profiling results
-------------------------
OSProfiler sends to collector a message at every trace point. We should not
care about supported OSProfiler backends and use only OSProfiler as
entry-point.
The full trace can be obtained via trace-id after profiling is initialized.
.. code-block:: python
from osprofiler import profiler
trace_id = profiler.get().get_base_id()
At the first step of integration OSProfiler in Rally, let's store that trace-id
just like simple text. It will allow to show trace-id in Rally HTML and JSON
reports.
.. code-block:: python
self.add_output(complete={"title": "OSProfiler Trace-ID",
"chart_plugin": "TextArea",
"data": [trace_id]})
We can execute these lines in the same place where we initialize OSProfiler.
In future, we should develop a separate chart that will embed OSProfiler html
report as a separate tab in the Rally report.
Enabling profiling
------------------
Enabling/disabling profiling should be done via rally configuration file:
* It is common place for storing different kinds of options.
* There is planned feature that will able to re-set config options via
deployment config or task file.
The default value of that options should be True. In case of missing HMAC key
in credentials, attempt to initialize OSProfiler should not be started.
Alternatives
------------
Here [5]_ you can find the answer to that section.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Andrey Kurilin <andr.kurilin@gmail.com>
Work Items
----------
* Extend OpenStack credentials
* Add new configuration option to Rally
* Extend OpenStack scenario base class to initialize OSProfiler and store
trace id
Dependencies
============
None
References
==========
.. [0] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L154
.. [1] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L26
.. [2] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L161
.. [3] https://github.com/openstack/osprofiler/blob/1.8.0/osprofiler/profiler.py#L29-L30
.. [4] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/scenario.py#L28-L55
.. [5] https://docs.openstack.org/osprofiler/latest/user/background.html#why-not-cprofile-and-etc