Add Ussuri cycle specs

Change-Id: I48924a5438395af77531c0b7f20137069415d5c2

doc/source/index.rst

@@ -25,6 +25,7 @@ Here you can find the specs, and spec template, for each release:
    :glob:
    :maxdepth: 1
 
+   specs/ussuri/index
    specs/train/index
    specs/stein/index
    specs/rocky/index

doc/source/specs/ussuri/approved

@@ -0,0 +1 @@
+../../../../specs/ussuri/approved/

doc/source/specs/ussuri/implemented

@@ -0,0 +1 @@
+../../../../specs/ussuri/implemented/

doc/source/specs/ussuri/index.rst

@@ -0,0 +1,26 @@
+=============================
+Watcher Ussuri Specifications
+=============================
+
+Template:
+
+.. toctree::
+   :maxdepth: 1
+
+   Specification Template (Ussuri release) <template>
+
+Ussuri implemented specs:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   implemented/*
+
+Ussuri approved (but not implemented) specs:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   approved/*

doc/source/specs/ussuri/redirects

@@ -0,0 +1 @@
+../../../../specs/ussuri/redirects

doc/source/specs/ussuri/template.rst

@@ -0,0 +1 @@
+../../../../specs/ussuri-template.rst

priorities/ussuri-priorities.rst

@@ -0,0 +1,12 @@
+.. ussuri-priorities:
+
+=========================
+Ussuri Project Priorities
+=========================
+
+List of priorities the Watcher drivers team is prioritizing in Ussuri.
+
++--------------------------------------+----------------------+
+| Priority                             | Owner                |
++======================================+======================+
++--------------------------------------+----------------------+

specs/ussuri-template.rst

@@ -0,0 +1,379 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==========================================
+Example Spec - The title of your blueprint
+==========================================
+
+Include the URL of your launchpad blueprint:
+
+https://blueprints.launchpad.net/watcher/+spec/example
+
+Introduction paragraph -- why are we doing anything? A single paragraph of
+prose that operators can understand. The title and this first paragraph
+should be used as the subject line and body of the commit message
+respectively.
+
+Some notes about the watcher-spec and blueprint process:
+
+* Not all blueprints need a spec. For more information see
+  http://docs.openstack.org/developer/nova/blueprints.html#specs
+
+* The aim of this document is first to define the problem we need to solve,
+  and second agree the overall approach to solve that problem.
+
+* This is not intended to be extensive documentation for a new feature.
+  For example, there is no need to specify the exact configuration changes,
+
+  nor the exact details of any DB model changes. But you should still define
+  that such changes are required, and be clear on how that will affect
+  upgrades.
+
+* You should aim to get your spec approved before writing your code.
+  While you are free to write prototypes and code before getting your spec
+  approved, its possible that the outcome of the spec review process leads
+  you towards a fundamentally different solution than you first envisaged.
+
+* But, API changes are held to a much higher level of scrutiny.
+  As soon as an API change merges, we must assume it could be in production
+  somewhere, and as such, we then need to support that API change forever.
+  To avoid getting that wrong, we do want lots of details about API changes
+  upfront.
+
+Some notes about using this template:
+
+* Your spec should be in ReSTructured text, like this template.
+
+* Please wrap text at 79 columns.
+
+* The filename in the git repository should match the launchpad URL, for
+  example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing
+  should be named awesome-thing.rst
+
+* Please do not delete any of the sections in this template.  If you have
+  nothing to say for a whole section, just write: None
+
+* For help with syntax, see http://sphinx-doc.org/rest.html
+
+* To test out your formatting, build the docs using tox and see the generated
+  HTML file in doc/build/html/specs/<path_of_your_file>
+
+* If you would like to provide a diagram with your spec, ascii diagrams are
+  required.  http://asciiflow.com/ is a very nice tool to assist with making
+  ascii diagrams.  The reason for this is that the tool used to review specs is
+  based purely on plain text.  Plain text will allow review to proceed without
+  having to look at additional files which can not be viewed in gerrit.  It
+  will also allow inline feedback on the diagram itself.
+
+* If your specification proposes any changes to the Watcher REST API such
+  as changing parameters which can be returned or accepted, or even
+  the semantics of what happens when a client calls into the API, then
+  you should add the APIImpact flag to the commit message. Specifications with
+  the APIImpact flag can be found with the following query:
+
+  https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z
+
+
+Problem description
+===================
+
+A detailed description of the problem. What problem is this blueprint
+addressing?
+
+Use Cases
+----------
+
+What use cases does this address? What impact on actors does this change have?
+Ensure you are clear about the actors in each use case: Developer, End User,
+Deployer etc.
+
+
+Proposed change
+===============
+
+Here is where you cover the change you propose to make in detail. How do you
+propose to solve this problem?
+
+If this is one part of a larger effort make it clear where this piece ends. In
+other words, what's the scope of this effort?
+
+At this point, if you would like to just get feedback on if the problem and
+proposed change fit in Watcher, you can stop here and post this for review to
+get preliminary feedback. If so please say:
+Posting to get preliminary feedback on the scope of this spec.
+
+Alternatives
+------------
+
+What other ways could we do this thing? Why aren't we using those? This doesn't
+have to be a full literature review, but it should demonstrate that thought has
+been put into why the proposed solution is an appropriate one.
+
+Data model impact
+-----------------
+
+Changes which require modifications to the data model often have a wider impact
+on the system.  The community often has strong opinions on how the data model
+should be evolved, from both a functional and performance perspective. It is
+therefore important to capture and gain agreement as early as possible on any
+proposed changes to the data model.
+
+Questions which need to be addressed by this section include:
+
+* What new data objects and/or database schema changes is this going to
+  require?
+
+* What database migrations will accompany this change.
+
+* How will the initial set of new data objects be generated, for example if you
+  need to take into account existing instances, or modify other existing data
+  describe how that will work.
+
+REST API impact
+---------------
+
+Each API method which is either added or changed should have the following
+
+* Specification for the method
+
+  * A description of what the method does suitable for use in
+    user documentation
+
+  * Method type (POST/PUT/GET/DELETE)
+
+  * Normal http response code(s)
+
+  * Expected error http response code(s)
+
+    * A description for each possible error code should be included
+      describing semantic errors which can cause it such as
+      inconsistent parameters supplied to the method, or when an
+      instance is not in an appropriate state for the request to
+      succeed. Errors caused by syntactic problems covered by the JSON
+      schema definition do not need to be included.
+
+  * URL for the resource
+
+  * Parameters which can be passed via the url
+
+  * JSON schema definition for the body data if allowed
+
+  * JSON schema definition for the response data if any
+
+* Example use case including typical API samples for both data supplied
+  by the caller and the response
+
+* Discuss any policy changes, and discuss what things a deployer needs to
+  think about when defining their policy.
+
+Note that the schema should be defined as restrictively as
+possible. Parameters which are required should be marked as such and
+only under exceptional circumstances should additional parameters
+which are not defined in the schema be permitted (e.g.,
+additionalProperties should be False).
+
+Reuse of existing predefined parameter types such as regexps for
+passwords and user defined names is highly encouraged.
+
+Security impact
+---------------
+
+Describe any potential security impact on the system.  Some of the items to
+consider include:
+
+* Does this change touch sensitive data such as tokens, keys, or user data?
+
+* Does this change alter the API in a way that may impact security, such as
+  a new way to access sensitive information or a new way to login?
+
+* Does this change involve cryptography or hashing?
+
+* Does this change require the use of sudo or any elevated privileges?
+
+* Does this change involve using or parsing user-provided data? This could
+  be directly at the API level or indirectly such as changes to a cache layer.
+
+* Can this change enable a resource exhaustion attack, such as allowing a
+  single API interaction to consume significant server resources? Some examples
+  of this include launching subprocesses for each connection, or entity
+  expansion attacks in XML.
+
+For more detailed guidance, please see the OpenStack Security Guidelines as
+a reference (https://wiki.openstack.org/wiki/Security/Guidelines).  These
+guidelines are a work in progress and are designed to help you identify
+security best practices.  For further information, feel free to reach out
+to the OpenStack Security Group at openstack-security@lists.openstack.org.
+
+Notifications impact
+--------------------
+
+Please specify any changes to notifications. Be that an extra notification,
+changes to an existing notification, or removing a notification.
+
+Other end user impact
+---------------------
+
+Aside from the API, are there other ways a user will interact with this
+feature?
+
+* Does this change have an impact on python-watcherclient? What does the user
+  interface there look like?
+
+Performance Impact
+------------------
+
+Describe any potential performance impact on the system, for example
+how often will new code be called, and is there a major change to the calling
+pattern of existing code.
+
+Examples of things to consider here include:
+
+* A periodic task might look like a small addition but if it calls conductor or
+  another service the load is multiplied by the number of nodes in the system.
+
+* Scheduler filters get called once per host for every instance being created,
+  so any latency they introduce is linear with the size of the system.
+
+* A small change in a utility function or a commonly used decorator can have a
+  large impacts on performance.
+
+* Calls which result in a database queries (whether direct or via conductor)
+  can have a profound impact on performance when called in critical sections of
+  the code.
+
+* Will the change include any locking, and if so what considerations are there
+  on holding the lock?
+
+Other deployer impact
+---------------------
+
+Discuss things that will affect how you deploy and configure OpenStack
+that have not already been mentioned, such as:
+
+* What config options are being added? Are the default values ones which will
+  work well in real deployments?
+
+* Is this a change that takes immediate effect after its merged, or is it
+  something that has to be explicitly enabled?
+
+* If this change is a new binary, how would it be deployed?
+
+* Please state anything that those doing continuous deployment, or those
+  upgrading from the previous release, need to be aware of. Also describe
+  any plans to deprecate configuration values or features.  For example, if we
+  change the directory name that instances are stored in, how do we handle
+  instance directories created before the change landed?  Do we move them?  Do
+  we have a special case in the code? Do we assume that the operator will
+  recreate all the instances in their cloud?
+
+Developer impact
+----------------
+
+Discuss things that will affect other developers working on OpenStack.
+
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Who is leading the writing of the code? Or is this a blueprint where you're
+throwing it out there to see who picks it up?
+
+If more than one person is working on the implementation, please designate the
+primary author and contact.
+
+Primary assignee:
+  <launchpad-id or None>
+
+Other contributors:
+  <launchpad-id or None>
+
+Work Items
+----------
+
+Work items or tasks -- break the feature up into the things that need to be
+done to implement it. Those parts might end up being done by different people,
+but we're mostly trying to understand the timeline for implementation.
+
+
+Dependencies
+============
+
+* Include specific references to specs and/or blueprints in Watcher, or in
+  other projects, that this one either depends on or is related to.
+
+* If this requires functionality of another project that is not currently used
+  by Watcher (such as the glance v2 API when we previously only required v1),
+  document that fact.
+
+* Does this feature require any new library dependencies or code otherwise not
+  included in OpenStack? Or does it depend on a specific version of library?
+
+
+Testing
+=======
+
+Please discuss the important scenarios needed to test here, as well as
+specific edge cases we should be ensuring work correctly. For each
+scenario please specify if this requires specialized hardware, a full
+openstack environment, or can be simulated inside the Watcher tree.
+
+Please discuss how the change will be tested. We especially want to know what
+tempest tests will be added. It is assumed that unit test coverage will be
+added so that doesn't need to be mentioned explicitly, but discussion of why
+you think unit tests are sufficient and we don't need to add more tempest
+tests would need to be included.
+
+Is this untestable in gate given current limitations (specific hardware /
+software configurations available)? If so, are there mitigation plans (3rd
+party testing, gate enhancements, etc).
+
+
+Documentation Impact
+====================
+
+What is the impact on the docs team of this change? Some changes might require
+donating resources to the docs team to have the documentation updated. Don't
+repeat details discussed above, but please reference them here.
+
+
+References
+==========
+
+Please add any useful references here. You are not required to have any
+reference. Moreover, this specification should still make sense when your
+references are unavailable. Examples of what you could include are:
+
+* Links to mailing list or IRC discussions
+
+* Links to notes from a summit session
+
+* Links to relevant research, if appropriate
+
+* Related specifications as appropriate (e.g.  if it's an EC2 thing, link the
+  EC2 docs)
+
+* Anything else you feel it is worthwhile to refer to
+
+
+History
+=======
+
+Optional section for liberty intended to be used each time the spec
+is updated to describe new design, API or any database schema
+updated. Useful to let reader understand what's happened along the
+time.
+
+.. list-table:: Revisions
+   :header-rows: 1
+
+   * - Release Name
+     - Description
+   * - Ussuri
+     - Introduced
+

specs/ussuri/approved/general-purpose-decision-engine-threadpool.rst

@@ -0,0 +1,170 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==========================================
+General purpose decision engine threadpool
+==========================================
+
+https://blueprints.launchpad.net/watcher/+spec/general-purpose-decision-engine-threadpool
+
+Many I/O operations such as writing to disk or waiting for network responses
+take a long amount of time. By leveraging parallelism the time taken to
+perform such operations can be drastically reduced. The general purpose
+threadpool for the decision engine will allow to perform methods in parallel,
+thereby reducing the time required to perform these operations.
+
+Problem description
+===================
+
+The general purpose threadpool will reduce the time it takes to perform certain
+operations of the decision engine. Currently, the gathering of metrics or the
+building of the data model can take large amounts of time proportionally for
+example.
+
+Use Cases
+----------
+
+- As a user I want the execution of an audit to be performed as quickly as
+  possible.
+
+Proposed change
+===============
+
+Introduce a general purpose threadpool using the `futurist`_ library for which
+end users can configure the amount of threads. This will introduce a new class
+that uses a singular pattern so it can be used throughout the decision engine.
+The threadpool will use the GreenThreadPoolExecutor as it does not conflict
+with other threadpools used in the decision engine. The singleton will contain
+several methods to simplify the submission of tasks and waiting for their
+completion.
+
+.. _futurist: https://docs.openstack.org/futurist/latest/
+
+With the threadpool the building of the data model will be parallelized which
+is done in three steps. First the calls to retrieve information about
+aggregates and availability zones are executed in parallel. The information is
+used to submit tasks to gather the information about each compute node.
+Finally, using `futures`_ will allow to immediately submit a task to retrieve
+the information about instances for a given compute node.
+
+.. _futures: https://docs.python.org/dev/library/concurrent.futures.html
+
+Alternatives
+------------
+
+A purpose build threadpool specific for building the data model. However, this
+would mean that introducing parallelism in other parts of the decision engine
+would again require the design and implementation of yet another threadpool.
+
+Data model impact
+-----------------
+
+None
+
+REST API impact
+---------------
+
+None
+
+Security impact
+---------------
+
+None
+
+Notifications impact
+--------------------
+
+None
+
+Other end user impact
+---------------------
+
+None
+
+Performance Impact
+------------------
+
+Overall the time take to perform audits should significantly decrease, however,
+a large amount of threads could negatively impact external services. Because of
+this the default amount of threads should be a relatively safe value for most
+OpenStack infrastructures.
+
+Other deployer impact
+---------------------
+
+Deployers should evaluate the amount of threads they want Watcher to use based
+on the scale of their infrastructure.
+
+Developer impact
+----------------
+
+None
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+  <dantalion>
+
+Work Items
+----------
+
+- Implement the threadpool singleton based on previous proof of concepts.
+
+- Modify the method to build the data model to use the three step parallelism
+  as described.
+
+- Implement relevant test cases to test the threadpool and the building of the
+  data model.
+
+Dependencies
+============
+
+* futurist, A library maintained and developed by OpenStack.
+
+
+Testing
+=======
+
+Both the behavior of the building of the data model as well as the threadpool
+should be evaluated.
+
+- The verifying of behavior for helper methods in the threadpool.
+
+- The verifying of the completeness of the data model when build with
+  parallelization.
+
+Documentation Impact
+====================
+
+The documentation will be updated to include usage examples that indicate to
+other developers how to best use the threadpool. Additionally, typical use
+cases in what situations this threadpool will be most useful will be provided.
+
+
+References
+==========
+
+* `futurist`_
+* `futures`_
+
+.. _futurist: https://docs.openstack.org/futurist/latest/
+.. _futures: https://docs.python.org/dev/library/concurrent.futures.html
+
+History
+=======
+
+.. list-table:: Revisions
+   :header-rows: 1
+
+   * - Release Name
+     - Description
+   * - Ussuri
+     - Introduced
+

specs/ussuri/approved/ussuri-template.rst

@@ -0,0 +1 @@
+../../ussuri-template.rst

specs/ussuri/implemented/ussuri-template.rst

@@ -0,0 +1 @@
+../../ussuri-template.rst