openinfra/interop
Code Issues Proposed changes
interop/working_materials/scoring.txt
Mark T. Voelker fbd9c0dccc Add cutoff_score to next.json, update script 
We recently added a field in Schema 1.4 to record a
"cutoff_score".[1]  The field reflects the minimum score that a
Capability needs to achieve for a given Guideline in order to
warrant inclusion.  The current draft Guideline, however, hasn't yet
 been updated to include a cutoff score.

This patch adds a cutoff_score field to next.json and sets it to
a preliminary vale of 74 (which is indicative of the score we've
used in the past as a cutoff, but which can be changed by the
DefCore Committee any time between now and when the Guideline goes to
the Board for approval before it becomes binding).  The patch also
updates the tabulate_scores.py script to read the cutoff_score from the
JSON document and adds an asterisk (*) to lines in the scoring file whose
totals meet or exceed the cutoff as a visual indicator of which
Capabilities have scored highly enough to warrant inclusion in the
Guideline.  Verbage in the header of the scoring.txt file has also been
updated to reflect the meaning of the indicator.

The patch also adds the cutoff_score field to the 2016.01 document,
where it was inadvertently left off since the field was only added
to the schema in close proximity to the document being finalized for
Board approval.

[1] https://review.openstack.org/#/c/224868/

Change-Id: I73d08baae13f373c955ff94640fac43109125886
2016-04-22 12:26:37 -04:00

244 lines
14 KiB
Plaintext
Raw Blame History

DefCore Scoring Worksheet
=========================
This file is used to maintain capabilities scoring for DefCore.
It is considered "working materials" under the 2016A process and
is not an authoritative, Board-approved source of information
as to which capabilities are required for a particular Guideline.
Rather, it is a point-in-time listing used by DefCore Committee
members while determining which capabilities may be included in
a future Guideline. Each line takes the following form:
Capability Name: [Widely Deployed,Used By Tools,Used by Clients],
[TC Future Direction, Complete, Stable],
[Discoverable, Doc'd, Core in Last Release],
[Foundational, Atomic, Proximity],
[Non-Admin] [Total]
Possible scores: 0, 1, ? (not scored yet or needs discussion)
The Total score is calculated by multiplying the score for each
Criteria by the weight assigned to that Criteria in the Guideline
JSON file. It is not necessary to calculate the score by hand:
use the tabulate_scores.py script found in this directory. The
script will also append an asterisk (*) to the end of the line if
the total score is greater than or equal to the cutoff_score listed
in the Guideline JSON file, indicating that the Capability
has scored high enough to merit inclusion in the Guideline.
Networking
----------
network-floating-ips-associate: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-create: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-update: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-delete: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-l2-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-external: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-port: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-convert: [0,0,0] [1,0,0] [0,0,0] [0,0,0] [1] [11]
network-l3-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-router: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-extensions-provider-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-subnet-pools: [1,0,0] [1,1,1] [1,1,0] [0,1,1] [1] [66]
Notes:
* DVR-related testcases (if any) will not be added until DVR gains wider
acceptance and isn't driver-specific.
* IPv6 is currently not widely deployed, so v6 versions of these tests need
not be added at this time. Vendor feedback on v6 support is requested,
though it is understood that IPv6 capabilities may be driver-dependent.
* A few folks have asked about provider network extensions as they are
frequently used for high performance networks and a simplified networking
model that provides for use cases similar to nova-net. However with default
policy settings, using provider networks requires admin access and therefore
is disqualified. I've listed them here anyway just to show that they were
considered but can't be added to the Guideline.
* Based on feedback from earlier patchsets, capabilities will be grouped
together where they had previously been listed separately but are adjacent.
For example, there are several tests for network creation, but only one or
few for deletion, listing, and updating. Combining these into a single
"CRUD" group reduces the chance the chance of running into problems with
DefCore's 2015A process (specifically section C3(4)) in cases where there
is only a single test for a capability and the test needs to be flagged on
account of bugs or other valid reasons. The combined groups are:
* networks-floating-ips-CRUD-and-associate
* networks-floating-ips-associate
* networks-floating-ips-create
* networks-floating-ips-update
* networks-floating-ips-delete
* networks-l2-CRUD
* networks-l2-create
* networks-l2-delete
* networks-l2-external
* networks-l2-list
* networks-l2-port
* networks-l2-show
* networks-l2-update
* networks-l3-CRUD
* networks-l3-add
* networks-l3-create
* networks-l3-delete
* networks-l3-show
* networks-l3-update
* networks-l3-router
* networks-security-groups-CRUD
* networks-security-groups-create
* networks-security-groups-delete
* networks-security-groups-router
* networks-security-groups-show
* networks-security-groups-update
* Subnet pools look like a good future candidate, but as of 2016.08
probably aren't quite ready. Subnet pools were introduced in Kilo
(the first release covered by this Guideline) and had incomplete
support in Horizon. A Heat resource for subnet pools wasn't added
until Mitaka. Toolkits like jClouds, Fog, and libcloud do not yet
support subnet pools. OpenStackClient added support late in the
Mitaka cycle (https://review.openstack.org/#/c/279918/).
Compute
-------
compute-list-api-versions: [1,0,0] [1,1,1] [1,1,0] [1,1,1] [1] [75]*
Notes:
* There has been some discussion that this capability is not widely
deployed [1]. However it appears that in most cases the API either:
* Is present and works, but only if a non-SSL endpoint is exposed.
SSL endpoints may have issues with redirects (which may actually
merely be configuration issues).
* Is accidentally blocked due to load balancer or API gateway
misconfiguration which is being remedied by affected providers.
Taking those situations into account, it is believed that it's actually
safe to call this "widely deployed".
* I'm having trouble finding evidence that tools actually support this
API (refer to [2] for information about what constitutes "tools").
Unlike most of the CRUD-like API's, this one lists information about
the API versions available instead of on end user actionable resources
(such as instances or security groups), and the need to do so is newish
with the advent of microversioning. Hence, I think many tools may not
actually support this yet. Please feel free to provide evidence to the
contrary; I'd love to be proven wrong.
* Similarly, I'm having trouble finding evidence that clients [3] like
jClouds or Fog support this API (yet?) as they tend to focus on actions.
Please feel free to provide evidence to the contrary; I'd love to be
proven wrong.
* I'm tentatively scoring this API as Proximate [4]. Generally this applies
to CRUD operations (e.g. the delete API is considered proximate to the
create API). However I'm providing a 1 here on the grounds that due
to the shift to microversioning in the Nova API, this will be somewhat
proximate to everything going forward.
* The DefCore Committee is considering this capability for inclusion
in it's projected 2016.01 Guideline even though the test for it wasn't
developed until (narrowly) outside the normal window for identifying
new Capabilities on account of strong indications from the community
that such a Capability is important for API microversioning going
forward [5].
[1] https://bugs.launchpad.net/python-novaclient/+bug/1491579
[2] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n44
[3] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n48
[4] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n90
[5] http://eavesdrop.openstack.org/meetings/defcore_flag_14/2015/defcore_flag_14.2015-09-09-15.03.log.html#l-13
Image
-----
images-v1-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-index: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-list: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-register: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-add: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-get: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-remove: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-update: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v2-remove: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-update: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-share: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-import: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-list: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-delete: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-get: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
Notes:
* Image creation is captured under the register operation.
* No public image tests have explicit support for task API.
* Scoring for v1 remains in place, but v2 is preferred interop
standard (as reflected in worksheet).
Volume
----------
volumes-v2-create-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-attach-detach: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-snapshot-create-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
volumes-v2-copy-image-to-volume: [1,0,0] [1,1,1] [1,1,1] [1,1,1] [1] [84]*
volumes-v2-clone: [1,0,1] [1,1,1] [1,1,1] [1,1,1] [1] [94]*
volumes-v2-qos: [1,0,1] [1,1,1] [1,1,1] [1,1,0] [1] [86]*
volumes-v2-availability-zones: [1,0,0] [1,1,1] [1,1,1] [1,1,0] [1] [76]*
volumes-v2-extensions: [1,0,0] [0,1,1] [1,1,1] [1,0,0] [1] [59]
volumes-v2-metadata: [1,0,1] [1,1,1] [1,1,1] [1,1,0] [1] [86]*
volumes-v2-transfer: [1,0,0] [1,1,1] [1,1,1] [1,1,0] [1] [76]*
volumes-v2-reserve: [1,0,0] [1,1,1] [1,1,1] [1,1,0] [1] [76]*
volumes-v2-readonly: [1,0,1] [1,1,1] [1,1,1] [1,1,0] [1] [86]*
volumes-v2-extend: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
volumes-v2-upload: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
Identity
--------
identity-v3-api-discovery: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
identity-v3-catalog: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
identity-v3-list-projects: [1,0,1] [1,1,1] [1,1,0] [0,1,0] [1] [68]
identity-v3-list-groups: [1,0,1] [1,1,1] [1,1,0] [0,1,0] [1] [68]
Notes:
* identity-v3-catalog is returned when the api for
identity-v3-tokens-create is called (GET /v3/auth/tokens). It is
important to consider it because end users may be relying on this
catalog for their apps (even though there are other API calls that
also show the catalog such as GET /v3/auth/catalog). There is one test
available for this capability but it is in the admin part of the test
suite, so not yet tested for non-admin users. Even though it scores enough
to be included as advisory, we cannot do this due to lack of non-admin
test case.
* identity-v3-list-projects and identity-v3-list-groups are here because
they deserve some visibility and some explicit test cases, which at the
moment they are lacking. It seems important for users to be able to
discriminate between projects and groups when running their apps.
Object Store
------------
objectstore-object-copy: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-put: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-upload: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-versioned: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-temp-url-get: [1,1,1] [1,1,1] [1,1,1] [1,1,0] [1] [92]*
objectstore-account-quotas: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-account-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-container-acl: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-container-quotas: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-container-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-container-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-container-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
View Git Blame Copy Permalink