openinfra/interop
Code Issues Proposed changes
e60481879b
interop/working_materials/scoring.txt
Luz Cazares b757001456 Keystone scoring for 2017.08 
Updated scores for existing capabilities.
identity-v3-validate-token but it is admin only.

Change-Id: I111aecf36eae125126cb27bfc4c69bd1dbd2fa9a
2017-05-24 17:02:13 +00:00

362 lines
21 KiB
Plaintext
Raw Blame History

Interop Working Group Scoring Worksheet
========================================
This file is used to maintain capabilities scoring.
It is considered "working materials" under the 2016A process and
is not an authoritative, Board-approved source of information
as to which capabilities are required for a particular Guideline.
Rather, it is a point-in-time listing used by Interop
Working Group members while determining which capabilities may be
included in a future Guideline. Each line takes the following form:
Capability Name: [Widely Deployed,Used By Tools,Used by Clients],
[TC Future Direction, Complete, Stable],
[Discoverable, Doc'd, Required in Last Release],
[Foundational, Atomic, Proximity],
[Non-Admin] [Total]
Possible scores: 0, 1, ? (not scored yet or needs discussion)
The Total score is calculated by multiplying the score for each
Criteria by the weight assigned to that Criteria in the Guideline
JSON file. It is not necessary to calculate the score by hand:
use the tabulate_scores.py script found in this directory. The
script will also append an asterisk (*) to the end of the line if
the total score is greater than or equal to the cutoff_score listed
in the Guideline JSON file, indicating that the Capability
has scored high enough to merit inclusion in the Guideline.
Networking
----------
network-floating-ips-associate: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-create: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-update: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-floating-ips-delete: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
network-l2-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-external: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-port: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l2-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-convert: [0,0,0] [1,0,0] [0,0,0] [0,0,0] [1] [11]
network-l3-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-router: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-l3-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
networks-list-api-versions: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
network-security-groups-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-security-groups-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
network-extensions-provider-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
network-extensions-provider-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
networks-subnet-pools-CRUD: [1,0,1] [1,1,1] [1,1,0] [0,1,1] [1] [76]*
Notes:
* DVR-related testcases (if any) will not be added until DVR gains wider
acceptance and isn't driver-specific.
* IPv6 is currently not widely deployed, so v6 versions of these tests need
not be added at this time. Vendor feedback on v6 support is requested,
though it is understood that IPv6 capabilities may be driver-dependent.
* A few folks have asked about provider network extensions as they are
frequently used for high performance networks and a simplified networking
model that provides for use cases similar to nova-net. However with default
policy settings, using provider networks requires admin access and therefore
is disqualified. I've listed them here anyway just to show that they were
considered but can't be added to the Guideline.
* Based on feedback from earlier patchsets, capabilities will be grouped
together where they had previously been listed separately but are adjacent.
For example, there are several tests for network creation, but only one or
few for deletion, listing, and updating. Combining these into a single
"CRUD" group reduces the chance the chance of running into problems with
the Interop Working Group's 2015A process (specifically section
C3(4)) in cases where there is only a single test for a capability and the
test needs to be flagged on account of bugs or other valid reasons. The
combined groups are:
* networks-floating-ips-CRUD-and-associate
* networks-floating-ips-associate
* networks-floating-ips-create
* networks-floating-ips-update
* networks-floating-ips-delete
* networks-l2-CRUD
* networks-l2-create
* networks-l2-delete
* networks-l2-external
* networks-l2-list
* networks-l2-port
* networks-l2-show
* networks-l2-update
* networks-l3-CRUD
* networks-l3-add
* networks-l3-create
* networks-l3-delete
* networks-l3-show
* networks-l3-update
* networks-l3-router
* networks-security-groups-CRUD
* networks-security-groups-create
* networks-security-groups-delete
* networks-security-groups-router
* networks-security-groups-show
* networks-security-groups-update
* Subnet pools were deemed not ready in 2016.08 but a good candidate for
the future. 2017.01 seems like the right time to make them advisory.
Subnet pools were introduced in Kilo but had incomplete support in
Horizon at the time. A Heat resource for subnet pools was added
in Mitaka, and OpenStackClient added support late in the Mitaka cycle
as well (https://review.openstack.org/#/c/279918/). If we make subnet
pools advisory now, they could become required in 2017.08--in which
the earliest release in the Guideline will be Mitaka. Toolkits like
jClouds, Fog, and libcloud do not yet support subnet pools that I see.
Subnet pools will be used by Neutron's upcoming get-me-a-network
capability and Kuryr appears to have plans to use them as part of
Kubernets integration.
* Extended port attribute tests were not considered because the capability
generally requires admin privileges.
* tempest.api.network.test_ports.PortsTestJSON.test_create_update_port_with_second_ip
was considered but was removed because not all Neutron plugins support
more than one IP per port (and some providers limit the number of
IP's per port for security reasons, e.g. spoofguard).
* The new trunk port API extension for the vlan-aware VM's feature may be
interesting to consider in the future, but didn't land until Newton and is
not yet sufficiently well adopted.
Compute
-------
compute-availability-zones-list: [1,1,1] [1,1,1] [1,1,0] [0,1,1] [1] [82]*
compute-flavors-list: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
compute-images-create: [0,1,1] [0,1,1] [1,1,1] [0,1,1] [1] [72]
compute-instance-actions-get: [1,1,1] [1,1,1] [1,0,0] [1,1,0] [1] [75]*
compute-instance-actions-list: [1,1,1] [1,1,1] [1,0,0] [1,1,0] [1] [75]*
compute-keypairs-create: [1,1,1] [1,1,1] [1,1,1] [0,1,0] [1] [83]*
compute-list-api-versions: [1,0,0] [1,1,1] [1,0,1] [1,1,1] [1] [76]*
compute-quotas-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-host: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-invalid: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-lock: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-metadata-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-metadata-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-metadata-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-metadata-set: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-metadata-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-name: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-reboot: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-rebuild: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-resize: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-stop: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-suspend-resume: [0,1,1] [1,0,1] [1,1,0] [0,1,1] [1] [66]
compute-servers-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-servers-verify: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
compute-volume-attach: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
Notes:
* compute-servers-suspend-resume is not widely supported by
different hypervisors, and is not recommended as a capability.
* There has been some discussion that this capability is not widely
deployed [1]. However it appears that in most cases the API either:
* Is present and works, but only if a non-SSL endpoint is exposed.
SSL endpoints may have issues with redirects (which may actually
merely be configuration issues).
* Is accidentally blocked due to load balancer or API gateway
misconfiguration which is being remedied by affected providers.
Taking those situations into account, it is believed that it's actually
safe to call this "widely deployed".
* I'm having trouble finding evidence that tools actually support this
API (refer to [2] for information about what constitutes "tools").
Unlike most of the CRUD-like API's, this one lists information about
the API versions available instead of on end user actionable resources
(such as instances or security groups), and the need to do so is newish
with the advent of microversioning. Hence, I think many tools may not
actually support this yet. Please feel free to provide evidence to the
contrary; I'd love to be proven wrong.
* Similarly, I'm having trouble finding evidence that clients [3] like
jClouds or Fog support this API (yet?) as they tend to focus on actions.
Please feel free to provide evidence to the contrary; I'd love to be
proven wrong.
* I'm tentatively scoring this API as Proximate [4]. Generally this applies
to CRUD operations (e.g. the delete API is considered proximate to the
create API). However I'm providing a 1 here on the grounds that due
to the shift to microversioning in the Nova API, this will be somewhat
proximate to everything going forward.
* The Interop Working Group is considering this capability for inclusion
in it's projected 2016.01 Guideline even though the test for it wasn't
developed until (narrowly) outside the normal window for identifying
new Capabilities on account of strong indications from the community
that such a Capability is important for API microversioning going
forward [5].
[1] https://bugs.launchpad.net/python-novaclient/+bug/1491579
[2] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n44
[3] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n48
[4] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n90
[5] http://eavesdrop.openstack.org/meetings/defcore_flag_14/2015/defcore_flag_14.2015-09-09-15.03.log.html#l-13
Image
-----
images-v1-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-index: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-list: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-register: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-add: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-get: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-shared-images-remove: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v1-update: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
images-v2-remove: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-update: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]*
images-v2-share: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
images-v2-import: [0,0,0] [1,0,0] [0,0,0] [1,0,1] [1] [28]
images-v2-list: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]*
images-v2-delete: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]*
images-v2-get: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]*
Notes:
* Image creation is captured under the register operation.
* No public image tests have explicit support for task API.
* Scoring for v1 remains in place, but v2 is preferred interop
standard (as reflected in worksheet).
* images-v2-share inherently requires more than one credential,
and currently out of scope for required capabilities. Additionally,
this capability may not be exposed at all. For more detail on why,
see https://wiki.openstack.org/wiki/OSSN/OSSN-0065#Recommended_Actions
* images-v2-remove has no test as of August, 2016. This was most recently
defined in the 2016.01 guideline.
* images-v2-import capability not delivered in Ocata.
Volume
----------
volumes-list-api-versions: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
volumes-v2-create-delete: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-attach-detach: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-snapshot-create-delete: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-get: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-list: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-update: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]*
volumes-v2-copy-image-to-volume: [1,0,0] [0,1,1] [1,1,1] [1,1,1] [1] [73]
volumes-v2-clone: [1,0,1] [0,1,1] [1,1,1] [1,1,1] [1] [83]*
volumes-v2-availability-zones: [1,0,0] [0,1,1] [1,1,1] [1,1,0] [1] [65]
volumes-v2-extensions: [1,0,0] [0,1,1] [1,1,1] [1,0,0] [1] [59]
volumes-v2-metadata: [1,0,1] [0,1,1] [1,1,1] [1,1,0] [1] [75]*
volumes-v2-reserve: [1,0,0] [0,1,1] [1,1,1] [1,1,0] [1] [65]
volumes-v2-readonly: [1,0,1] [0,1,1] [1,1,1] [1,1,0] [1] [75]*
volumes-v2-upload: [1,0,1] [0,1,1] [1,1,0] [1,1,0] [1] [66]
volumes-v3-create-delete: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-attach-detach: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-snapshot-create-delete: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-get: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-list: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-update: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-copy-image-to-volume: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-clone: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
volumes-v3-availability-zones: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
volumes-v3-extensions: [1,0,1] [0,1,1] [1,1,0] [1,0,0] [1] [60]
volumes-v3-metadata: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
volumes-v3-reserve: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
volumes-v3-readonly: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
volumes-v3-upload: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
Notes:
* 2017.01
* volumes-v3 uses microversioning and is compatible with v2 including tests
therefore v3 will be included as advisory to ensure that the community is
aware of this transitory state between v2/v3
* volumes-list-api-versions is implemented but currently does not have any
tests inside tempest
* volumes-v2-qos is an admin-only capability
* test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use
was previously put in advisory status, but was dropped due to the
fact that the force flag necessary to create a snapshot on an in-use
volume wasn't added until Liberty.
* 2017.08
* All v2 APIs are being marked down for TC Future direction because of the
deprecation of the v2 API in favor of the backward compatible v3
Identity
--------
identity-v3-api-discovery: [1,0,1] [1,1,1] [1,1,1] [1,1,1] [1] [94]*
identity-v3-catalog: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
identity-v3-list-projects: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
identity-v3-list-groups: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
identity-v3-validate-token: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
Notes:
* identity-v3-catalog is returned when the api for
identity-v3-tokens-create is called (GET /v3/auth/tokens). It is
important to consider it because end users may be relying on this
catalog for their apps (even though there are other API calls that
also show the catalog such as GET /v3/auth/catalog). There is one test
available for this capability but it is in the admin part of the test
suite, so not yet tested for non-admin users. Even though it scores enough
to be included as advisory, we cannot do this due to lack of non-admin
test case.
* identity-v3-list-projects and identity-v3-list-groups didn't have usable
tests in the past, but one was added for identity-v3-list-projects last year.
Providers like Fog.io
now actually use the /v3/users/{user_id}/[projects|groups] API's:
https://git.io/vX9S6
https://git.io/vX9SP
* identity-v3-change-password was considered here but it's applicability is
a bit hard to gauge: many systems using third-party authentication (such as
an LDAP/AD server, an external oauth system, etc) require password changes
to be done on the backend system. It probably needs further study to see
if it's really interoperable, but it seems unlikely at this point (I also
don't see it being supported by many external tools, etc).
* identity-v3-validate-token A given user can validate its own token. An
admin user is able to validate any token. This is enought for capability to
be considered non admin.
At the time of scoring, there is no non-admin test case in Tempest. Patch
https://review.openstack.org/#/c/467493 will add the test case but due to
timing, capability won't be added in this cycle - not until TC is available.
Object Store
------------
objectstore-object-copy: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-temp-url-put: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-object-versioned: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-temp-url-get: [1,1,1] [1,1,1] [1,1,1] [1,1,0] [1] [92]*
objectstore-account-quotas: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-account-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-acl: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-quotas: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-bulk-operations: [1,0,0] [1,1,1] [1,1,0] [0,1,0] [1] [58]
objectstore-info-request: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
objectstore-container-metadata: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
objectstore-staticweb: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
objectstore-crossdomain: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
objectstore-healthcheck: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
Notes:
objectstore-info-request is a new capability through re-orginization. The test
it uses is currently under "objectstore-account-list". Re-org as per PTL
request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L87
objectstore-object-upload removed and test moved to objectstore-object-create as per PTL
request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L209
objectstore-object-put renamed to objectstore-temp-url-put as per PTL request:
https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L251
objectstore-container-metadata used in Fog:
https://github.com/fog/fog-openstack/blob/master/docs/storage.md#additional-parameters
Also in jClouds: https://jclouds.apache.org/guides/openstack/#swift
View Git Blame Copy Permalink