Interop Working Group Scoring Worksheet ======================================== This file is used to maintain capabilities scoring. It is considered "working materials" under the 2016A process and is not an authoritative, Board-approved source of information as to which capabilities are required for a particular Guideline. Rather, it is a point-in-time listing used by Interop Working Group members while determining which capabilities may be included in a future Guideline. Each line takes the following form: Capability Name: [Widely Deployed,Used By Tools,Used by Clients], [TC Future Direction, Complete, Stable], [Discoverable, Doc'd, Required in Last Release], [Foundational, Atomic, Proximity], [Non-Admin] [Total] Possible scores: 0, 1, ? (not scored yet or needs discussion) The Total score is calculated by multiplying the score for each Criteria by the weight assigned to that Criteria in the Guideline JSON file. It is not necessary to calculate the score by hand: use the tabulate_scores.py script found in this directory. The script will also append an asterisk (*) to the end of the line if the total score is greater than or equal to the cutoff_score listed in the Guideline JSON file, indicating that the Capability has scored high enough to merit inclusion in the Guideline. Networking ---------- network-floating-ips-associate: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]* network-floating-ips-create: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]* network-floating-ips-update: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]* network-floating-ips-delete: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]* network-l2-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-external: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-port: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l2-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-convert: [0,0,0] [1,0,0] [0,0,0] [0,0,0] [1] [11] network-l3-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-router: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-l3-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* networks-list-api-versions: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* network-security-groups-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-security-groups-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-security-groups-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-security-groups-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* network-extensions-provider-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0] network-extensions-provider-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0] network-extensions-provider-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0] network-extensions-provider-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0] networks-subnet-pools-CRUD: [1,0,1] [1,1,1] [1,1,0] [0,1,1] [1] [76]* Notes: * DVR-related testcases (if any) will not be added until DVR gains wider acceptance and isn't driver-specific. * IPv6 is currently not widely deployed, so v6 versions of these tests need not be added at this time. Vendor feedback on v6 support is requested, though it is understood that IPv6 capabilities may be driver-dependent. * A few folks have asked about provider network extensions as they are frequently used for high performance networks and a simplified networking model that provides for use cases similar to nova-net. However with default policy settings, using provider networks requires admin access and therefore is disqualified. I've listed them here anyway just to show that they were considered but can't be added to the Guideline. * Based on feedback from earlier patchsets, capabilities will be grouped together where they had previously been listed separately but are adjacent. For example, there are several tests for network creation, but only one or few for deletion, listing, and updating. Combining these into a single "CRUD" group reduces the chance the chance of running into problems with the Interop Working Group's 2015A process (specifically section C3(4)) in cases where there is only a single test for a capability and the test needs to be flagged on account of bugs or other valid reasons. The combined groups are: * networks-floating-ips-CRUD-and-associate * networks-floating-ips-associate * networks-floating-ips-create * networks-floating-ips-update * networks-floating-ips-delete * networks-l2-CRUD * networks-l2-create * networks-l2-delete * networks-l2-external * networks-l2-list * networks-l2-port * networks-l2-show * networks-l2-update * networks-l3-CRUD * networks-l3-add * networks-l3-create * networks-l3-delete * networks-l3-show * networks-l3-update * networks-l3-router * networks-security-groups-CRUD * networks-security-groups-create * networks-security-groups-delete * networks-security-groups-router * networks-security-groups-show * networks-security-groups-update * Subnet pools were deemed not ready in 2016.08 but a good candidate for the future. 2017.01 seems like the right time to make them advisory. Subnet pools were introduced in Kilo but had incomplete support in Horizon at the time. A Heat resource for subnet pools was added in Mitaka, and OpenStackClient added support late in the Mitaka cycle as well (https://review.openstack.org/#/c/279918/). If we make subnet pools advisory now, they could become required in 2017.08--in which the earliest release in the Guideline will be Mitaka. Toolkits like jClouds, Fog, and libcloud do not yet support subnet pools that I see. Subnet pools will be used by Neutron's upcoming get-me-a-network capability and Kuryr appears to have plans to use them as part of Kubernets integration. * Extended port attribute tests were not considered because the capability generally requires admin privileges. * tempest.api.network.test_ports.PortsTestJSON.test_create_update_port_with_second_ip was considered but was removed because not all Neutron plugins support more than one IP per port (and some providers limit the number of IP's per port for security reasons, e.g. spoofguard). * The new trunk port API extension for the vlan-aware VM's feature may be interesting to consider in the future, but didn't land until Newton and is not yet sufficiently well adopted. Compute ------- compute-availability-zones-list: [1,1,1] [1,1,1] [1,1,0] [0,1,1] [1] [82]* compute-flavors-list: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]* compute-images-create: [0,1,1] [0,1,1] [1,1,1] [0,1,1] [1] [72] compute-instance-actions-get: [1,1,1] [1,1,1] [1,0,0] [1,1,0] [1] [75]* compute-instance-actions-list: [1,1,1] [1,1,1] [1,0,0] [1,1,0] [1] [75]* compute-keypairs-create: [1,1,1] [1,1,1] [1,1,1] [0,1,0] [1] [83]* compute-list-api-versions: [1,0,0] [1,1,1] [1,0,1] [1,1,1] [1] [76]* compute-quotas-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-host: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-invalid: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-lock: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-metadata-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-metadata-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-metadata-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-metadata-set: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-metadata-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-name: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-reboot: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-rebuild: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-resize: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-stop: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-suspend-resume: [0,1,1] [1,0,1] [1,1,0] [0,1,1] [1] [66] compute-servers-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-servers-verify: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* compute-volume-attach: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* Notes: * compute-servers-suspend-resume is not widely supported by different hypervisors, and is not recommended as a capability. * There has been some discussion that this capability is not widely deployed [1]. However it appears that in most cases the API either: * Is present and works, but only if a non-SSL endpoint is exposed. SSL endpoints may have issues with redirects (which may actually merely be configuration issues). * Is accidentally blocked due to load balancer or API gateway misconfiguration which is being remedied by affected providers. Taking those situations into account, it is believed that it's actually safe to call this "widely deployed". * I'm having trouble finding evidence that tools actually support this API (refer to [2] for information about what constitutes "tools"). Unlike most of the CRUD-like API's, this one lists information about the API versions available instead of on end user actionable resources (such as instances or security groups), and the need to do so is newish with the advent of microversioning. Hence, I think many tools may not actually support this yet. Please feel free to provide evidence to the contrary; I'd love to be proven wrong. * Similarly, I'm having trouble finding evidence that clients [3] like jClouds or Fog support this API (yet?) as they tend to focus on actions. Please feel free to provide evidence to the contrary; I'd love to be proven wrong. * I'm tentatively scoring this API as Proximate [4]. Generally this applies to CRUD operations (e.g. the delete API is considered proximate to the create API). However I'm providing a 1 here on the grounds that due to the shift to microversioning in the Nova API, this will be somewhat proximate to everything going forward. * The Interop Working Group is considering this capability for inclusion in it's projected 2016.01 Guideline even though the test for it wasn't developed until (narrowly) outside the normal window for identifying new Capabilities on account of strong indications from the community that such a Capability is important for API microversioning going forward [5]. [1] https://bugs.launchpad.net/python-novaclient/+bug/1491579 [2] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n44 [3] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n48 [4] http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst#n90 [5] http://eavesdrop.openstack.org/meetings/defcore_flag_14/2015/defcore_flag_14.2015-09-09-15.03.log.html#l-13 Image ----- images-v1-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-index: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-list: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-register: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-shared-images-add: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-shared-images-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-shared-images-get: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-shared-images-remove: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v1-update: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58] images-v2-remove: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]* images-v2-update: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]* images-v2-share: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]* images-v2-import: [0,0,0] [1,0,0] [0,0,0] [1,0,1] [1] [28] images-v2-list: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]* images-v2-delete: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]* images-v2-get: [1,0,1] [1,1,1] [1,1,1] [1,0,1] [1] [88]* Notes: * Image creation is captured under the register operation. * No public image tests have explicit support for task API. * Scoring for v1 remains in place, but v2 is preferred interop standard (as reflected in worksheet). * images-v2-share inherently requires more than one credential, and currently out of scope for required capabilities. Additionally, this capability may not be exposed at all. For more detail on why, see https://wiki.openstack.org/wiki/OSSN/OSSN-0065#Recommended_Actions * images-v2-remove has no test as of August, 2016. This was most recently defined in the 2016.01 guideline. * images-v2-import capability not delivered in Ocata. Volume ---------- volumes-list-api-versions: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67] volumes-v2-create-delete: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-attach-detach: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-snapshot-create-delete: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-get: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-list: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-update: [1,1,1] [0,1,1] [1,1,1] [1,1,1] [1] [89]* volumes-v2-copy-image-to-volume: [1,0,0] [0,1,1] [1,1,1] [1,1,1] [1] [73] volumes-v2-clone: [1,0,1] [0,1,1] [1,1,1] [1,1,1] [1] [83]* volumes-v2-availability-zones: [1,0,0] [0,1,1] [1,1,1] [1,1,0] [1] [65] volumes-v2-extensions: [1,0,0] [0,1,1] [1,1,1] [1,0,0] [1] [59] volumes-v2-metadata: [1,0,1] [0,1,1] [1,1,1] [1,1,0] [1] [75]* volumes-v2-reserve: [1,0,0] [0,1,1] [1,1,1] [1,1,0] [1] [65] volumes-v2-readonly: [1,0,1] [0,1,1] [1,1,1] [1,1,0] [1] [75]* volumes-v2-upload: [1,0,1] [0,1,1] [1,1,0] [1,1,0] [1] [66] volumes-v3-create-delete: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-attach-detach: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-snapshot-create-delete: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-get: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-list: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-update: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-copy-image-to-volume: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-clone: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* volumes-v3-availability-zones: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]* volumes-v3-extensions: [1,0,1] [0,1,1] [1,1,0] [1,0,0] [1] [60] volumes-v3-metadata: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]* volumes-v3-reserve: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]* volumes-v3-readonly: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]* volumes-v3-upload: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]* Notes: * 2017.01 * volumes-v3 uses microversioning and is compatible with v2 including tests therefore v3 will be included as advisory to ensure that the community is aware of this transitory state between v2/v3 * volumes-list-api-versions is implemented but currently does not have any tests inside tempest * volumes-v2-qos is an admin-only capability * test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use was previously put in advisory status, but was dropped due to the fact that the force flag necessary to create a snapshot on an in-use volume wasn't added until Liberty. * 2017.08 * All v2 APIs are being marked down for TC Future direction because of the deprecation of the v2 API in favor of the backward compatible v3 Identity -------- identity-v3-api-discovery: [1,0,1] [1,1,1] [1,1,1] [1,1,1] [1] [94]* identity-v3-catalog: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]* identity-v3-list-projects: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]* identity-v3-list-groups: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]* identity-v3-validate-token: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]* Notes: * identity-v3-catalog is returned when the api for identity-v3-tokens-create is called (GET /v3/auth/tokens). It is important to consider it because end users may be relying on this catalog for their apps (even though there are other API calls that also show the catalog such as GET /v3/auth/catalog). There is one test available for this capability but it is in the admin part of the test suite, so not yet tested for non-admin users. Even though it scores enough to be included as advisory, we cannot do this due to lack of non-admin test case. * identity-v3-list-projects and identity-v3-list-groups didn't have usable tests in the past, but one was added for identity-v3-list-projects last year. Providers like Fog.io now actually use the /v3/users/{user_id}/[projects|groups] API's: https://git.io/vX9S6 https://git.io/vX9SP * identity-v3-change-password was considered here but it's applicability is a bit hard to gauge: many systems using third-party authentication (such as an LDAP/AD server, an external oauth system, etc) require password changes to be done on the backend system. It probably needs further study to see if it's really interoperable, but it seems unlikely at this point (I also don't see it being supported by many external tools, etc). * identity-v3-validate-token A given user can validate its own token. An admin user is able to validate any token. This is enought for capability to be considered non admin. At the time of scoring, there is no non-admin test case in Tempest. Patch https://review.openstack.org/#/c/467493 will add the test case but due to timing, capability won't be added in this cycle - not until TC is available. Object Store ------------ objectstore-object-copy: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-object-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-object-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-object-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-temp-url-put: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-object-versioned: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-temp-url-get: [1,1,1] [1,1,1] [1,1,1] [1,1,0] [1] [92]* objectstore-account-quotas: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-account-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-acl: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-quotas: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-bulk-operations: [1,0,0] [1,1,1] [1,1,0] [0,1,0] [1] [58] objectstore-info-request: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]* objectstore-container-metadata: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]* objectstore-staticweb: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58] objectstore-crossdomain: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58] objectstore-healthcheck: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58] Notes: objectstore-info-request is a new capability through re-orginization. The test it uses is currently under "objectstore-account-list". Re-org as per PTL request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L87 objectstore-object-upload removed and test moved to objectstore-object-create as per PTL request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L209 objectstore-object-put renamed to objectstore-temp-url-put as per PTL request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L251 objectstore-container-metadata used in Fog: https://github.com/fog/fog-openstack/blob/master/docs/storage.md#additional-parameters Also in jClouds: https://jclouds.apache.org/guides/openstack/#swift