Added keystone scoring for Mitaka release cycle.

Scored an existing advisory capability (identity-v3-api-discovery) and
made it required in next.json.

Scored 3 new capabilities for which I couldn't find tests, adding
them however for discussion. One of the new capabilities,
identity-v3-catalog refers to the catalog that is returned when calling
the current required capability identity-v3-tokens-create. Users may be
relying on the catalog to be there, so I think it is worth discussing.

identity-v3-list-projects and identity-v3-list-groups are worth
discussing and consider for addition of new tests.

The rationale for this scoring as well as input from keystone's PTL can
be found in working_materials/keystone_capabilities_info.csv.

Change-Id: Id444f5e982f2e81f140e285c305e9c322f5b9f42

next.json

@@ -17,6 +17,7 @@
     "compute": {
       "required": [
         "identity-v3-tokens-create",
+        "identity-v3-api-discovery",
         "compute-auth-create",
         "compute-auth-get",
         "compute-auth-set",
@@ -81,8 +82,7 @@
         "volumes-v2-metadata",
         "volumes-v2-transfer",
         "volumes-v2-reserve",
-        "volumes-v2-readonly",
-        "identity-v3-api-discovery"
+        "volumes-v2-readonly"
       ],
       "deprecated": [
         "identity-v2-tokens-create",
@@ -1366,6 +1366,7 @@
           "atomic",
           "complete",
           "deployed",
+	  "clients",
           "discover",
           "doc",
           "foundation",
@@ -1376,7 +1377,7 @@
         "admin": false,
         "description": "API discovery operations within the Identity v3 API",
         "project": "keystone",
-        "required-since": "",
+        "required-since": "2016.08",
         "tests": {
             "tempest.api.identity.v3.TestApiDiscovery.test_api_version_resources": {
                 "idempotent_id": "b9232f5e-d9e5-4d97-b96c-28d3db4de1bd"

working_materials/keystone_capabilities_info.csv

@@ -0,0 +1,32 @@
+Capability,Program,Status,Method,Endpoint,Test available?,interop relevant?,PTL Comments,From Defcore Discussion,Scorer Comments,
+identity-v3-tokens-create,platform/compute/object,required,POST,/v3/auth/tokens,1,yes,The returned token value is in the X-Auth-Token header,stay?,tempest.api.identity.v3.test_tokens{test_create_token},
+identity-v3-api-discovery,platform/compute,advisory,,,3,yes,,make required,"tempest.api.identity.v3.test_api_discovery{test_api_version_resources, test_api_media_types, test_api_version_statuses}",
+,,,,,,,,,,
+identity-v2-list-versions,,,GET,/,1,yes,,soon to be deprecated,,
+identity-v2-show-version,,,GET,/v2.0,1,yes,,soon to be deprecated,,
+identity-v2-token-generation,,,POST,/v2.0/tokens,1,yes,,soon to be deprecated,,
+identity-v2-tenants,,,GET,/v2.0/tenants,1,yes,,"is this an admin call? if so, not a candidate",,
+identity-v2-list-extensions,,,GET,/v2.0/extensions,,,,soon to be deprecated,,
+identity-v2-show-extension,,,GET,/v2.0/extensions/{alias},,,,soon to be deprecated,,
+,,,,,,,,,,
+identity-v3-create-ec2-credentials,,,POST,/v3/credentials,1,yes,,Should we make ec2 compatibility required? unclear,,
+identity-v3-list-ec2-credentials,,,GET,/v3/credentials,1,yes,,Should we make ec2 compatibility required? unclear,,
+identity-v3-show-ec2-credentials,,,GET,/v3/credentials/{credential_id},1,yes,,Should we make ec2 compatibility required? unclear,,
+identity-v3-delete-ec2-credentials,,,DELETE,/v3/credentials/{credential_id},1,yes,,Should we make ec2 compatibility required? unclear,,
+identity-v3-update-ec2-credentials,,,PATCH,/v3/credentials/{credential_id},,,,Should we make ec2 compatibility required? unclear,,
+identity-v3-catalog,(make sure it works on all supported releases),,,,,,returned with the token,,,
+identity-v3-password-update,,,POST,/v3/users/{user_id}/password,1,yes,,"
+Untestable without changing user's password, security risk. Also password policies are very particular to different companies, making a test that would pass on all is near impossible.",tempest.api.identity.v3.test_users{test_update_own_password},
+,,,,,,,,,,
+identity-v3-list-projects,platform/compute,,GET,/v3/users/{user_id}/projects,0,yes,,,no test available for this feature,
+identity-v3-list-groups,platform/compute,,GET,/v3/users/{user_id}/groups,0,yes,,,no test available for this feature,
+identity-v3-get-project,platform/compute,,GET,/v3/projects/{project_id},0,yes,,,admin required,
+identity-v3-list-roles,platform/compute,,GET,/v3/roles,0,no,,,admin required,
+identity-v3-get-role,platform/compute,,GET,/v3/roles/{role_id},,no,,,admin required,
+identity-v3-list-domains,platform/compute,,GET,/v3/domains,,no,,,admin required,
+identity-v3-get-domain,platform/compute,,GET,/v3/domains/{domain_id},,no,,,admin required,
+,,,,,,,,,,
+identity-v3-validate-token,platform/compute,,GET,/v3/auth/tokens,,yes,Token to be validated is passed in the X-Subject-Token header,,,"This sounds backwards to me, need to check with steve, shouldn't it be POST for validating and GET for getting a token??"
+identity-v3-revoke-token,platform/compute,,DELETE,/v3/auth/tokens,1,yes,Token to be revoked is passed in the X-Subject-Token header,keystone.keystone.tests.unit.test_revoke{test_revoke_by_user},,
+identity-v3-get-catalog,platform/compute/object,,GET,/v3/auth/catalog,0,yes,,,"couldn't find a test specific for this, there are some tests related in keystone.tests.unit.test_v3_auth.py",
+identity-v3-get-auth-projects,platform/compute,,GET,/v3/auth/projects,0,yes,,,"equivalent as far as I can tell to identity-v3-list-projects. couldn't find a test specific for this, there are some tests related in keystone.tests.unit.test_v3_auth.py",

working_materials/scoring.txt

@@ -184,4 +184,22 @@ volumes-v2-readonly:               [1,0,0] [1,1,1] [1,1,0] [1,1,0] [1] [67]
 
 Identity
 --------
-identity-v3-api-discovery:   [0,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [77]
+identity-v3-api-discovery:       [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]
+identity-v3-catalog:             [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]
+identity-v3-list-projects:       [1,0,1] [1,1,1] [1,1,0] [0,1,0] [1] [68]
+identity-v3-list-groups:         [1,0,1] [1,1,1] [1,1,0] [0,1,0] [1] [68]
+
+Notes:
+  * identity-v3-catalog is returned when the api for
+  identity-v3-tokens-create is called (GET /v3/auth/tokens). It is
+  important to consider it because end users may be relying on this
+  catalog for their apps (even though there are other API calls that
+  also show the catalog such as GET /v3/auth/catalog). There is one test
+  available for this capability but it is in the admin part of the test
+  suite, so not yet tested for non-admin users. Even though it scores enough
+  to be included as advisory, we cannot do this due to lack of non-admin
+  test case.
+  * identity-v3-list-projects and identity-v3-list-groups are here because
+  they deserve some visibility and some explicit test cases, which at the
+  moment they are lacking. It seems important for users to be able to
+  discriminate between projects and groups when running their apps.

working_materials/tabulated_scores.csv

@@ -58,4 +58,7 @@ volumes-v2-metadata,1,0,0,1,1,1,1,1,0,1,1,0,1,67
 volumes-v2-transfer,1,0,0,1,1,1,1,1,0,1,1,0,1,67
 volumes-v2-reserve,1,0,0,1,1,1,1,1,0,1,1,0,1,67
 volumes-v2-readonly,1,0,0,1,1,1,1,1,0,1,1,0,1,67
-identity-v3-api-discovery,0,0,1,1,1,1,1,1,0,1,1,1,1,77
+identity-v3-api-discovery,1,0,1,1,1,1,1,1,0,1,1,1,1,85
+identity-v3-catalog,1,0,1,1,1,1,1,1,0,1,1,1,1,85
+identity-v3-list-projects,1,0,1,1,1,1,1,1,0,0,1,0,1,68
+identity-v3-list-groups,1,0,1,1,1,1,1,1,0,0,1,0,1,68