From d4a9f68b8f4a19018dfd79b9b90be19c292fcfc6 Mon Sep 17 00:00:00 2001 From: robhirschfeld Date: Thu, 4 Jun 2015 23:17:28 -0500 Subject: [PATCH] Create a preliminary flag guidelines list This was discussed at the Flag.2 meeting [1] where the included initial suggestions were made. The intent is to start a discussion using this patch about the flagging process where we can establish clear and transparent criteria. [1] https://etherpad.openstack.org/p/DefCoreFlag.2 Co-authored-by: Rob Hirschfeld Co-authored-by: Mark T. Voelker Co-authored-by: Jim Meyer Change-Id: Ieacf8f8b4307658392a21f57cb1a2031e3573383 --- HACKING.rst | 66 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/HACKING.rst b/HACKING.rst index 321c041b..b09e4a6d 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -3,7 +3,12 @@ DefCore Style Commandments - Step 1: Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/ -- Step 2: Read on +- Step 2: Read the following DefCore process documents in the following recommended order: + + - `Core Definition `_ + - `OpenStack DefCore Process 2015A `_ + +- Step 3: Read on DefCore Specific Commandments ----------------------------- @@ -18,18 +23,18 @@ DefCore Specific Commandments in an approved capability list, add a "flagged" block to the test. Do _not_ remove it from the "tests" section. - [D302] If a Capability is found to not meet the `Core Criteria - <./process/CoreCriteria.rst>`_ after the Board has approved a Guideline, - the corresponding tests should have a "flagged" block added to the - the relevant tests in the "tests" section of the relevant Board-approved - Guidelines. The tests should also be removed from the "tests" section of - the .next.json file. - - See [D307] for details about the flagged block + `_ after the Board has approved + a Guideline, the corresponding tests should have a "flagged" block added + to the the relevant tests in the "tests" section of the relevant + Board-approved Guidelines. + + - See [D307] and [D309] for details about format requirements. + - See [D308] for conditions on also adding to the .next.json. + - [D303] Tests that are found to inadequately test the underlying Capability due to bugs or design flaws, should have a "flagged" block added to the section for the test in the "tests" section of the most recent Board-approved Guideline. - - See [D307] for details about the flagged block - - See [D308] for conditions on also adding to the .next.json. - [D304] Before the Board approves the capabilities listed in the .next.json file, a committee member will submit a patch that copies the .next.json file to an appropriately named new file, updates the @@ -54,3 +59,46 @@ DefCore Specific Commandments to be resolved before the next Guideline is submitted to tbe Board for approval, then matching entries should also be made in the .next.json Guideline. +- [D309] The "reason" field of the "flagged" section must begin with the + flag type. For example: + + ``"reason" : "[D400] The Foo test doesn't meet DefCore criteria because ..."`` + +- [D310] If you believe a test needs to be flagged but the reason for doing + so doesn't appear in the list below, you must do the following: + + #. Submit the test for flagging using [D404] as the flag type. Please also + provide the reason you believe this test needs to be flagged; see [D309] + above for details. [D404] indicates that you are uncertain about which + flag to use or believe that a new reason for considering flags needs to be + discussed. *[D404] is only a placeholder to facilitate this discussion.* + #. In that same proposed change, also submit a patch against the HACKING + file adding your proposed new flag to the list below. + #. If at all possible, please include a link to code and/or test runs which + demonstrate the reason a new flag type is needed. + #. The DefCore committee will discuss and consider the flagging proposal as + well as the proposed new reason. They may accept or decline either proposal. + + +DefCore Test Flagging Guidelines +-------------------------------- + +The DefCore Committee may "flag" tests to mark them as not required for a +given Guideline. There are different flag types; each flag type indicates a +fairly narrow category of reasons for flagging a given test. + +Valid reasons for flagging a test are limited to the following: + +- [D400] The test is for a Capability that fails to meet DefCore Criteria + as set out in the + `Core Criteria document `_. +- [D401] The test fails or is skipped due to a bug in the test and the bug is + accepted by the OpenStack project which maintains the test. +- [D402] The test fails or is skipped due to a bug in the code that provides + the Capability and the bug is accepted by the OpenStack project which + maintains the Capability. +- [D403] The test fails because other non-DefCore Capabilities are also + tested. +- [D404] Flag Not Found - Use this flag if none of the others fit. +- [D405] The test reflects an implementation choice that is not widely + deployed even if the Capability is widely deployed.