openinfra/interop
Code Issues Proposed changes
ddedcf10d5
interop/doc/source/process/2021A.rst
Arkady Kanevsky 0e3a1c3149 Update process to reflect that human-readable format of the guidelines 
is optional but encouraged.

Change-Id: I0953b2975ad358a098def003b46dcc0d584ff82c
2021-07-06 10:41:58 -05:00

18 KiB
Raw Blame History

OpenStack Interop Working Group Process 2021A

Status

Approved #June 29, 2021 OIF board meeting

Replaces

2017A

This document describes the Interop Working Group's working process as required by the OpenStack bylaws (see https://www.openstack.org/legal/bylaws-of-the-openstack-foundation/) and approved by the OpenStack Technical Committee and Open Infrastructure Foundation Board.

Expected Time line:

Time Frame Activities Lead By
-1 month Draft of next guidelines. Interop WG
Review next guidelines Community

+ PTG +--------------------------------------+---------------------+ | | Review status | Interop WG | +------------+--------------------------------------+---------------------+ | +1 month | Update draft of guidelines | Interop WG | +------------+--------------------------------------+---------------------+ | +2 months | Testing draft guidelines | Refstack, Vendors | +------------+--------------------------------------+---------------------+ | +3 months | Approve Guidance | approval_committee | +------------+--------------------------------------+---------------------+

Note: Time line is aligned with releases schedules. See https://releases.openstack.org/xena/schedule.html as example of Xena schedule. Draft is done during the release for which guideline is being developed, while its completion is done during next release schedule. The Interop Working Group may accelerate the process to correct errors and omissions.

Process Definition

The Guideline process has four primary phases: Draft, Review, Validation and Approval. - During the Draft phase (A), the Interop Working Group creates new draft of guideline with input from the community with updated coverage, scores of the components, and other details. - During the Review phase (B), guidelines for each project are reviewed during the PTG with that project, input from refstack community, and general feedback from the community and vendors. - During the Validation phase (C) Guidelines go through and implementation and testing cycle. - Finally, Approval by approval_committee

The Open Infrastructure Board of Directors is not involved in the guideline process and review, and approve only changes to the Interop WG process.

This section provides specific rules and structure for each phase.

Guidelines Draft Phase (A)

Starting: S-1

A1. New Guidelines Start From Previous Guidelines

  1. New Guidelines start from the previously approved Guidelines document.
  2. New Guidelines are given the preliminary name of "next.json".
  3. For each of the new Guidelines add-on component there is a separate "next.json" document.

A2. Community Groups Tests for Capabilities

  1. The Interop Working Group coordinates community activities with the Technical Leadership to revise the capabilities based on current technical needs and functionality.
  2. Capabilities must be a subset of the OpenStack Technical Committee Approved Release as determined by the Board of Directors (see bylaws of the Foundation, section 4.1(b)(iii)).
  3. Groupings may change between iterations.
  4. Tests must have unique identifiers that are durable across releases and changes in grouping.
  5. Tests must be under OpenStack Technical Committee governance.
  6. The Interop Working Group will provide the test groupings in JSON format for scoring.
  7. The Interop Working Group optionally will provide a human-readable summary of the Guideline generated from the JSON version.
A3. Interop Working Group Collects Recommendations for

Designated Sections

1. Designated Sections will not be removed without being deprecated in the

previous Guideline.

2. Designated Sections will not be added without being advisory in the

previous Guideline.

3. Designated Sections will not be added or be made advisory unless the

corresponding code base is a subset of the OpenStack Technical Committee Approved Release as determined by the Board of Directors (see bylaws of the Foundation, section 4.1(b)(iii)).

4. TC & PTLs may, but is not required to, assist the

Interop Working Group with defining advisory Designated Sections for projects that have advisory or required Capabilities.

5. Designated Sections may be sufficiently defined for Guidelines using

general descriptions.

6. The Interop Working Group will present A3.4 descriptions to

approval_committee for approval.

7. TC & PTLs may, but is not required to, provide more specific

details describing the Designated Sections for a project.

  1. Designated Sections will be included in the JSON Guideline.

A4. Interop Working Group identifies required capabilities

  1. The Interop Working Group uses Board approved scoring criteria scoring criteria to evaluate Capabilities.
  2. The Interop Working Group needs Board approval to change scoring criteria.
  3. Scoring criteria factors or weights cannot change after Draft is published.
  4. The Interop Working Group identifies a cut-off score for determining that a Capability is required.
  5. Capabilities will not be removed without being deprecated in the previous Guideline.
  6. Capabilities will not be added without being advisory in the previous Guideline.
  7. For the "widely deployed" adoption criteria, the size of the pool being considered will match the scope of the community being considered. Capabilities will be evaluated based on their use in their Component. Components will be evaluated based on their use in the Platform.
A5. The Interop Working Group recommends OpenStack Components and OpenStack Platform

Scope

1. The Interop Working Group recommends Capabilities to include

in each OpenStack Component.

2. The Interop Working Group recommends which Components are required for

the OpenStack Powered Platform.

3. The Interop Working Group will apply the approved scoring criteria to

evaluate if a component should be included in the Platform (see A4).

A6. Additional Capabilities and Tests

  1. The Interop Working Group will work with the OpenStack community to define new Capabilities.
  2. Test grouping for new Capabilities will be included in the Interop Working Group documents.
  3. The Interop Working Group will publish a list of missing Capabilities and Capabilities with inadequate test coverage.
  4. Interop Working Group in conjuction with TC and Foundation Marketplace product owner can propose additional Add-on component guidelines.
  5. Each Add-on is provided as a seperate additional guideline, that follows the same process as "OpenStack powered" ones as defined by this document.

A7. The Interop Working Group creates recommendation for Draft.

  1. The Interop Working Group coordinates activities to create the Guideline draft.
  2. The Interop Working Group may choose to ignore recommendations with documented justification.

Guidelines Review Phase (B)

Starting: PTG

B1. All Reference Artifacts are reviewed via Gerrit

  1. Draft Guideline
  2. Designated sections
  3. Test-Capability groupings
  4. Flagged Test List
  5. Capability Scoring criteria and weights
  6. May not be in Gerrit: Working materials (spreadsheets, etc)

B2. Presentation of Draft Guidelines for Review

  1. The Interop Working Group will distribute Draft Guidelines to the OpenStack community for review.
  2. A link to the Gerrit document must be provided with the review materials.
  3. The draft Guidelines will consist of single document for "OpenStack-Powered" Logo and separate document for each of the add-ons components.

B3. Changes to Guideline made by Gerrit Review Process

  1. Guidelines proposal and review must go through Gerrit.
  2. The Interop Working Group will proxy for users who do not participate in the Gerrit process with attribution.
B4. For Gerrit reviews, Interop Working Group Co-Chairs act as

core reviewers.

  1. Interop Working Group Co-Chairs serve as "core" reviewers (+2).
2. Requests for changes must be submitted as patches by the requesting

party.

3. Interop Working Group members may proxy change requests as

long as the requesting party is explicitly acknowledged.

4. All Interop Working Group Co-Chair will be elected by the

Interop Working Group. Election quorum is composed of attendees present during the election meeting.

  1. Additional core reviewers (+2) can be appointed by Co-Chairs.
  2. Election meetings must be posted at least one meeting prior.
B5. Projects Interlocks
  1. For the PTG, the Interop Working Group requests a meeting with each project under the "OpenStack Powered" guideline and each of the add-on guidelines.
  2. Project meeting covers a review of the current guidelines for the projects, any changes/addition/deprecation/removal of APIs. Tests must be available in repositories under OpenStack TC governance. The tests cannot be in repositories outside of list in the openstack governance repository: https://opendev.org/openstack/governance/src/branch/master/reference/projects.yaml
  3. Project meeting covers any test result for specific drivers of that project to ensure consistent functionality coverage for all required and designated interoperability functionality of the project.
B6. Draft Guidelines update
  1. Following the PTG meeting the Interop Working Group updates draft Guidelines using feedback from each of the projects.
  2. Ensure any new or modified guidelines must have the corresponding test(s) in Tempest or Tempest plugins, and Refstack wrapper include it.
  3. For any new add-on program Refstack need to be able to collect submissions for marketplace.
  4. For any new add-on programs Foundation Marketplace product owner need to prepare ability to issue, record and administer new add-on Logo under Marketplace program.

Validation (C)

Starting: S and continues until S+2

C0. RefStack validation
  1. Refstack makes necessary changes to Page
  2. Refstack makes necessary changes to handle new guidelines.
  3. Refstack representative share test results of new guidelines on default platform with Interop Working Group.
  4. Refstack flags any tests that do not pass on the default platform.

C1. Vendor Self-Tests

  1. Vendors are responsible for executing tests identified by the Interop Working Group.
  2. The Interop Working Group may, but is not required to, provide tooling for running tests through Refstack.
  3. The Interop Working Group may, but is not required to, define a required reporting format.
  4. Self-test results may be published by Vendors in advance of Open Infrastructure Foundation Marketplace manager review, but must be clearly labeled as "Unofficial Results - Not Yet Accepted By The Open Infrastructure Foundation".
  5. Vendors who publish self-tests MUST provide them in the same format that would be submitted to the Open Infrastructure Foundation but MAY provide additional formats if they choose to do so.
  6. Self-test results cannot be used as proof of compliance.

C2. Vendor submits results to Foundation for review

  1. The Open Infrastructure Foundation Marketplace manager determines the acceptable format for submissions.
  2. The Open Infrastructure Foundation Marketplace manager has final authority to determine if Vendor meets criteria.
  3. The Open Infrastructure Foundation Marketplace manager will provide a review of the results within 30 days.
  4. Vendors can submit results for "OpenStack Powered" Logo and any of the add-on programs together or separately.
  5. The Open Infrastructure Foundation Marketplace manager can provide review of the results for all vendor submissions together or separately for each Logo.

C3. Vendor Grievance Process

  1. Vendors may raise concerns with specific tests to the Interop Working Group.
  2. The Interop Working Group may choose to remove tests from a Guideline (known as flagging).
  3. The Interop Working Group will acknowledge vendor requests to flag tests within 30 days.

C4. Results of Vendor Self-Tests will be open

  1. The Open Infrastructure Foundation Marketplace manager will make the final results of approved vendors available to the community.
  2. The Open Infrastructure Foundation Marketplace manager will not publish incomplete or unapproved results.
  3. Only "pass" results will be reported. Skipped and failed results will be omitted from the reports.
  4. Reports will include individual test results, not just Capability scoring.
  5. Vendors are required to submit a description of the system and configuration used to achieve the results.

C5. API Usage Data Report

  1. The Open Infrastructure Foundation Marketplace manager will provide the Interop Working Group with an open report about API usage based on self-tests based on Refstack submitted results.
  2. To the extent the data is available, Capabilities beyond the Interoperability Guideline list will be included in the report.

C6. Only Two Approved Guidelines at a time:

  1. Vendors seeking validation are limited to using the two latest approved Guidelines.
  2. Since past validations are respected, older Guidelines will be maintained as superseded for historical reference.
  3. Guideline status progresses as follows:
draft

initial work, pre-PTG discussion material

review

as presented at the PTG (S) for community review

approved

Committee approved, one of the two official guidelines

superseded

Committee approved, now superseded by two latest guidelines

Guideline Approval (D)

Starting: S+3 or earlier

D0. Approval Committee

  1. Approval Committee consists of representatives from 4 bodies
  1. Interop Working Group Co-chairs are approval authority of this committee.
  2. Refstack core member or repesentative(s) as advisory member
  3. Open Infrastructure Foundation Marketplace product owner as advisory member
  4. OpenStack TC representative(s) as advisory member

D1. Committee will review and approve Interoperability Guidelines from draft

  1. Interop Working Group must approve the proposed guidelines.
  2. Foundation staff member(s) and OpenStack TC member(s) vote in advisory capacity but do not have veto power.
  3. The approval is done through formal vote and its results recorded with the date of Approval.
  4. Upon approval guideline document is marked as approved with the date of approval.
  5. approval_committee members may request delay of the formal vote till foundation staff and/or Refstack is capable of handling new guidelines.
  6. Delay of the vote can be resolved by at most 1 month prior to next PTG meeting, which is the start of work on the new guidelines.
  7. Guidelines are set at the Platform, Component and Capability level only.
  8. The Interop Working Group will submit the summary of Capabilities (see section A2[6]) to the Committee for approval. The human-readable format of the summary is encouraged but optional.
  9. By voting to approve the summary, the Committee delegates responsibility for maintaining test groupings to the Interop Working Group subject to the limitations described in section D2.
  10. Guidelines only apply to the identified releases (a.k.a. release tags).
  11. The Add-on Guidelines are set for OpenStack projects in addition to "OpenStack Powered" ones.
  12. All guidelines follow the same review and approval process irrespective if they are "OpenStack Powered" or "OpenStack Add-on" guidelines.

D2. Interop Working Group has authority on test categorization

  1. The Interop Working Group can add flagged tests before and after Guideline approval.
  2. The Interop Working Group cannot add additional Tests to Capability mappings after approval.
  3. The Interop Working Group maintains the test to Capability mappings in the JSON representation.

D3. Designated Sections only enforced for projects with required Capabilities

  1. Designated Sections may be defined for any project.
  2. Designated Sections apply to the releases (a.k.a. release tags) identified in the Guideline.
  3. Designated Sections will be included in the JSON Capabilities file to ensure a single source of identification.

D4. Guidelines are named based on the date of Board approval

  1. Naming pattern will be: 4-digit year, dot (period), and 2-digit month.

Process Change

E1. Process Draft
  1. Any process change follows the process of draft, review and approval.
  2. Any process changes are handled thru gerrit process.
  3. Proposed changes submitted to Gerrit for review by Interop Working Group as a draft document.
  4. Interop Working Group adds all Committee members for review of the draft.
  5. Once that draft is approved, Interop Working Group co-chairs present it to Open Infrastructure Board for approval.
  6. Once Board approved the changes the new process is marked as approved and is linked from Interop WG wiki page.

Functional Information

Format

RestructuredText

Layout

1.0