docs/doc/source/contributor/development_process.rst

StarlingX Development Process

This section describes the StarlingX development process.

Introduction

The StarlingX development process is designed

• to enable project teams to work according to the Four Opens
• for work to be managed by the project's community members
• to specify a common software development methodology that enables the community to collaborate effectively.

The StarlingX community follows many of the practices (but not all!) of the OpenStack software development process. Differences between the StarlingX and OpenStack practices are highlighted below.

Project personas

The following personas are used to describe the development process. Some of these same terms are used in the project's governance document.

User

A user is someone who is building, operating, or maintaining a StarlingX cloud.

Developer

A developer is someone who submits a code, document, or test change to the StarlingX git repositories.

Reviewer

A reviewer is someone who submits review comments to StarlingX Gerrit reviews.

Core Reviewer

A core reviewer is a StarlingX reviewer who can approve code to be merged into the StarlingX git repositories.

Technical Lead / Project Lead

The StarlingX project is divided into a number of sub-projects. Each sub-project has a technical lead (TL) and a project lead (PL) who serve as leaders of that sub-project. Each sub-project has a team which can consist of users, developers and (core) reviewers.

Technical Steering Committee

The StarlingX Technical Steering Committee (TSC) members are the leaders of the StarlingX project.

Notifying the community

There are several points in the process where as a user or developer you will be asked to notify the community. You can do so by sending a message to the project's mailing list. Notifying the community both early and often will help the community engage with you.

Another good way to notify the community is to attend the weekly community call <https://wiki.openstack.org/wiki/Starlingx/ Meetings#7am_Pacific_-_Community_Call> and/or the sub-project calls for the area you are working on. Both are good places to ask questions, raise issues, and discuss technical topics.

Please be patient. StarlingX is a global project and the person with the knowledge to help you may be twelve timezones away from you.

Development process overview

The StarlingX development process includes the following basic steps:

• Decide what change you want to make. Many of the changes needed for StarlingX have already been documented in the project's bug backlog and in existing Stories. <https://storyboard.openstack.org/#!/ story/list?status=active&project_group_id=86> If you are reporting a new bug or requesting a new feature, please document the change in a bug report (for a defect in the code) or in a StoryBoard story (for new functionality) as described below in the Defect submission process or the Feature submission process.
• If you are working on a large change (like a new feature), need technical guidance, or would like to ask other community members to join you in the work, please notify the community that you are starting to work on a change.
• If you are working on a new feature, create a specification and follow the spec process. This step will allow you to collaborate with the community on the design of your change and helps to identify other teams and services within the StarlingX project that could be effected by your change.
• Implement the code change(s) on your local system, posting your changes to Gerrit early and often. Respond to the community's feedback and please notify the community if your changes do not get reviewed in a timely fashion.
• Test your changes on your local system. If needed, ask the test project members to help you run tests on their systems. It's a good idea to check with the sub-project's TL and PL, who can help you ensure your testing covers all needed areas. You should be testing your changes from the very beginning of your work and should include thorough and complete testing in your plan for your changes.
• Update the project's documentation if needed.
• Your change will be merged once the core reviewers for the repo(s) approve your change in Gerrit.

Complete descriptions for these steps are provided in the detailed descriptions below.

Submitting defect reports and feature requests

Any user or other StarlingX community member can request a change to the software. The process for requesting a change depends on whether the change is a fix to a bug, or a request for a new feature.

Defect submission process

The StarlingX project uses launchpad for tracking bugs.

To report a new bug, go to the launchpad link above and click Report a bug. You will be prompted to search for similar bugs which helps reduce duplicates in the database. Please use the StarlingX Bug Template (displayed in the bug window) when filing a new bug to help ensure that the team can quickly triage and fix the bug.

Capturing logs related to a defect is often required in order to root cause and fix the issue. Please use the StarlingX log collection tool to post collect log files and post a link to the logs within your Launchpad.

As the reporter of a bug, once you complete the template and fill in the information requested, you can submit the bug report. Please do not change the status of the bug. Leave it as "New". There is also no need to enter any tags. This will be done as part of the triage process.

After reporting a bug, please notify the community if your bug is urgent or impacting your operations.

Feature submission process

The StarlingX project uses the OpenDev StoryBoard tool to document new features for the project. You can use the tool to search for existing stories or to create new ones. To submit a new feature request for StarlingX please follow these steps.

1. In your browser navigate to the StarlingX StoryBoard project group.
2. Click on the Create New button on the top of the page and select "Story" from the pull down menu. A dialog box will appear.
3. In the "Title" field in the dialog box, enter in a name for the feature or a one sentence description.
4. In the "Description" field of the dialog box, enter in a description of what you want the feature to do and why you want it.
5. In the "Project" field in the dialog box, enter in the name of the StarlingX sub-project repository that contains the code to implement your feature. These names all start with "starlingx/" which you can type to have a drop down list shown of all sub-project repositories. If you don't know which repository to select, you can use "starlingx/integ".
6. Click Save Changes and your feature request is submitted.

After submitting a new feature request, please notify the community Your story will be reviewed according to the planning process described below. While anyone can submit a feature request to StarlingX, you can greatly increase the chance of the feature being implemented by joining the team(s) who are working on the feature and contributing to the effort.

Development process details

This section describes the StarlingX development process in detail.

Basic git workflow

The OpenStack Contributor Guide provides a good overview of the standard OpenStack process. The StarlingX process is very similar.

The steps described in the StarlingX Build Guide in the Developer environment setup section <https://docs.starlingx.io/developer_resources/ build_guide.html#development-environment-setup> must be completed before any of the steps described here.

Clone the desired StarlingX source repos using either:

git clone <a starlingX repo>

Or clone all of the repos using the repo tool as described in the StarlingX build guide:

repo init -u https://opendev.org/starlingx/manifest -m default.xml

The basic developer workflow looks like this:

cd <your local repo>
git pull # ensure your repo is up to date with the latest changes
git branch <branch name> # always work on a branch
# work on the change - edit the code, build it and test it
git add <the files you changed> # or git add -a
git commit -s
# Your commit message should include
   A) A short title
   B) a blank line
   C) a description of the change
   D) A blank line
   E) An optional tag for a story or launchpad issue number
      Closes-Bug: ######
      Partial-Bug: ######
      Related-Bug: ######
      Task: ######
      Story: ######
git review  # Post your code changes to Gerrit

This will post your change for community review and eventual approval by the core reviewers. If needed, you can respond to community feedback by posting an updated version of your change as follows:

git add <the files you changed>
git commit -a --amend
git review

Bug fix process

StarlingX follows the OpenStack project team guidelines for bug management, but with a few small differences as described below.

Bug triage

The stx-bugs team is responsible for triaging, assigning and tagging StarlingX bugs. The team includes the StarlingX project leads and delegates. One or more members of the stx-bugs team reviews the new bugs and adds the applicable sub-project tags. This allows each sub-project team to see their bug backlog.

The sub-project project lead (or delegate) then triages the bug further and adds a release tag based on severity and when they believe they can fix the bug. At this point, they also set the bug importance (if not already done) and the status is updated to "Triaged".

If an issue is minor and is deemed not gating for the next release, a release tag is not added. In this case, the bug importance should be set to "Low" to indicate that it does not gate any release. It is recommended that the project lead triaging the bug add a comment with the rationale.

For more information on the use of tags in StarlingX, see the Tags & Prefixes page.

Bug resolution

It is the responsibility of each sub-project team to manage their bug backlog.

Each project lead has the ability to assign bugs to members of the team. Team members can also assign bugs to themselves (but not to others).

When working on a bug, it is recommended that the developer sets the status to "In Progress".

By default, the reporter is subscribed to the bug and will receive email notifications when comments are added. You can use this to communicate with the reporter if you have questions or need clarification.

It is expected that the reporter responds by adding another comment to the bug in launchpad.

Fixing a bug shares some of the same process steps as the feature development process described below. In particular bug fixes require thorough testing before the fix is committed, as per the testing phase below.

All bug fixes must be fixed in master first. The release sub-project team may request that fixes be merged into older release branches at their discretion.

When the work to fix a bug is complete, the developer specifies the bug ID in their commit messages using Closes-Bug: <bug ID> so that Gerrit automatically marks the bug as "Fix Released" when the code is merged. See the StarlingX code submission guidelines.

During an active release RC period (prior to the official release), each sub-project team decides whether a reported bug gates the release as they are in the best position to articulate the impact. If gating, the bug must be tagged with the appropriate release tag. The developer is responsible for cherry picking the fix from the master branch to the release branch.

Similarly, the sub-project team decides if any bugs need to be cherry-picked to a released branch. Only critical or high impact issues will be cherry-picked.

A bug can also be marked as "Invalid" or "Won't Fix" based on further investigation. Notes must be added to the bug explaining the rationale. The bug should remain assigned to the developer who investigated the bug. Do not assign it back to the reporter. This makes it easier to find bugs you worked on.

Bug verification

Launchpad does not have a distinction between "Fix Resolved" and "Fix Verified". Once code merges in master, the bug is automatically updated to "Fix Released" and considered "Closed". This doesn't provide a way to query bugs that need to be explicitly retested by the reporter.

An optional tag (stx.retestneeded) will be used to track bugs that need explicit verification. The tag is added at the time the bug is triaged (or the reporter at the time the bug is created).

Once the bug is verified by the reporter, a note should be added to the bug by the reporter and the label will be removed by the stx-bugs team.

Recommended Launchpad display

It is recommended to customize your display in Launchpad to show the following information:

• Importance
• Status
• Number
• Assignee
• Tags

Feature development process

The feature development process takes place in a number of phases, each described below. These phases overlap each other in time and are used here to describe how the work is done and are not meant to be gates for forward progress.

Concept phase

The concept phase begins when a user or developer or other community member has an idea for a new feature for StarlingX. The idea needs to be documented by following the feature submission process as described above. Once the new StoryBoard story is submitted, the concept phase is complete.

Specification phase

New features for StarlingX require a specification ("spec") to be written for review and approval by the TSC. The spec is a key deliverable for the planning phase as the work can not be fully planned until it is understood and agreed by the community.

The PL for a sub-project can waive a spec for small changes, but should not do so for any change that impacts the project's UI, APIs or spans multiple StarlingX sub-projects. And note that it is possible to cause major changes to StarlingX by making a one line change in the right place, so the project team is advised to carefully review a change for its impact on the project and the need for broader review before approving a waiver for a spec.

StarlingX specs are stored in a git repo. To start a spec, clone the StarlingX spec repo:

git clone https://opendev.org/starlingx/specs

Please notify the community when you start working on a specification. Then read the file specs/instructions.rst which describes the process of submitting a spec for review.

The spec itself should be submitted for review by the community early and often. The spec will be reviewed and approved by the TSC members, who are the core reviewers for the specs repo. Once the spec is approved (merged), the spec phase is complete.

Planning phase

The planning phase is largely the responsibility of each StarlingX sub-project team, who maintain and manage a backlog of stories for the new features they would like to implement in their part of the project.

Each sub-project PL and TL share the responsibility to review new story submissions and work with their team members to prioritize and plan the work. Initial planning should include completing a specification (see above), breaking down the story into tasks, assigning the tasks to developers, and making an initial estimate as to which release the story should be targeted to. All of this planning should be done and reviewed in the team's regularly scheduled calls.

The Open Infrastructure Foundation holds a project teams gathering (PTG) every 6 months. The PTG is often scheduled close to the time of other Open Infrastructure events. Review and planning for new features are discussed as part of the StarlingX PTG meeting. The StarlingX release cycle is tied to the OpenStack release cycle, so planning features for the next release at the PTG is a great time for the community to come together and discuss the technical issues face to face.

Some features may span multiple StarlingX sub-projects. In that case, the sub-project PLs should consult with the TSC for how the work should be done early in the planning phase.

The project PL should notify the community as features work through the planning process, separately or in their meeting minutes.

Once the sub-project PL and TL agree that feature planning is complete, the planning phase is complete.

Implementation phase

The implementation phase can begin at any time in the process and includes the development of the software changes as well as the documentation changes and test cases identified in the specification.

Test cases can and should be developed in parallel with the code changes or ideally before the code changes. You can use test cases to reproduce the behavior of a defect, for instance, and then run the test cases with your fix to show that it really is fixed.

As your feature work nears the end your testing should take on a broader scope. For instance you may need to work with the test team to run their test suites. You should also update the release team, especially near release milestones, to make sure they are aware of the status of your work.

The Testing phase section contains additional details on the testing process to be followed.

See below for guidance on the testing that developers need to perform prior to code completion.

The implementation phase is complete once all required changes have been merged by the core reviewers.

Release phase

The StarlingX release sub-project has the overall responsibility to manage the delivery of StarlingX releases. The team tracks new features as they are planned and developed and can move content into or out of StarlingX releases. All but the smallest features will likely require close collaboration with the release team. See the StarlingX Release Process for more details on the release process.

Once a software feature is included in a StarlingX release, the release phase is complete.

Testing phase

Testing a newly developed fix or feature can be as challenging as developing the fix or feature itself. The effort and work items needed to complete the testing phase should be considered in the planning phase.

It is possible to cause major changes in the behavior of the StarlingX software with very small code changes so thorough testing is important to the stability of the code. We suggest that feature developers create a test plan in collaboration with the sub-project's PL and TL, the core reviewers and subject matter experts within the StarlingX community. Consultation with members of the project's Test team are also advised.

Test plans can cover one or more StarlingX configurations, define test cases focused on the functional changes made, and sometimes include sanity tests or full regression test runs by the test team, depending on the impact of the changes made.

Test cases should be automated, and ideally run within Zuul jobs that are triggered upon code check in. These tests can also be run locally.

It is the responsibility of each project team to ensure that they have the proper test automation in Zuul jobs for their repos. It is the responsibility of every developer to submit new automated tests with their code changes.

Some issues can be configuration specific and for instance only reproduce on standard configurations on bare metal, or on simplex configurations in a virtual environment. We don't expect every developer to have access to multiple hardware setups so collaboration with the test team is important. Collaboration with the test team may also be needed to help create new test cases in their test suites.

Testing should also include unit tests or functional tests. Such tests should be added to the git repos as part of the code check in.

Once all of the required tests are written, have been checked in and have been successfully executed, the test phase is complete. Completion of developer testing is usually a requirement for acceptance of the code by the core reviewers, so the implementation and test phases usually complete at the same time.

Documentation phase

Code changes to StarlingX that change the user interface, or the project's APIs or the behavior of the system should be documented.

The process to submit a documentation change is described in the Documentation Contributor's Guide.

Some changes may require an update to the project's release notes. Please consult with your sub-project's PL and TL to see if release notes are needed, and follow the Release Note to contibute your changes to the release notes.

The documentation changes needed for a code change should be included in the planning phase, working with project's docs team as for help with the documentation file format or for how to place your documentation changes into the projects formal documentation.

The documentation phase is complete when all documents impacted by a change are complete and have been merged by the docs project's core reviewers.