449 lines
18 KiB
ReStructuredText
449 lines
18 KiB
ReStructuredText
========================
|
|
Team and repository tags
|
|
========================
|
|
|
|
.. image:: https://governance.openstack.org/tc/badges/requirements.svg
|
|
:target: https://governance.openstack.org/tc/reference/tags/index.html
|
|
|
|
.. Change things from this point on
|
|
|
|
============================================
|
|
Global Requirements for OpenStack Projects
|
|
============================================
|
|
|
|
Why Global Requirements?
|
|
========================
|
|
|
|
During the Havana release cycle we kept running into coherency issues
|
|
with trying to install all the OpenStack components into a single
|
|
environment. The issue is that syncing of ``requirements.txt`` between
|
|
projects was an eventually consistent problem. Some projects would
|
|
update quickly, others would not. We'd never have the same versions
|
|
specified as requirements between packages.
|
|
|
|
Because of the way that python package installation with pip works,
|
|
this means that if you get lucky you'll end up with a working
|
|
system. If you don't you can easily break all of OpenStack on a
|
|
requirements update.
|
|
|
|
An example of how bad this had gotten is that python-keystoneclient
|
|
would typically be installed / uninstalled 6 times during the course
|
|
of a DevStack gate run during Havana. If the last version of python
|
|
keystoneclient happened to be incompatible with some piece of
|
|
OpenStack a very hard to diagnose break occurs.
|
|
|
|
We also had an issue with projects adding dependencies of python
|
|
libraries without thinking through the long term implications of those
|
|
libraries. Is the library actively maintained? Is the library of a
|
|
compatible license? Does the library duplicate the function of existing
|
|
libraries that we already have in requirements? Is the library python
|
|
3 compatible? Is the library something that already exists in Linux
|
|
Distros that we target (Ubuntu / Fedora). The answer to many of these
|
|
questions was no.
|
|
|
|
Global requirements gives us a single place where we can evaluate
|
|
these things so that we can make a global decision for OpenStack on
|
|
the suitability of the library.
|
|
|
|
Since Havana we've also observed significant CI disruption occurring due to
|
|
upstream releases of software that are incompatible (whether in small
|
|
or large ways) with OpenStack. So Global Requirements also serves as a control
|
|
point to determine the precise versions of dependencies that will be used
|
|
during CI.
|
|
|
|
Solution
|
|
========
|
|
|
|
The mechanics of the solution are relatively simple. We maintain a
|
|
central list of all the requirements (``global-requirements.txt``)
|
|
that are allowed in OpenStack projects. This is enforced for
|
|
``requirements.txt``, ``test-requirements.txt``,
|
|
``doc/requirements.txt``, and extras defined in
|
|
``setup.cfg``. This is maintained by hand, with changes going through CI.
|
|
|
|
We also maintain a compiled list of the exact versions, including transitive
|
|
dependencies, of packages that are known to work in the OpenStack CI system.
|
|
This is maintained via an automated process that calculates the list and
|
|
proposes a change back to this repository. A consequence of this is that
|
|
new releases of OpenStack libraries are not immediately used: they have to
|
|
pass through this automated process before we can benefit from (or be harmed
|
|
by) them.
|
|
|
|
Format
|
|
------
|
|
|
|
``global-requirements.txt`` supports a subset of pip requirement file
|
|
contents. Distributions may only be referenced by name, not URL. Options
|
|
(such as -e or -f) may not be used. Version specifiers, environment markers
|
|
and comments are all permitted. A single distribution may be listed more than
|
|
once if different specifiers are required with different markers - for
|
|
instance, if a dependency has dropped Python 2.7 support.
|
|
|
|
``upper-constraints.txt`` is machine generated and nothing more or less than
|
|
an exact list of versions.
|
|
|
|
``lower-constraints.txt`` is manually maintained and can be consumed by projects
|
|
for tracking their individual project specific constraints as well as giving a
|
|
good indication to deployers what the global minimum requirements are
|
|
for the set of projects in the integrated gate.
|
|
|
|
|
|
Enforcement for Test Runs
|
|
-------------------------
|
|
|
|
DevStack
|
|
++++++++
|
|
|
|
DevStack uses the pip ``-c`` option to pin all the libraries to known good
|
|
versions. ``edit-constraints`` can be used to unpin a single constraint, and
|
|
this is done to install libraries from git.
|
|
|
|
Tox
|
|
+++
|
|
|
|
We are working on the necessary changes to use ``upper-constraints.txt`` in
|
|
tox jobs but it is not yet complete.
|
|
|
|
Enforcement in Projects
|
|
-----------------------
|
|
|
|
All projects that have accepted the requirements contract (as listed
|
|
in ``projects.txt``) are expected to run a requirements compatibility
|
|
job. This job ensures that a project can not change any dependencies to
|
|
versions not compatible with ``global-requirements.txt``. It also ensures that
|
|
those projects can not add a requirement that is not already in
|
|
``global-requirements.txt``. This ``check-requirements`` job should
|
|
be merged in infra before proposing the change to ``projects.txt`` in
|
|
``openstack/requirements``.
|
|
|
|
Automatic Sync of Accepted Requirements
|
|
---------------------------------------
|
|
|
|
If an updated requirement is proposed to OpenStack and accepted to
|
|
``global-requirements.txt``, the system then also automatically pushes
|
|
a review request for the new requirements definition to the projects
|
|
that include it.
|
|
|
|
For instance: if a review is accepted to ``global-requirements.txt``
|
|
that increases the minimum version of python-keystoneclient, the
|
|
system will submit patches to all the OpenStack projects that list
|
|
python-keystoneclient as a requirement or test requirement to match
|
|
this new version definition.
|
|
|
|
This is intended as a time saving device for projects, as they can
|
|
fast approve requirements syncs and not have to manually worry about
|
|
whether or not they are up to date with the global definition.
|
|
|
|
Tools
|
|
=====
|
|
|
|
All the tools require openstack_requirements to be installed (e.g. in a Python
|
|
virtualenv). They all have help, which is the authoritative documentation.
|
|
|
|
update-requirements
|
|
-------------------
|
|
|
|
This will update the requirements in a project from the global requirements
|
|
file found in ``.``. Alternatively, pass ``--source`` to use a different
|
|
global requirements file::
|
|
|
|
update-requirements --source /opt/stack/requirements /opt/stack/nova
|
|
|
|
Entries in all requirements files will have their versions updated to match
|
|
the entries listed in the global requirements. Excess entries will cause
|
|
errors in hard mode (the default) or be ignored in soft mode.
|
|
|
|
generate-constraints
|
|
--------------------
|
|
|
|
Compile a constraints file showing the versions resulting from installing all
|
|
of ``global-requirements.txt``::
|
|
|
|
generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
|
|
-b blacklist.txt -r global-requirements.txt > new-constraints.txt
|
|
|
|
edit-constraints
|
|
----------------
|
|
|
|
Replace all references to a package in a constraints file with a new
|
|
specification. Used by DevStack to enable git installations of libraries that
|
|
are normally constrained::
|
|
|
|
edit-constraints oslo.db "-e file://opt/stack/oslo.db#egg=oslo.db"
|
|
|
|
Proposing changes
|
|
=================
|
|
|
|
Look at the `Review Guidelines` and make sure your change meets them.
|
|
|
|
All changes to ``global-requirements.txt`` may dramatically alter the contents
|
|
of ``upper-constraints.txt`` due to adding or removing transitive
|
|
dependencies. As such you should always generate a diff against the current
|
|
merged constraints, otherwise your change may fail if it is incompatible with
|
|
the current tested constraints.
|
|
|
|
A change to the minimum specified vesion of a library in ``global-requirements.txt``
|
|
currenty requires adjusting the ``lower-constraints.txt`` file alongside with the
|
|
new constrainted coinstallable version of minimums.
|
|
|
|
Regenerating involves five steps.
|
|
|
|
1) Install the dependencies needed to compile various Python packages::
|
|
|
|
sudo apt-get install $(bindep -b)
|
|
|
|
2) Create a reference file (do this without your patch applied)::
|
|
|
|
generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
|
|
-b blacklist.txt -r global-requirements.txt > baseline
|
|
|
|
3) Apply your patch and generate a new reference file::
|
|
|
|
generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
|
|
-b blacklist.txt -r global-requirements.txt > updated
|
|
|
|
4) Diff them::
|
|
|
|
diff -p baseline updated
|
|
|
|
5) Apply the patch to ``upper-constraints.txt``. This may require some
|
|
fiddling. ``edit-constraint`` can do this for you **when the change
|
|
does not involve multiple lines for one package**.
|
|
|
|
Review Guidelines
|
|
=================
|
|
|
|
There are a set of questions that every reviewer should ask on any
|
|
proposed requirements change. Proposers can make reviewing easier by
|
|
including the answers to these questions in the commit message for
|
|
their change.
|
|
|
|
General Review Criteria
|
|
-----------------------
|
|
|
|
- No specifications for library versions should contain version caps
|
|
|
|
As a community we value early feedback of broken upstream
|
|
requirements, so version caps should be avoided except when dealing
|
|
with exceptionally unstable libraries.
|
|
|
|
If a library is exceptionally unstable, we should also be
|
|
considering whether we want to replace it over time with one that
|
|
*is* stable, or to contribute to the upstream community to help
|
|
stabilize it.
|
|
|
|
- Libraries should contain a sensible known working minimum version
|
|
|
|
Bare library names are bad. If it's unknown what a working minimum
|
|
is, look at the output of pip freeze at the end of a successful
|
|
DevStack/tempest run and use that version. At least that's known to
|
|
be working now.
|
|
|
|
- Commit message should refer to consuming projects(s)
|
|
|
|
Preferably, the comments should also identify which feature or
|
|
blueprint requires the new specification. Ideally, changes should
|
|
already be proposed, so that its use can be seen.
|
|
|
|
- The blacklist is for handling dependencies that cannot be constrained.
|
|
For instance, linters which each project has at a different release level,
|
|
and which make projects fail on every release (because they add rules) -
|
|
those cannot be globally constrained unless we coordinate updating all of
|
|
OpenStack to the new release at the same time - but given the volunteer
|
|
and loosely coupled nature of the big tent that is infeasible. Dependencies
|
|
that are only used in unconstrained places should not be blacklisted - they
|
|
may be constrained in future, and there's no harm caused by constraining
|
|
them today. Entries in the blacklist should have a comment explaining the
|
|
reason for blacklisting.
|
|
|
|
- Reviews that only update ``projects.txt`` should be workflow approved
|
|
alongside or before other reviews in order to have the OpenStack Proposal Bot
|
|
propagation be useful as soon as possible for the other projects. For project
|
|
removal or addition, the +1 from the current PTL (or core if the PTL proposed
|
|
the change) should be enough.
|
|
|
|
For new Requirements
|
|
--------------------
|
|
|
|
- Is the library actively maintained?
|
|
|
|
We *really* want some indication that the library is something we
|
|
can get support on if we or our users find a bug, and that we
|
|
don't have to take over and fork the library.
|
|
|
|
Pointers to recent activity upstream and a consistent release model
|
|
are appreciated.
|
|
|
|
- Is the library good code?
|
|
|
|
It's expected, before just telling everyone to download arbitrary 3rd
|
|
party code from the internet, that the submitter has taken a deep dive
|
|
into the code to get a feel on whether this code seems solid enough
|
|
to depend on. That includes ensuring the upstream code has some
|
|
reasonable testing baked in.
|
|
|
|
- Is the library python 3 compatible?
|
|
|
|
OpenStack will eventually need to support python 3. At this point
|
|
adding non python 3 compatible libraries should only be done under
|
|
*extreme* need. It should be considered a very big exception.
|
|
|
|
- Is the library license compatible?
|
|
|
|
The library should be licensed as described in `Licensing requirements`_,
|
|
and the license should be described in a comment on the same line as the
|
|
added dependency. If you have doubts over licensing compatibility, like
|
|
for example when adding a GPL test dependency, you can seek advice from
|
|
Robert Collins (lifeless), Monty Taylor (mordred) or Jim Blair (jeblair).
|
|
|
|
- Is the library already packaged in the distros we target (Ubuntu
|
|
latest / Fedora latest)?
|
|
|
|
By adding something to OpenStack ``global-requirements.txt`` we are
|
|
basically demanding that Linux Distros package this for the next
|
|
release of OpenStack. If they already have, great. If not, we should
|
|
be cautious of adding it. :ref:`finding-distro-status`
|
|
|
|
- Is the function of this library already covered by other libraries
|
|
in ``global-requirements.txt``?
|
|
|
|
Everyone has their own pet libraries that they like to use, but we
|
|
do not need three different request mocking libraries in OpenStack.
|
|
|
|
If this new requirement is about replacing an existing library with
|
|
one that's better suited for our needs, then we also need the
|
|
transition plan to drop the old library in a reasonable amount of
|
|
time.
|
|
|
|
- Is the library required for OpenStack project or related dev or
|
|
infrastructure setup? (Answer to this should be Yes, of course)
|
|
Which?
|
|
|
|
Please provide details such as gerrit change request or launchpad
|
|
bug/blueprint specifying the need for adding this library.
|
|
|
|
- If the library release is managed by the Openstack release process does
|
|
it use the `cycle-with-intermediary` release type?
|
|
|
|
This is needed to ensure that updated releases that consume requirements
|
|
updates are available for integration/coninstallability tests with other
|
|
projects.
|
|
|
|
- Do I need to update anything else?
|
|
|
|
When new library is added, initial version of release needs to be added
|
|
to ``upper-constraints.txt``. After that, OpenStack Proposal Bot will
|
|
propose updates.
|
|
|
|
.. _Licensing requirements: https://governance.openstack.org/tc/reference/licensing.html
|
|
|
|
For Upgrading Requirements Versions
|
|
-----------------------------------
|
|
|
|
- Why is it impossible to use the current version definition?
|
|
|
|
Everyone likes everyone else to use the latest version of their
|
|
code. However, deployers really don't like to be constantly updating
|
|
things. Unless it's actually **impossible** to use the minimum
|
|
version specified in ``global-requirements.txt``, it should not be
|
|
changed.
|
|
|
|
Leave that decision to deployers and distros.
|
|
|
|
- Changes to update the minimum version of a library developed by the
|
|
OpenStack community can be approved by one reviewer, as long as the
|
|
constraints are correct and the tests pass.
|
|
|
|
.. _finding-distro-status:
|
|
|
|
Finding Distro Status
|
|
---------------------
|
|
|
|
From the OpenStack distro support policy:
|
|
|
|
OpenStack will target its development efforts to latest Ubuntu/Fedora,
|
|
but will not introduce any changes that would make it impossible to
|
|
run on the latest Ubuntu LTS or latest RHEL.
|
|
|
|
As such we really need to know what the current state of packaging is
|
|
on these platforms (and ideally Debian, Gentoo, and SUSE as well).
|
|
|
|
For people unfamiliar with Linux Distro packaging you can use the
|
|
following tools to search for packages:
|
|
|
|
- Ubuntu - http://packages.ubuntu.com/
|
|
- Fedora - https://apps.fedoraproject.org/packages/
|
|
- Gentoo - https://packages.gentoo.org/
|
|
- SUSE - https://build.opensuse.org/project/show/devel:languages:python
|
|
|
|
For ``upper-constraints.txt`` changes
|
|
-------------------------------------
|
|
|
|
If the change was proposed by the OpenStack CI bot, then if the change has
|
|
passed CI, only one reviewer is needed and they should +2 +A without thinking
|
|
about things.
|
|
|
|
If the change was not proposed by the OpenStack CI bot, and only
|
|
changes the ``upper-constraints.txt`` entry for a new library release,
|
|
then the change should be approved if it passes the tests. See the
|
|
README.rst in openstack/releases for more details of the release
|
|
process.
|
|
|
|
If the change was not proposed by the OpenStack CI bot, and is not
|
|
related to releasing one of our libraries, and does not include a
|
|
``global-requirements.txt`` change, then it should be rejected: the CI
|
|
bot will generate an appropriate change itself. Ask in
|
|
#openstack-infra if the bot needs to be run more quickly.
|
|
|
|
Otherwise the change may be the result of recalculating the constraints which
|
|
changed when a ``global-requirements.txt`` change is proposed. In this case, ignore
|
|
the changes to ``upper-constraints.txt`` and review the
|
|
``global-requirements.txt`` component of the change.
|
|
|
|
stable-branch maintenance
|
|
-------------------------
|
|
|
|
Upper-constraints
|
|
+++++++++++++++++
|
|
|
|
Most of the work is done by stable-maint in the releases project. The releases
|
|
project ensures valid stable releases (little to no API level changes, bugfix
|
|
only, etc). Once released, the new version is requested to be updated in
|
|
requirements. The following restrictions are in place to help ensure stable
|
|
branches do not break.
|
|
|
|
- In stable branches, we usually only update constraints for projects managed
|
|
within the OpenStack community. Exceptions are made for other projects when
|
|
there are gate issues. Those updates must be proposed by hand.
|
|
|
|
- The requirements team also verifies the new version's requirements changes
|
|
line up with the requirements in the stable branch (GR and UC).
|
|
|
|
Global-requirements
|
|
+++++++++++++++++++
|
|
|
|
These should be few and far between on stable branches, mainly masking known
|
|
bad versions or in extreme adding a maximum version allowable for a package.
|
|
We work to remove these caps as well. Raising effective minimums is only
|
|
acceptable during `Phase I`, and only due to security issues.
|
|
|
|
.. _Phase I: https://docs.openstack.org/project-team-guide/stable-branches.html#support-phases
|
|
|
|
New requirements
|
|
++++++++++++++++
|
|
|
|
In nearly all cases this is not allowed. An example where this is allowed
|
|
would be: A dependency of a dependency has an issue that impacts OpenStack.
|
|
It wasn't listed in global-requirements.txt but it is required. In order to
|
|
block the affected releases and still be able to keep requirements in sync, we
|
|
list the library in global-requirements.txt and update all projects that
|
|
require it.
|
|
|
|
Resources
|
|
=========
|
|
|
|
- Documentation: https://docs.openstack.org/requirements/latest/
|
|
- Wiki: https://wiki.openstack.org/wiki/Requirements
|
|
- Bugs: https://launchpad.net/openstack-requirements
|
|
|