Add documentation for TripleO/RDO/OSP Pipelines
This documentation covers the promotion pipelines and the use of the DLRN API Promoter script to promote and update images, repos and symlinks. Change-Id: Idcd8f4f4cd9bb8468ede95f4e879f0e2f710c9ba
This commit is contained in:
Ronelle Landy
parent
9e6170f498
commit
2f9d4b96c2
141
doc/source/contributor/dlrn-promoter-overview.rst
Normal file
141
doc/source/contributor/dlrn-promoter-overview.rst
Normal file
@ -0,0 +1,141 @@
|
|||||||
|
How the TripleO-RDO Pipelines' Promotions Work
|
||||||
|
==============================================
|
||||||
|
|
||||||
|
Building consumable RDO repos and images involves various stages.
|
||||||
|
Each stage takes inputs and outputs artifacts. This document explains the
|
||||||
|
stages comprising the promotion pipelines, and the tools used to create
|
||||||
|
and manage the resulting artifacts.
|
||||||
|
|
||||||
|
What is DLRN?
|
||||||
|
-------------
|
||||||
|
|
||||||
|
DLRN is a tool to build RPM packages from each commit to a set of
|
||||||
|
OpenStack-related git repositories that are included in RDO.
|
||||||
|
DLRN builds are run through CI and to detect packaging issues with the
|
||||||
|
upstream branches of these Openstack projects.
|
||||||
|
|
||||||
|
DLRN Artifacts - Hashes and Repos
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
When a DLRN build completes, it produces a new hash and related repo version.
|
||||||
|
For example, the Pike builds on CentOS are available at:
|
||||||
|
https://trunk.rdoproject.org/centos7-pike/.
|
||||||
|
The builds are placed in directories by DLRN hash. Each directory contains
|
||||||
|
the RPMs as well as a repo file
|
||||||
|
https://trunk.rdoproject.org/centos7-pike/current-tripleo/delorean.repo
|
||||||
|
and a ``commit.yaml`` file
|
||||||
|
https://trunk.rdoproject.org/centos7-pike/current-tripleo/commit.yaml.
|
||||||
|
|
||||||
|
There are some standard links that are updated as the builds complete and pass
|
||||||
|
stages of CI. Examples are these links are:
|
||||||
|
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/current/
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/consistent/
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/current-tripleo/
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo/
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo-internal/
|
||||||
|
- https://trunk.rdoproject.org/centos7-pike/tripleo-ci-testing/
|
||||||
|
|
||||||
|
The above links will be referenced in the sections below.
|
||||||
|
|
||||||
|
Promoting through the Stages - DLRN API
|
||||||
|
---------------------------------------
|
||||||
|
|
||||||
|
DLRN API Client
|
||||||
|
```````````````
|
||||||
|
|
||||||
|
`The DLRN API
|
||||||
|
<https://github.com/softwarefactory-project/DLRN/blob/master/doc/source/api.rst>`_
|
||||||
|
`client <https://github.com/softwarefactory-project/dlrnapi_client/>`_
|
||||||
|
enables users to query repo status, upload new hashes and create promotions.
|
||||||
|
Calls to the dlrnapi_client retrieve the inputs to stages and upload artifacts
|
||||||
|
after stages.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
$ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \
|
||||||
|
promotion-get --promote-name tripleo-ci-testing
|
||||||
|
|
||||||
|
[{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
|
||||||
|
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
|
||||||
|
'promote_name': 'tripleo-ci-testing',
|
||||||
|
'timestamp': 1506427440},
|
||||||
|
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
|
||||||
|
[..]
|
||||||
|
|
||||||
|
|
||||||
|
$ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \
|
||||||
|
repo-status --commit-hash ec650aa2c8ce952e4a33651190301494178ac562 \
|
||||||
|
--distro-hash 9a7acc684265872ff288a11610614c3b5739939b
|
||||||
|
|
||||||
|
[{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
|
||||||
|
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
|
||||||
|
'in_progress': False,
|
||||||
|
'job_id': 'consistent',
|
||||||
|
'notes': '',
|
||||||
|
'success': True,
|
||||||
|
'timestamp': 1506409403,
|
||||||
|
'url': ''},
|
||||||
|
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
|
||||||
|
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
|
||||||
|
'in_progress': False,
|
||||||
|
'job_id': 'periodic-singlenode-featureset023',
|
||||||
|
'notes': '',
|
||||||
|
'success': True,
|
||||||
|
'timestamp': 1506414726,
|
||||||
|
'url': 'https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-centos-7-master-containers-build/8a76883'},
|
||||||
|
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
|
||||||
|
[..]
|
||||||
|
|
||||||
|
|
||||||
|
DLRN API Promoter
|
||||||
|
`````````````````
|
||||||
|
|
||||||
|
`The DLRN API Promoter script
|
||||||
|
<https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter>`_
|
||||||
|
is a Python script that, based on the information in an input config file,
|
||||||
|
will promote an existing DLRN link to another link, provided the required tests
|
||||||
|
return successful results.
|
||||||
|
|
||||||
|
For example,
|
||||||
|
`the master ini config file
|
||||||
|
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/master.ini>`_
|
||||||
|
is passed to the `promoter script
|
||||||
|
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py>`_
|
||||||
|
to promote the ``current-tripleo`` link to ``current-tripleo-rdo``. See the
|
||||||
|
sections above where both these links (for Pike) were shown.
|
||||||
|
|
||||||
|
In the RDO Phase 1 pipeline, the tests listed under the ``[current-tripleo-rdo]``
|
||||||
|
are run with the ``current-tripleo`` hash. Each test reports its ``success`` status to the
|
||||||
|
DLRN API endpoint for the Master release, ``api-centos-master-uc``.
|
||||||
|
|
||||||
|
If each test reports ``SUCCESS: true``, the content of the ``current-tripleo``
|
||||||
|
will become the new content of the ``current-tripleo-rdo`` hash.
|
||||||
|
|
||||||
|
For complete documentation on how to run the Promoter script see:
|
||||||
|
https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/README.md
|
||||||
|
|
||||||
|
|
||||||
|
Pushing RDO containers to ``docker.io``
|
||||||
|
```````````````````````````````````````
|
||||||
|
|
||||||
|
The DLRN Promotor script calls the `container push playbook
|
||||||
|
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml>`_
|
||||||
|
to push the RDO containers at each stage to `docker.io
|
||||||
|
<https://hub.docker.com/r/tripleopike/centos-binary-heat-api/tags/>`_.
|
||||||
|
Note that the the above ``docker.io`` link shows containers tagged with ``tripleo-ci-testing``,
|
||||||
|
``current-tripleo`` and ``current-tripleo-rdo``.
|
||||||
|
|
||||||
|
|
||||||
|
DLRN API Promoter Server
|
||||||
|
````````````````````````
|
||||||
|
|
||||||
|
It is recommended that the Promoter script is run from a dedicated server.
|
||||||
|
`The promoter-setup repo
|
||||||
|
<https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/promoter-setup>`_
|
||||||
|
contains the Ansible playbook used to setup the promoter-server in the RDO
|
||||||
|
Cloud environment. This playbook allows the promoter script server to be
|
||||||
|
rebuilt as required.
|
||||||
|
|
||||||
@ -8,3 +8,5 @@ TripleO Contributor Guide
|
|||||||
new_developers
|
new_developers
|
||||||
contributions
|
contributions
|
||||||
check_gates
|
check_gates
|
||||||
|
dlrn-promoter-overview
|
||||||
|
stages-overview
|
||||||
|
|||||||
99
doc/source/contributor/stages-overview.rst
Normal file
99
doc/source/contributor/stages-overview.rst
Normal file
@ -0,0 +1,99 @@
|
|||||||
|
Promotion Stages
|
||||||
|
================
|
||||||
|
|
||||||
|
The list below shows each stage within the RDO promotion workflow.
|
||||||
|
Each stage shows the inputs taken and the artifacts produced.
|
||||||
|
|
||||||
|
.. note:: All the links shown in the diagram refer to the Master release.
|
||||||
|
|
||||||
|
Links for stable branches would include the stable release name,
|
||||||
|
for example, `Pike stable release <https://trunk.rdoproject.org/centos7-pike/tripleo-ci-testing/>`_.
|
||||||
|
|
||||||
|
1. **Upstream TripleO**
|
||||||
|
|
||||||
|
`CI DLRN Master consistent
|
||||||
|
<https://trunk.rdoproject.org/centos7-master/consistent/>`_
|
||||||
|
is generated every 30 mins from upstream commits in case of no packaging errors.
|
||||||
|
|
||||||
|
The ``current`` build contains the latest packages that can be built,
|
||||||
|
even if some other packages are failing to build. The ``current`` and
|
||||||
|
``consistent`` builds will be equivalent where there are no problems with
|
||||||
|
the build. See https://trunk.rdoproject.org/centos7-master/report.html
|
||||||
|
(for Master) for the result of builds.
|
||||||
|
|
||||||
|
2. **Upstream Promotion Pipeline**
|
||||||
|
|
||||||
|
`rdoproject.org zuul <https://review.rdoproject.org/zuul/>`_
|
||||||
|
|
||||||
|
*Update* from consistent -> tripleo-ci-testing
|
||||||
|
https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/
|
||||||
|
|
||||||
|
*Push containers* to ``docker.io`` tagged with ``tripleo-ci-testing``
|
||||||
|
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
|
||||||
|
|
||||||
|
Run tests - if tests report success ->
|
||||||
|
*Promote* from tripleo-ci-testing -> current-tripleo using DLRN Promoter
|
||||||
|
https://trunk.rdoproject.org/centos7-master/current-tripleo/
|
||||||
|
|
||||||
|
*Push containers* to trunk.registry.rdoproject.org/master/ (tripleo-ci-testing)
|
||||||
|
and upon promotion, also tag them with ``current-tripleo``
|
||||||
|
https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build
|
||||||
|
|
||||||
|
*Update images* using DLRN Promoter
|
||||||
|
https://images.rdoproject.org/master/rdo_trunk/current-tripleo/stable/
|
||||||
|
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload
|
||||||
|
See artifacts referenced:
|
||||||
|
https://github.com/openstack/tripleo-quickstart/blob/master/config/release/master.yml
|
||||||
|
https://images.rdoproject.org/master/rdo_trunk/current-tripleo
|
||||||
|
RH1 mirror server for images http://66.187.229.139/
|
||||||
|
|
||||||
|
*Push containers* to ``docker.io`` tagged with ``current-tripleo``
|
||||||
|
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
|
||||||
|
|
||||||
|
Logs from the DLRN Promoter can be accessed on http://38.145.33.13/
|
||||||
|
`A Sova instance <http://38.145.34.234/>`_ is used to monitor these jobs
|
||||||
|
|
||||||
|
3. **RDO Phase 1**
|
||||||
|
|
||||||
|
https://ci.centos.org/job/rdo_trunk-promote-master-current-tripleo/
|
||||||
|
|
||||||
|
Get https://trunk-primary.rdoproject.org/centos7-pike/current-tripleo/delorean.repo
|
||||||
|
|
||||||
|
Run tests - if tests report success ->
|
||||||
|
*Promote* from current-tripleo -> current-tripleo-rdo using DLRN Promoter
|
||||||
|
https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo/
|
||||||
|
|
||||||
|
*Promote the images* to
|
||||||
|
https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo/
|
||||||
|
|
||||||
|
*Tag the containers* as ``current-tripleo-rdo``
|
||||||
|
|
||||||
|
*Push containers* to ``docker.io`` tagged with ``current-tripleo-rdo``
|
||||||
|
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
|
||||||
|
|
||||||
|
4. **RDO Phase 2**
|
||||||
|
|
||||||
|
Get RDO Phase 1 generated image (RDO on CentOS) from
|
||||||
|
https://images.rdoproject.org/master/$BUILD_SYS/$PIN/$hash_id
|
||||||
|
and cache internally
|
||||||
|
|
||||||
|
Build http://<internal>/ci-images/master (RDO on RHEL)
|
||||||
|
|
||||||
|
Run tests (RDO on CentOS for baremetal and RDO on RHEL)
|
||||||
|
If tests report success ->
|
||||||
|
*Update symlink* using DLRN Promoter
|
||||||
|
https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo-internal/
|
||||||
|
|
||||||
|
*Update the images link* as
|
||||||
|
https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo-internal
|
||||||
|
|
||||||
|
*Tag the containers* as ``current-tripleo-rdo-internal``
|
||||||
|
|
||||||
|
Run futher downstream jobs (scale etc.)
|
||||||
|
rdo-promote-master-rdo_trunk-nonvoting
|
||||||
|
|
||||||
|
5. **OSP Phase 0**
|
||||||
|
|
||||||
|
*Hand off* to RHOSP builds
|
||||||
|
https://access.redhat.com/documentation/en/red-hat-openstack-platform/
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user