diff --git a/doc/source/contributor/dlrn-promoter-overview.rst b/doc/source/contributor/dlrn-promoter-overview.rst new file mode 100644 index 00000000..734a5885 --- /dev/null +++ b/doc/source/contributor/dlrn-promoter-overview.rst @@ -0,0 +1,141 @@ +How the TripleO-RDO Pipelines' Promotions Work +============================================== + +Building consumable RDO repos and images involves various stages. +Each stage takes inputs and outputs artifacts. This document explains the +stages comprising the promotion pipelines, and the tools used to create +and manage the resulting artifacts. + +What is DLRN? +------------- + +DLRN is a tool to build RPM packages from each commit to a set of +OpenStack-related git repositories that are included in RDO. +DLRN builds are run through CI and to detect packaging issues with the +upstream branches of these Openstack projects. + +DLRN Artifacts - Hashes and Repos +--------------------------------- + +When a DLRN build completes, it produces a new hash and related repo version. +For example, the Pike builds on CentOS are available at: +https://trunk.rdoproject.org/centos7-pike/. +The builds are placed in directories by DLRN hash. Each directory contains +the RPMs as well as a repo file +https://trunk.rdoproject.org/centos7-pike/current-tripleo/delorean.repo +and a ``commit.yaml`` file +https://trunk.rdoproject.org/centos7-pike/current-tripleo/commit.yaml. + +There are some standard links that are updated as the builds complete and pass +stages of CI. Examples are these links are: + +- https://trunk.rdoproject.org/centos7-pike/current/ +- https://trunk.rdoproject.org/centos7-pike/consistent/ +- https://trunk.rdoproject.org/centos7-pike/current-tripleo/ +- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo/ +- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo-internal/ +- https://trunk.rdoproject.org/centos7-pike/tripleo-ci-testing/ + +The above links will be referenced in the sections below. + +Promoting through the Stages - DLRN API +--------------------------------------- + +DLRN API Client +``````````````` + +`The DLRN API +`_ +`client `_ +enables users to query repo status, upload new hashes and create promotions. +Calls to the dlrnapi_client retrieve the inputs to stages and upload artifacts +after stages. + +For example: + +:: + + $ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \ + promotion-get --promote-name tripleo-ci-testing + + [{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562', + 'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b', + 'promote_name': 'tripleo-ci-testing', + 'timestamp': 1506427440}, + {'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562', + [..] + + + $ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \ + repo-status --commit-hash ec650aa2c8ce952e4a33651190301494178ac562 \ + --distro-hash 9a7acc684265872ff288a11610614c3b5739939b + + [{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562', + 'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b', + 'in_progress': False, + 'job_id': 'consistent', + 'notes': '', + 'success': True, + 'timestamp': 1506409403, + 'url': ''}, + {'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562', + 'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b', + 'in_progress': False, + 'job_id': 'periodic-singlenode-featureset023', + 'notes': '', + 'success': True, + 'timestamp': 1506414726, + 'url': 'https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-centos-7-master-containers-build/8a76883'}, + {'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562', + [..] + + +DLRN API Promoter +````````````````` + +`The DLRN API Promoter script +`_ +is a Python script that, based on the information in an input config file, +will promote an existing DLRN link to another link, provided the required tests +return successful results. + +For example, +`the master ini config file +`_ +is passed to the `promoter script +`_ +to promote the ``current-tripleo`` link to ``current-tripleo-rdo``. See the +sections above where both these links (for Pike) were shown. + +In the RDO Phase 1 pipeline, the tests listed under the ``[current-tripleo-rdo]`` +are run with the ``current-tripleo`` hash. Each test reports its ``success`` status to the +DLRN API endpoint for the Master release, ``api-centos-master-uc``. + +If each test reports ``SUCCESS: true``, the content of the ``current-tripleo`` +will become the new content of the ``current-tripleo-rdo`` hash. + +For complete documentation on how to run the Promoter script see: +https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/README.md + + +Pushing RDO containers to ``docker.io`` +``````````````````````````````````````` + +The DLRN Promotor script calls the `container push playbook +`_ +to push the RDO containers at each stage to `docker.io +`_. +Note that the the above ``docker.io`` link shows containers tagged with ``tripleo-ci-testing``, +``current-tripleo`` and ``current-tripleo-rdo``. + + +DLRN API Promoter Server +```````````````````````` + +It is recommended that the Promoter script is run from a dedicated server. +`The promoter-setup repo +`_ +contains the Ansible playbook used to setup the promoter-server in the RDO +Cloud environment. This playbook allows the promoter script server to be +rebuilt as required. + diff --git a/doc/source/contributor/index.rst b/doc/source/contributor/index.rst index d3b6f3fd..421dbf0d 100644 --- a/doc/source/contributor/index.rst +++ b/doc/source/contributor/index.rst @@ -8,3 +8,5 @@ TripleO Contributor Guide new_developers contributions check_gates + dlrn-promoter-overview + stages-overview diff --git a/doc/source/contributor/stages-overview.rst b/doc/source/contributor/stages-overview.rst new file mode 100644 index 00000000..a423ea20 --- /dev/null +++ b/doc/source/contributor/stages-overview.rst @@ -0,0 +1,99 @@ +Promotion Stages +================ + +The list below shows each stage within the RDO promotion workflow. +Each stage shows the inputs taken and the artifacts produced. + +.. note:: All the links shown in the diagram refer to the Master release. + + Links for stable branches would include the stable release name, + for example, `Pike stable release `_. + +1. **Upstream TripleO** + + `CI DLRN Master consistent + `_ + is generated every 30 mins from upstream commits in case of no packaging errors. + + The ``current`` build contains the latest packages that can be built, + even if some other packages are failing to build. The ``current`` and + ``consistent`` builds will be equivalent where there are no problems with + the build. See https://trunk.rdoproject.org/centos7-master/report.html + (for Master) for the result of builds. + +2. **Upstream Promotion Pipeline** + + `rdoproject.org zuul `_ + + *Update* from consistent -> tripleo-ci-testing + https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/ + + *Push containers* to ``docker.io`` tagged with ``tripleo-ci-testing`` + https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ + + Run tests - if tests report success -> + *Promote* from tripleo-ci-testing -> current-tripleo using DLRN Promoter + https://trunk.rdoproject.org/centos7-master/current-tripleo/ + + *Push containers* to trunk.registry.rdoproject.org/master/ (tripleo-ci-testing) + and upon promotion, also tag them with ``current-tripleo`` + https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build + + *Update images* using DLRN Promoter + https://images.rdoproject.org/master/rdo_trunk/current-tripleo/stable/ + periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload + See artifacts referenced: + https://github.com/openstack/tripleo-quickstart/blob/master/config/release/master.yml + https://images.rdoproject.org/master/rdo_trunk/current-tripleo + RH1 mirror server for images http://66.187.229.139/ + + *Push containers* to ``docker.io`` tagged with ``current-tripleo`` + https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ + + Logs from the DLRN Promoter can be accessed on http://38.145.33.13/ + `A Sova instance `_ is used to monitor these jobs + +3. **RDO Phase 1** + + https://ci.centos.org/job/rdo_trunk-promote-master-current-tripleo/ + + Get https://trunk-primary.rdoproject.org/centos7-pike/current-tripleo/delorean.repo + + Run tests - if tests report success -> + *Promote* from current-tripleo -> current-tripleo-rdo using DLRN Promoter + https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo/ + + *Promote the images* to + https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo/ + + *Tag the containers* as ``current-tripleo-rdo`` + + *Push containers* to ``docker.io`` tagged with ``current-tripleo-rdo`` + https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ + +4. **RDO Phase 2** + + Get RDO Phase 1 generated image (RDO on CentOS) from + https://images.rdoproject.org/master/$BUILD_SYS/$PIN/$hash_id + and cache internally + + Build http:///ci-images/master (RDO on RHEL) + + Run tests (RDO on CentOS for baremetal and RDO on RHEL) + If tests report success -> + *Update symlink* using DLRN Promoter + https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo-internal/ + + *Update the images link* as + https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo-internal + + *Tag the containers* as ``current-tripleo-rdo-internal`` + + Run futher downstream jobs (scale etc.) + rdo-promote-master-rdo_trunk-nonvoting + +5. **OSP Phase 0** + + *Hand off* to RHOSP builds + https://access.redhat.com/documentation/en/red-hat-openstack-platform/ +