opendev/base-jobs
Code Issues Proposed changes
base-jobs/zuul.d/jobs.yaml
Jeremy Stanley 16eef4ddd4 Add a opendev-publish-unversioned-tox-docs job 
It's mildly annoying that, in the wake of tagging a new release,
documentation lags behind because we only publish in the promote
pipeline from what was built by the last change to merge in the
gate. There is an existing job which can build and publish
documentation in the release pipeline, but does so into a
version-specific path leading to multiple versioned copies of the
documentation.

For fast-moving projects it's not much of a problem
since there will be a new change coming along soon to refresh the
default version of the documents and reflect the recent tag in page
footers and release notes. In simple projects with only a single
branch and infrequent changes, however, there can be long periods of
time where the published documentation makes no indication of the
existence of the latest release. Address this with a new job which
publishes to "latest" instead of the tag name.

Note that this requires a copy of the existing secret, in order to
update the "tag" variable attached to it. Also worth noting, this
job could be run in parallel with the existing
opendev-publish-tox-docs job in the release pipeline for the same
project in order to get the benefits of both.

Change-Id: Ibd4f31b9d06137f9cc52bc1a07e1c37766494e7b
2022-04-19 16:13:08 +00:00

640 lines
20 KiB
YAML
Raw Blame History

# Shared zuul config common to all opendev tenants.
# Contains definitions of trusted jobs
# Changes to this job require a special procedure, because they can
# not be tested before landing, and if they are faulty, they will
# break all jobs, meaning subsequent corrections will not be able to
# land. To make a change:
#
# 1) Ensure that base-test and its playbooks are identical to base.
# 2) Make the change to base-test and/or its playbooks.
# 3) Merge the change from step 2. No jobs normally use base-test, so
# this is safe.
# 4) Propose a change to a job to reparent it to base-test. Choose a
# job which will exercise whatever you are changing. The
# "unittests" job in zuul-jobs is a good choice. Use [DNM] in the
# commit subject so that people know not to merge the change. Set
# it to "Work in progress" so people don't review it.
# 5) Once test results arrive for the change in step 2, make a change
# which copies the job and/or playbooks of base-test to base. In
# the commit message, link to (without using Depends-On:) the
# change from step 4 so reviewers can see the test results.
# 6) Once the change in step 5 merges, abandon the change from step 4.
- job:
name: base
parent: null
abstract: true
description: |
The base job for OpenDev's installation of Zuul.
All jobs ultimately inherit from this. It runs a pre-playbook
which copies all of the job's prepared git repos on to all of
the nodes in the nodeset. It runs a post-playbook which copies
all of the files in the logs/ subdirectory of the executor
work directory to the logserver.
It also sets default timeout and nodeset values (which may be
overidden).
Responds to these variables:
.. zuul:jobvar:: base_serial
:default: Omitted
This sets the serial keyword in the pre and post playbooks
which can be an integer or percentage.
See ansible documentation for more information:
http://docs.ansible.com/ansible/latest/playbooks_delegation.html
pre-run: playbooks/base/pre.yaml
post-run:
- playbooks/base/post.yaml
- playbooks/base/post-logs.yaml
cleanup-run: playbooks/base/cleanup.yaml
roles:
- zuul: zuul/zuul-jobs
timeout: 1800
post-timeout: 1800
vars:
ara_report_type: html
ara_report_path: ara-report
ara_compress_html: false
nodeset: ubuntu-focal
secrets: &log_clouds
- opendev_cloud_ovh_bhs
- opendev_cloud_ovh_gra
- opendev_cloud_rax_dfw
- opendev_cloud_rax_iad
- opendev_cloud_rax_ord
- opendev_cloud_vexxhost_ymq1
# See the procedure described above "base" before making changes to
# this job.
- job:
name: base-test
parent: null
ansible-version: 2.8
description: |
A job to test changes to the base job without disturbing the
main job in production. Not for general use.
pre-run: playbooks/base-test/pre.yaml
post-run:
- playbooks/base-test/post.yaml
- playbooks/base-test/post-logs.yaml
cleanup-run: playbooks/base-test/cleanup.yaml
roles:
- zuul: zuul/zuul-jobs
timeout: 1800
post-timeout: 1800
nodeset: ubuntu-focal
secrets: *log_clouds
- job:
name: base-minimal
parent: null
abstract: true
description: |
A subset of what the 'base' job provides: the absolute minimum considered
required to run for any one job.
It doesn't set up cached git repositories, will not set up mirrors,
doesn't validate the node, etc.
It is meant to be used, amongst other things, to test roles and
playbooks that would otherwise be included by default as part of the
'base' job.
These tasks, if required, can be included by the dependant jobs
themselves on a need basis.
pre-run: playbooks/base-minimal/pre.yaml
post-run:
- playbooks/base-minimal/post.yaml
- playbooks/base-minimal/post-logs.yaml
cleanup-run: playbooks/base-minimal/cleanup.yaml
roles:
- zuul: zuul/zuul-jobs
timeout: 1800
post-timeout: 1800
allowed-projects: zuul/zuul-jobs
vars:
ara_report_type: html
ara_report_path: ara-report
ara_compress_html: false
nodeset: ubuntu-focal
secrets: *log_clouds
- secret:
name: opendev-intermediate-registry
data:
host: insecure-ci-registry.opendev.org
port: 5000
username: zuul
password: !encrypted/pkcs1-oaep
- sN5wugpZqGCp8kwHLDLydHi7HUH5e5gLKA/Xhge0DdtcTLy4TLWASDvPwKkC+w3Y1CdQG
V0kPMJCKodsx3jz83zbcjptAWV90RMfgOmwy1FOs9/fnQtcnH2UDkoFhf0m3qYu05kbyB
yJ9ybIewEFt9v1GORcFSAr7Z8YIyhBNuJaRGb9dDW/T9WTBS+bhxMVq1hlOAq17458UXV
MhRTb5+nFhJxVaXYpV3rGjqr2TsryBk16iu5kf8I8/6QkwpttI1i/F6wFrjWBNg1FkDIN
OwcoGnqY9sTmCya54Ko03FroRP4pe09KcsFoGHMVYNT4CAbhrMEIBtyRk5q/MHls/o82p
fHu1Xr8dC/u8vHfYnL3TEtO87tucRzW9CJBCrrrq77a/i0BS6UFpMY63dnwI9M9h1v4Zp
y/ITLk/YNdqR8lxG/T7fQg1YmBjSAQCJKfHS4uulujrcbPp2X1RXAFooUPH+uZ4m+Ml4b
KlDb/XtMzYw2zjVIi0nMfPG0jfEgtzHBW23Nf62joUrmmtUWSq1aij5J+Vb2tBiuDFpN6
+v5Gg5Wb07OWvBjg0jcL9IvNwQXgWh39y4uq4TmKcYXHL6RnICtN5Hp4Gc7xDxNvp2+uC
Na/OOailVlvkWMcpO/l77/qFtNXoRxFTDWYqp5rE3fGmdq09hCiflFWgMMre08=
- job:
name: opendev-buildset-registry
description: |
Starts a buildset registry which interacts with the intermediate
CI registry to share speculative container images between
projects.
Configure any jobs which require the use of a buildset registry
to depend on this job using the "dependencies" job attribute.
This job will pause after starting the registry so that it is
available to any jobs which depend on it. Once all such jobs
are complete, this job will finish.
pre-run: playbooks/buildset-registry/pre.yaml
run: playbooks/buildset-registry/run.yaml
post-run: playbooks/buildset-registry/post.yaml
secrets:
- secret: opendev-intermediate-registry
name: intermediate_registry
vars:
docker_mirror_base_url: "http://{{ zuul_site_mirror_fqdn }}/deb-docker/{{ ansible_distribution_release }}"
- job:
name: opendev-build-docker-image-base
parent: opendev-buildset-registry
description: |
This is a parent for an image build job which expects a
buildset registry to be running and pulls images from the
intermediate registry into it. It mostly exists so that
the intermediate registry secret need not be supplied to the
image build playbook.
pre-run: playbooks/docker-image/pre.yaml
secrets:
- secret: opendev-intermediate-registry
name: intermediate_registry
- job:
name: opendev-build-docker-image
parent: opendev-build-docker-image-base
description: |
Starts a buildset registry (if one has not already been started,
e.g., by invoking :zuul:job:`opendev-buildset-registry` and
specifying it as a dependency) and builds one or more docker
images.
Analog of build-docker-image job, but with a buildset registry.
This job will pause after starting the registry so that it is
available to any jobs which depend on it. Once all such jobs
are complete, this job will finish.
.. include:: ../../playbooks/docker-image/README.rst
run: playbooks/docker-image/run.yaml
- job:
name: opendev-upload-docker-image
parent: opendev-build-docker-image
description: |
Starts a buildset registry and builds and uploads one or more
docker images to docker.io.
Analog of upload-docker-image job, but with a buildset registry.
.. include:: ../../playbooks/docker-image/README.rst
.. include:: ../../playbooks/docker-image/credentials.rst
post-run: playbooks/docker-image/upload.yaml
- job:
name: opendev-promote-docker-image
parent: promote-docker-image
description: |
Retag a previously-uploaded docker image.
Analog of promote-docker-image job.
.. include:: ../../playbooks/docker-image/README.rst
.. include:: ../../playbooks/docker-image/credentials.rst
- job:
name: opendev-buildset-registry-consumer
description: |
Pull from the intermediate registry
This is a parent for jobs which use container images and expect
a buildset registry to be running. It pulls images from the
intermediate registry into it.
pre-run: playbooks/docker-image/pre.yaml
secrets:
- secret: opendev-intermediate-registry
name: intermediate_registry
- job:
name: opendev-tox-docs
# This is not parented to tox-docs because the post playbook
# differs.
description: |
Build documentation with "tox".
Uses tox with the ``docs`` environment.
vars:
tox_envlist: docs
bindep_profile: compile doc
pre-run: playbooks/tox-docs/pre.yaml
run: playbooks/tox-docs/run.yaml
post-run: playbooks/tox-docs/post.yaml
- job:
name: opendev-publish-tox-docs-base
# This is not parented to opendev-tox-docs because the post
# playbook differs.
description: |
Publish a ref-based documentation build.
Use this in the tag or release pipelines to publish a build
based on a newly-created tag.
This is an abstract job intended to be inherited from in an
OpenDev tenant and an appropriate secret added.
.. zuul:jobvar:: afs
:type: dict
This is expected to be a Zuul Secret with these keys:
.. zuul:jobvar:: keytab
The AFS keytab for the service principal.
.. zuul:jobvar:: service_name
The name of the service princpal.
.. zuul:jobvar:: targets
This is a dict containing information about where docs should be
published.
.. zuul:jobvar:: master
This is expected to be a dict with a single key value pair:
`path: the full docs publication path to use if the job is
run on the master branch.`
.. zuul:jobvar:: branch
This is expected to be a dict with a key value pair:
`path: the full docs publication path to use if the job is run on
any other branch.`
.. zuul:jobvar:: tag
This is expected to be a dict with a key value pair:
`path: the full docs publication path to use if the job is run on
a tag.`
abstract: True
vars:
tox_envlist: docs
bindep_profile: compile doc
pre-run: playbooks/tox-docs/pre.yaml
run: playbooks/tox-docs/run.yaml
post-run:
- playbooks/tox-docs/post.yaml
- playbooks/tox-docs/publish.yaml
- job:
name: opendev-publish-unversioned-tox-docs
parent: opendev-publish-tox-docs-base
description: |
Publish a ref-based documentation build to "latest" always.
Use this in the tag or release pipelines to publish a build
based on a newly-created tag but published as if it were a
branch tip build instead. This is useful when the project has
only a single branch and multiple version-specific copies of
its documentation are undesirable, or when triggered by a tag
alongside a version-specific publication job.
post-run: playbooks/tox-docs/publish.yaml
secrets:
- secret: opendev-unversioned-docs
name: afs
pass-to-parent: true
- job:
name: opendev-publish-tox-docs
parent: opendev-publish-tox-docs-base
description: |
Publish a ref-based documentation build.
Use this in the tag or release pipelines to publish a build
based on a newly-created tag.
post-run: playbooks/tox-docs/publish.yaml
secrets:
- secret: opendev-zuul-docs
name: afs
pass-to-parent: true
- job:
name: opendev-promote-docs-base
description: |
Publish a previously built branch-tip documentation tarball.
Use this in the promote pipeline to publish a branch tip tarball
built in the gate pipeline.
This is an abstract job intended to be inherited from in an
OpenDev tenant and an appropriate secret added.
.. zuul:jobvar:: afs
:type: dict
This is expected to be a Zuul Secret with these keys:
.. zuul:jobvar:: keytab
The AFS keytab for the service principal.
.. zuul:jobvar:: service_name
The name of the service princpal.
.. zuul:jobvar:: targets
This is a dict containing information about where docs should be
published.
.. zuul:jobvar:: master
This is expected to be a dict with a single key value pair:
`path: the full docs publication path to use if the job is
run on the master branch.`
.. zuul:jobvar:: branch
This is expected to be a dict with a key value pair:
`path: the full docs publication path to use if the job is run on
any other branch.`
.. zuul:jobvar:: tag
This is expected to be a dict with a key value pair:
`path: the full docs publication path to use if the job is run on
a tag.`
.. zuul:jobvar:: docs_redirect_path
If this variable is present, a .htaccess redirect will be
created at this path when the job is run on the master
branch. For example, it can be used to redirect "project/"
to "project/latest".
.. zuul:jobvar:: docs_redirect_content
The contents of the .htaccess file in docs_redirect_path.
.. zuul:jobvar:: download_artifact_job
The name of the job which built the docs artifact which this
job should download and promote.
.. zuul:jobvar:: write_root_marker
If this is set to false, then the root marker file is not
written.
Warning: setting this parameter incorrectly can result in loss of published data.
abstract: True
run: playbooks/docs/promote.yaml
vars:
write_root_marker: true
nodeset:
nodes: []
- job:
name: opendev-promote-docs
parent: opendev-promote-docs-base
description: |
Publish a previously built branch-tip documentation tarball.
Use this in the promote pipeline to publish a branch tip tarball
built in the gate pipeline.
The documentation tarball is published to
https://docs.opendev.org/{{ zuul.project.name }}.
Publishes depending on branch to latest/ (for master), or the
basename of the branch like train (for stable/train).
vars:
download_artifact_job: opendev-tox-docs
secrets:
- secret: opendev-zuul-docs
name: afs
pass-to-parent: true
- job:
name: opendev-promote-artifact-base
description: |
Publish a previously built branch-tip artifact.
Use this in the promote pipeline to publish a branch tip artifact
built in the gate pipeline.
This is an abstract job intended to be inherited from in an
OpenDev tenant and an appropriate secret added.
.. zuul:jobvar:: afs
:type: dict
This is expected to be a Zuul Secret with these keys:
.. zuul:jobvar:: keytab
The AFS keytab for the service principal.
.. zuul:jobvar:: service_name
The name of the service princpal.
.. zuul:jobvar:: artifacts_path
The full publication path to use.
.. zuul:jobvar:: download_artifact_job
The name of the job which built the artifacts which this
job should download and promote.
.. zuul:jobvar:: download_artifact_type
The type of the artifact to download (as specified in the
``type`` attribute of the artifact metadata).
.. zuul:jobvar:: artifact_extra_name
The artifact will be renamed to PROJECT-BRANCH.ext; if this
argument is present, it will be PROJECT-EXTRA-BRANCH.ext.
abstract: True
run: playbooks/artifacts/promote.yaml
nodeset:
nodes: []
- job:
name: opendev-promote-python
parent: opendev-promote-artifact-base
description: |
Publish a previously built branch-tip sdist/wheels.
Use this in the promote pipeline to publish a branch tip
sdist and wheel(s) built in the gate pipeline.
vars:
download_artifact_job: build-python-release
download_artifact_type:
- python_sdist
- python_wheel
secrets:
- secret: opendev-zuul-tarballs
name: afs
pass-to-parent: true
- job:
name: opendev-promote-javascript-content
parent: opendev-promote-artifact-base
description: |
Publish a previously built branch-tip javascript content archive.
Use this in the promote pipeline to publish a branch tip
javascript content archive built in the gate pipeline.
vars:
download_artifact_job: build-javascript-content-tarball
download_artifact_type: javascript_content
artifact_extra_name: js-content
secrets:
- secret: opendev-zuul-tarballs
name: afs
pass-to-parent: true
- job:
name: opendev-promote-javascript-deployment-tarball
parent: opendev-promote-artifact-base
description: |
Publish a previously built branch-tip javascript content archive.
Use this in the promote pipeline to publish a branch tip
javascript content archive built in the gate pipeline.
vars:
download_artifact_job: build-javascript-deployment
download_artifact_type: javascript_content
artifact_extra_name: js-content
secrets:
- secret: opendev-zuul-tarballs
name: afs
pass-to-parent: true
- job:
name: opendev-promote-javascript-deployment
parent: opendev-promote-artifact-base
description: |
Publish previously built branch-tip javascript content
Use this in the promote pipeline to publish branch tip
javascript content built in the gate pipeline. Expects
a tarball to have been published which will be extracted
into the target location.
.. zuul:jobvar:: download_artifact_job
The name of the job which built the artifacts which this
job should download and promote.
.. zuul:jobvar:: download_artifact_type
The type of the artifact to download (as specified in the
``type`` attribute of the artifact metadata).
run: playbooks/artifacts/promote-deployment.yaml
vars:
download_artifact_job: build-javascript-deployment
download_artifact_type: javascript_content
secrets:
- secret: opendev-zuul-tarballs
name: afs
- job:
name: opendev-release-python
description: Release python tarballs / wheels to pypi.
pre-run: playbooks/release-python/pre.yaml
run: playbooks/release-python/run.yaml
post-run: playbooks/release-python/post.yaml
secrets:
- secret: opendev-pypi
name: pypi_info
- job:
name: opendev-upload-git-mirror
description: |
Mirrors a tested project repository to a remote git server. This is a
nodeless version of the job in zuul-jobs, defined here since this job's
playbook must be in a trusted repo.
.. zuul:jobvar:: git_mirror_credentials
:type: dict
This is expected to be a Zuul secret with these keys:
.. zuul:jobvar:: user
SSH user for the remote git repository
.. zuul:jobvar:: host
SSH host for the remote git repository
.. zuul:jobvar:: ssh_key
Literal private key contents.
Should start with something like ``-----BEGIN RSA PRIVATE KEY-----``.
.. zuul:jobvar:: host_key
SSH host key of the remote git server.
Can be obtained with ``ssh-keyscan -H <host>``.
.. zuul:jobvar:: target_repository
Path of the remote git repository
run: playbooks/upload-git-mirror/run.yaml
nodeset:
nodes: []
- job:
name: opendev-infra-prod-base
description: |
A base job for running production playbooks on OpenDev's bridge.
This is not for general use.
abstract: true
pre-run: playbooks/infra-prod/pre.yaml
- job:
name: opendev-infra-prod-setup-src
description: |
A base job for replicating source to OpenDev's bridge.
This is not for general use.
abstract: true
pre-run:
- playbooks/infra-prod/setup-keys.yaml
- playbooks/infra-prod/setup-src.yaml
- job:
name: opendev-infra-prod-setup-keys
description: |
A base job for allowing executors to log into OpenDev's bridge.
This is not for general use.
abstract: true
pre-run: playbooks/infra-prod/setup-keys.yaml
View Git Blame Copy Permalink