diff --git a/doc/source/docker-image.rst b/doc/source/docker-image.rst index 0ed67e8..288a9a2 100644 --- a/doc/source/docker-image.rst +++ b/doc/source/docker-image.rst @@ -169,3 +169,11 @@ Jobs .. zuul:autojob:: opendev-upload-docker-image .. zuul:autojob:: opendev-promote-docker-image + +.. zuul:autojob:: opendev-build-container-image-base + +.. zuul:autojob:: opendev-build-container-image + +.. zuul:autojob:: opendev-upload-container-image + +.. zuul:autojob:: opendev-promote-container-image diff --git a/playbooks/container-image/README.rst b/playbooks/container-image/README.rst new file mode 100644 index 0000000..4da7489 --- /dev/null +++ b/playbooks/container-image/README.rst @@ -0,0 +1,125 @@ +This is one of a collection of jobs which are designed to work +together to build, upload, and promote container images in a gating +context: + + * :zuul:job:`opendev-build-container-image`: Build the images. + * :zuul:job:`opendev-upload-container-image`: Build and stage the images in a registry. + * :zuul:job:`opendev-promote-container-image`: Promote previously uploaded images. + +The :zuul:job:`opendev-build-container-image` job is designed to be used in +a `check` pipeline and simply builds the images to verify that +the build functions. + +The :zuul:job:`opendev-upload-container-image` job builds and uploads the +images to a registry, but only with a single tag corresponding to the +change ID. This job is designed in a `gate` pipeline so that the +build produced by the gate is staged and can later be promoted to +production if the change is successful. + +The :zuul:job:`opendev-promote-container-image` job is designed to be +used in a `promote` pipeline. It requires no nodes and runs very +quickly on the Zuul executor. It simply re-tags a previously uploaded +image for a change with whatever tags are supplied by +:zuul:jobvar:`opendev-build-container-image.container_images.tags`. +It also removes the change ID tag from the repository in the registry. +If any changes fail to merge, this cleanup will not run and those tags +will need to be deleted manually. + +They all accept the same input data, principally a list of +dictionaries representing the images to build. YAML anchors_ can be +used to supply the same data to all three jobs. + +**Job Variables** + +.. zuul:jobvar:: zuul_work_dir + :default: {{ zuul.project.src_dir }} + + The project directory. Serves as the base for + :zuul:jobvar:`opendev-build-container-image.container_images.context`. + +.. zuul:jobvar:: container_filename + + The default container filename name to use. Serves as the base for + :zuul:jobvar:`opendev-build-container-image.container_images.container_filename`. + This allows a global overriding of the container filename name, for + example when building all images from different folders with + similarily named containerfiles. + + If omitted, the default depends on the container command used. + Typically, this is ``Dockerfile`` for ``docker`` and + ``Containerfile`` (with a fallback on ``Dockerfile``) for + ``podman``. + +.. zuul:jobvar:: container_command + :default: podman + + The command to use when building the image (E.g., ``docker``). + +.. zuul:jobvar:: container_images + :type: list + + A list of images to build. Each item in the list should have: + + .. zuul:jobvar:: context + + The build context; this should be a directory underneath + :zuul:jobvar:`opendev-build-container-image.zuul_work_dir`. + + .. zuul:jobvar:: container_filename + + The filename of the container file, present in the context + folder, used for building the image. Provide this if you are + using a non-standard filename for a specific image. + + .. zuul:jobvar:: registry + + The name of the target registry (E.g., ``quay.io``). Used by + the upload and promote roles. + + .. zuul:jobvar:: repository + + The name of the target repository in the registry for the image. + Supply this even if the image is not going to be uploaded (it + will be tagged with this in the local registry). This should + include the registry name. E.g., ``quay.io/example/image``. + + .. zuul:jobvar:: path + + Optional: the directory that should be passed to the build + command. Useful for building images with a container file in + the context directory but a source repository elsewhere. + + .. zuul:jobvar:: build_args + :type: list + + Optional: a list of values to pass to the ``--build-arg`` + parameter. + + .. zuul:jobvar:: target + + Optional: the target for a multi-stage build. + + .. zuul:jobvar:: tags + :type: list + :default: ['latest'] + + A list of tags to be added to the image when promoted. + + .. zuul:jobvar:: siblings + :type: list + :default: [] + + A list of sibling projects to be copied into + ``{{zuul_work_dir}}/.zuul-siblings``. This can be useful to + collect multiple projects to be installed within the same Docker + context. A ``-build-arg`` called ``ZUUL_SIBLINGS`` will be + added with each sibling project. Note that projects here must + be listed in ``required-projects``. + +.. zuul:jobvar:: container_build_extra_env + :type: dict + + A dictionary of key value pairs to add to the container build environment. + This may be useful to enable buildkit with docker builds for example. + +.. _anchors: https://yaml.org/spec/1.2/spec.html#&%20anchor// diff --git a/playbooks/container-image/credentials.rst b/playbooks/container-image/credentials.rst new file mode 100644 index 0000000..a624d8e --- /dev/null +++ b/playbooks/container-image/credentials.rst @@ -0,0 +1,40 @@ +.. zuul:rolevar:: container_registry_credentials + :type: dict + + This is only required for the upload and promote roles. This is + expected to be a Zuul Secret in dictionary form. Each key is the + name of a registry, and its value a dictionary with information + about that registry. + + Example: + + .. code-block:: yaml + + container_registry_credentials: + quay.io: + username: foo + password: bar + + .. zuul:rolevar:: [registry name] + :type: dict + + Information about a registry. The key is the registry name, and + its value a dict as follows: + + .. zuul:rolevar:: username + + The registry username. + + .. zuul:rolevar:: password + + The registry password. + + .. zuul:rolevar:: repository + + Optional; if supplied this is a regular expression which + restricts to what repositories the image may be uploaded. + The registry name should be included. The following example + allows projects to upload images to repositories within an + organization based on their own names:: + + repository: "^quay.io/myorgname/{{ zuul.project.short_name }}.*" diff --git a/playbooks/container-image/pre.yaml b/playbooks/container-image/pre.yaml new file mode 100644 index 0000000..4d952ed --- /dev/null +++ b/playbooks/container-image/pre.yaml @@ -0,0 +1,3 @@ +- hosts: localhost + roles: + - pull-from-intermediate-registry diff --git a/playbooks/container-image/run.yaml b/playbooks/container-image/run.yaml new file mode 100644 index 0000000..c2d40d3 --- /dev/null +++ b/playbooks/container-image/run.yaml @@ -0,0 +1,11 @@ +- hosts: all + roles: + - build-container-image + +# If buildset_registry is defined, that means a parent job is running it; +# only if it is not defined does it mean that we are running it. If we +# are running it, pause the job so that child jobs will automatically +# use it. +- hosts: localhost + roles: + - pause-buildset-registry diff --git a/playbooks/container-image/upload.yaml b/playbooks/container-image/upload.yaml new file mode 100644 index 0000000..36f9909 --- /dev/null +++ b/playbooks/container-image/upload.yaml @@ -0,0 +1,3 @@ +- hosts: all + roles: + - upload-container-image diff --git a/zuul.d/jobs.yaml b/zuul.d/jobs.yaml index 9766377..d721fa9 100644 --- a/zuul.d/jobs.yaml +++ b/zuul.d/jobs.yaml @@ -211,6 +211,62 @@ .. include:: ../../playbooks/docker-image/README.rst .. include:: ../../playbooks/docker-image/credentials.rst +- job: + name: opendev-build-container-image-base + parent: opendev-buildset-registry + description: | + This is a parent for an image build job which expects a + buildset registry to be running and pulls images from the + intermediate registry into it. It mostly exists so that + the intermediate registry secret need not be supplied to the + image build playbook. + pre-run: playbooks/container-image/pre.yaml + secrets: + - secret: opendev-intermediate-registry + name: intermediate_registry + +- job: + name: opendev-build-container-image + parent: opendev-build-container-image-base + description: | + Starts a buildset registry (if one has not already been started, + e.g., by invoking :zuul:job:`opendev-buildset-registry` and + specifying it as a dependency) and builds one or more docker + images. + + Analog of build-docker-image job, but with a buildset registry. + + This job will pause after starting the registry so that it is + available to any jobs which depend on it. Once all such jobs + are complete, this job will finish. + + .. include:: ../../playbooks/container-image/README.rst + run: playbooks/container-image/run.yaml + +- job: + name: opendev-upload-container-image + parent: opendev-build-container-image + description: | + Starts a buildset registry and builds and uploads one or more + container images to a registry. + + Analog of upload-container-image job, but with a buildset registry. + + .. include:: ../../playbooks/container-image/README.rst + .. include:: ../../playbooks/container-image/credentials.rst + post-run: playbooks/container-image/upload.yaml + +- job: + name: opendev-promote-container-image + parent: promote-container-image + description: | + Retag a previously-uploaded container image. + + Analog of promote-container-image job. + + .. include:: ../../playbooks/container-image/README.rst + .. include:: ../../playbooks/container-image/credentials.rst + - job: name: opendev-buildset-registry-consumer description: |