This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context: * :zuul:role:`build-container-image`: Build the images. * :zuul:role:`upload-container-image`: Upload the images to a registry. * :zuul:role:`promote-container-image`: Promote previously uploaded images. All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors_ can be used to supply the same data to all three roles. *Building* The :zuul:role:`build-container-image` role is designed to be used in `check` and `gate` pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry. *Uploading* The :zuul:role:`upload-container-image` role uploads the images to a registry. It can be used in one of two modes: 1. Using tags as part of a two-step `promote` pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository. In this mode, the role is intended to run in the `gate` pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. ``change_12345_``). Thus at the completion of the `gate` job, all the layers of the new container are uploaded, but the ```` in the remote repository will not have been updated. Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the ```` is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the `promote` pipeline which uses a very quick, executor-only job to retag ```` to ``change_12345_``. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by ```` will now reflect the underlying code closing the out-of-sync window. When running in this mode uploads are only made if ``promote_container_image_method`` is unset or set to ``tag``. Otherwise we skip upload to the registry. 2. The second mode allows for use of this job in `release` and `tag` pipelines to directly upload a release build with the final set of tags. In this mode, ``upload_container_image_promote: false`` should be set. The role will build and upload the resulting image to the remote repository with the final tags. This should be used with `tag` and `release` pipelines, where committed code has been tagged for publishing. The tagged commit is "known good" thanks to gating, so the build and upload process is expected to work unconditionally. This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the `gate` job. The alternative `promote` method can be thought of as a "speculative" upload. There is a possibility the `gate` job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the `gate` jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the `release` pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags. Transient network failures can cause upload errors in both cases. Although the `promote` job may fail, leaving the tag incorrectly unmodified, the `promote` job's relatively simplicity minimises potential error. The `release` pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure. *Promoting* As discussed above, the :zuul:role:`promote-container-image` role is designed to be used in a `promote` pipeline. In ``tag`` mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by :zuul:rolevar:`build-container-image.container_images.tags`. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor. Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the ``api_token`` requirements for your registry described below. For more details see :zuul:role:`remove-registry-tag`. In ``intermediate-registry`` mode, this role queries Zuul to find the build performed by the build role in the ``gate``. It then copies this image from the intermediate-registry to the final location in the remote registry. *Dependencies* The build and upload roles require a container runtime that should be installed before use; for example by using either the :zuul:role:`ensure-docker` or :zuul:role:`ensure-podman` roles. The promote job assumes `skopeo` is available on the executor. **Role Variables** .. zuul:rolevar:: zuul_work_dir :default: {{ zuul.project.src_dir }} The project directory. Serves as the base for :zuul:rolevar:`build-container-image.container_images.context`. .. zuul:rolevar:: container_filename The default container filename name to use. Serves as the base for :zuul:rolevar:`build-container-image.container_images.container_filename`. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles. If omitted, the default depends on the container command used. Typically, this is ``Dockerfile`` for ``docker`` and ``Containerfile`` (with a fallback on ``Dockerfile``) for ``podman``. .. zuul:rolevar:: container_command :default: podman The command to use when building the image (E.g., ``docker``). .. zuul:rolevar:: container_registry_credentials :type: dict This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry. Example: .. code-block:: yaml container_registry_credentials: quay.io: username: foo password: bar .. zuul:rolevar:: [registry name] :type: dict Information about a registry. The key is the registry name, and its value a dict as follows: .. zuul:rolevar:: username The registry username. .. zuul:rolevar:: password The registry password. .. zuul:rolevar:: repository Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:: repository: "^myorgname/{{ zuul.project.short_name }}.*" .. zuul:rolevar:: api_token Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see :zuul:role:`remove-registry-tag`. Some examples: * **docker** : API is access via username/password, does not require token. * **quay.io** : A token must be generated from an "application" that a user has allowed to operate on its behalf. See ``__. .. zuul:rolevar:: container_images :type: list A list of images to build. Each item in the list should have: .. zuul:rolevar:: context The build context; this should be a directory underneath :zuul:rolevar:`build-container-image.zuul_work_dir`. .. zuul:rolevar:: container_filename The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image. .. zuul:rolevar:: registry The name of the target registry (E.g., ``quay.io``). Used by the upload and promote roles. .. zuul:rolevar:: repository The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., ``quay.io/example/image``. .. zuul:rolevar:: path Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere. .. zuul:rolevar:: build_args :type: list Optional: a list of values to pass to the ``--build-arg`` parameter. .. zuul:rolevar:: target Optional: the target for a multi-stage build. .. zuul:rolevar:: tags :type: list :default: ['latest'] A list of tags to be added to the image when promoted. .. zuul:rolevar:: siblings :type: list :default: [] A list of sibling projects to be copied into ``{{zuul_work_dir}}/.zuul-siblings``. This can be useful to collect multiple projects to be installed within the same Docker context. A ``-build-arg`` called ``ZUUL_SIBLINGS`` will be added with each sibling project. Note that projects here must be listed in ``required-projects``. .. zuul:rolevar:: container_build_extra_env :type: dict A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example. .. zuul:rolevar:: promote_container_image_method :default: tag A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (``tag``) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (``intermediate-registry``). .. _anchors: https://yaml.org/spec/1.2/spec.html#&%20anchor//