Merge "Add promote-container-image role"

doc/source/container-roles.rst

@@ -11,6 +11,7 @@ Container Roles
 .. zuul:autorole:: ensure-podman
 .. zuul:autorole:: ensure-skopeo
 .. zuul:autorole:: pause-buildset-registry
+.. zuul:autorole:: promote-container-image
 .. zuul:autorole:: promote-docker-image
 .. zuul:autorole:: pull-from-intermediate-registry
 .. zuul:autorole:: push-to-intermediate-registry

roles/build-container-image/common.rst

@@ -3,20 +3,40 @@ together to build, upload, and promote container images in a gating
 context:
 
 * :zuul:role:`build-container-image`: Build the images.
-
+* :zuul:role:`upload-container-image`: Upload the images to a registry.
-.. note:: Build and upload roles are forthcoming.
+* :zuul:role:`promote-container-image`: Promote previously uploaded images.
 
 The :zuul:role:`build-container-image` role is designed to be used in
 `check` and `gate` pipelines and simply builds the images.  It can be
 used to verify that the build functions, or it can be followed by the
 use of subsequent roles to upload the images to a registry.
 
+The :zuul:role:`upload-container-image` role uploads the images to a
+registry.  It can be used in one of two modes: by default it will
+upload with a single tag corresponding to the change ID.  In this
+mode, the role role is designed to be used in a job in a `gate`
+pipeline so that the build produced by the gate is staged and can
+later be promoted to production if the change is successful.  The
+other mode allows for use of this job in a `release` pipeline to
+directly upload a release build with the final set of tags.
+
+The :zuul:role:`promote-container-image` role is designed to be used
+in a `promote` pipeline.  It requires no nodes and runs very quickly
+on the Zuul executor.  It simply re-tags a previously uploaded image
+for a change with whatever tags are supplied by
+:zuul:rolevar:`build-container-image.container_images.tags`.  It also
+removes the change ID tag from the repository in the registry, and
+removes any similar change ID tags.  This keeps the repository tidy in
+the case that gated changes fail to merge after uploading their staged
+images.
+
 They all accept the same input data, principally a list of
 dictionaries representing the images to build.  YAML anchors_ can be
 used to supply the same data to all three jobs.
 
-Use the :zuul:role:`ensure-docker` or :zuul:role:`ensure-podman`
+Use the :zuul:role:`ensure-skopeo` role as well as the
-role to install Docker or Podman before using these roles.
+:zuul:role:`ensure-docker`, or :zuul:role:`ensure-podman` roles before
+using the roles described here.
 
 **Role Variables**
 
@@ -109,7 +129,8 @@ role to install Docker or Podman before using these roles.
 
       The name of the target repository in the registry for the image.
       Supply this even if the image is not going to be uploaded (it
-      will be tagged with this in the local registry).
+      will be tagged with this in the local registry).  This should
+      include the registry name.  E.g., ``quay.io/example/image``.
 
    .. zuul:rolevar:: path
 

roles/build-docker-image/common.rst

@@ -3,7 +3,7 @@ together to build, upload, and promote docker images in a gating
 context:
 
 * :zuul:role:`build-docker-image`: Build the images.
-* :zuul:role:`upload-docker-image`: Stage the images on dockerhub.
+* :zuul:role:`upload-docker-image`: Upload the images to Dockeer Hub.
 * :zuul:role:`promote-docker-image`: Promote previously uploaded images.
 
 The :zuul:role:`build-docker-image` role is designed to be used in
@@ -12,10 +12,13 @@ used to verify that the build functions, or it can be followed by the
 use of subsequent roles to upload the images to Docker Hub.
 
 The :zuul:role:`upload-docker-image` role uploads the images to Docker
-Hub, but only with a single tag corresponding to the change ID.  This
+Hub.  It can be used in one of two modes: by default it will upload
-role is designed to be used in a job in a `gate` pipeline so that the
+with a single tag corresponding to the change ID.  In this mode, the
-build produced by the gate is staged and can later be promoted to
+role role is designed to be used in a job in a `gate` pipeline so that
-production if the change is successful.
+the build produced by the gate is staged and can later be promoted to
+production if the change is successful.  The other mode allows for use
+of this job in a `release` pipeline to directly upload a release build
+with the final set of tags.
 
 The :zuul:role:`promote-docker-image` role is designed to be used in a
 `promote` pipeline.  It requires no nodes and runs very quickly on the

roles/promote-container-image/README.rst

@@ -0,0 +1,3 @@
+Promote one or more previously uploaded container images.
+
+.. include:: ../../roles/build-container-image/common.rst

roles/promote-container-image/defaults/main.yaml

@@ -0,0 +1 @@
+zuul_work_dir: "{{ zuul.project.src_dir }}"

roles/promote-container-image/tasks/main.yaml

@@ -0,0 +1,31 @@
+- name: Verify repository names
+  when: |
+    container_registry_credentials is defined
+    and zj_image.registry not in container_registry_credentials
+  loop: "{{ container_images }}"
+  loop_control:
+    loop_var: zj_image
+  fail:
+    msg: "{{ zj_image.registry }} credentials not found"
+
+- name: Log in to registry
+  no_log: true
+  command: >-
+    skopeo login {{ zj_image.registry }} -u {{ container_registry_credentials[zj_image.registry].username }} -p {{ container_registry_credentials[zj_image.registry].password }}
+  register: result
+  until: result.rc == 0
+  retries: 3
+  delay: 30
+
+- name: Promote image
+  loop: "{{ container_images }}"
+  loop_control:
+    loop_var: zj_image
+  include_tasks: promote-retag.yaml
+
+# The docker roles prune obsolete tags here, but that relies on a
+# timestamp to make sure we're not deleting in-progress tags (that the
+# gate pipeline may be uploading at the same time we're promoting).
+# That timestamp is not available with skopeo list-tags, so some other
+# mechanism will need to be devised to clean them up.  In the
+# meantime, we hope that the cleanup in promote-retag succeeds.

roles/promote-container-image/tasks/promote-retag-inner.yaml

@@ -0,0 +1,19 @@
+- name: Set promote_tag_prefix
+  set_fact:
+    promote_tag_prefix: "{{ ('change_' + zuul.change) if (zuul.change is defined) else zuul.pipeline }}"
+
+- name: Create the final image tag
+  command: >-
+    skopeo copy --all docker://{{ zj_image.repository }}:{{ promote_tag_prefix }}_{{ zj_image_tag }} docker://{{ zj_image.repository }}:{{ zj_image_tag }}
+  register: result
+  until: result.rc == 0
+  retries: 3
+  delay: 30
+
+- name: Delete the current change tag
+  command: >-
+    skopeo delete docker://{{ zj_image.repository }}:{{ promote_tag_prefix }}_{{ zj_image_tag }}
+  register: result
+  until: result.rc == 0
+  retries: 3
+  delay: 30

roles/promote-container-image/tasks/promote-retag.yaml

@@ -0,0 +1,5 @@
+- name: Retag image
+  loop: "{{ zj_image.tags | default(['latest']) }}"
+  loop_control:
+    loop_var: zj_image_tag
+  include_tasks: promote-retag-inner.yaml

test-playbooks/container/test-build-container-image-inner.yaml

@@ -0,0 +1,59 @@
+- name: Set image variables
+  set_fact:
+    _arch_docker_images:
+      - context: test-playbooks/container/docker
+        repository: "testrepo"
+        # This is what the Zuul repo uses to tag its releases:
+        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
+        arch: ['linux/amd64', 'linux/arm64']
+    _normal_docker_images:
+      - context: test-playbooks/container/docker
+        repository: "testrepo"
+        # This is what the Zuul repo uses to tag its releases:
+        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
+    container_images:
+      - context: test-playbooks/container/docker
+        registry: 127.0.0.1:5200
+        repository: 127.0.0.1:5200/testrepo
+        # This is what the Zuul repo uses to tag its releases:
+        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
+
+- name: Save zuul variables
+  set_fact:
+    old_zuul: "{{ zuul }}"
+
+- name: Set simulated zuul variables
+  set_fact:
+    new_zuul:
+      pipeline: "{{ old_zuul.pipeline }}"
+      change_url: "{{ old_zuul.change_url }}"
+      executor: "{{ old_zuul.executor }}"
+      newrev: c12f3fe1defe8b61d59061363c9c04fb520dae18
+      project: "{{ old_zuul.project }}"
+      ref: refs/tags/3.19.0
+      tag: 3.19.0
+
+- name: "Build a container image"
+  include_role:
+    name: "build-{{ (container_command == 'docker') | ternary('docker', 'container') }}-image"
+  vars:
+    zuul: "{{ new_zuul }}"
+    docker_images: "{{ multiarch | ternary(_arch_docker_images, _normal_docker_images) }}"
+
+- name: "Upload container image"
+  include_role:
+    name: "upload-{{ (container_command == 'docker') | ternary('docker', 'container') }}-image"
+  vars:
+    zuul: "{{ new_zuul }}"
+    docker_images: "{{ multiarch | ternary(_arch_docker_images, _normal_docker_images) }}"
+
+- name: "Upload container image"
+  when: "upload_container_image_promote or upload_docker_image_promote"
+  include_role:
+    name: "upload-{{ (container_command == 'docker') | ternary('docker', 'container') }}-image"
+  vars:
+    zuul: "{{ new_zuul }}"
+    docker_images: "{{ multiarch | ternary(_arch_docker_images, _normal_docker_images) }}"
+
+- name: "Show local container images for debugging"
+  command: "{{ container_command }} image ls"

test-playbooks/container/test-build-container-image-pre.yaml

@@ -18,6 +18,10 @@
       include_role:
         name: "ensure-{{ container_command }}"
 
+    - name: Install skopeo
+      include_role:
+        name: "ensure-skopeo"
+
     - name: Enable insecure registries
       block:
         - name: Create podman configration directory

test-playbooks/container/test-build-container-image-promote.yaml

@@ -0,0 +1,7 @@
+- hosts: all
+  vars:
+    upload_container_image_promote: true
+    upload_docker_image_promote: true
+  tasks:
+    - name: Include build tasks
+      include_tasks: test-build-container-image-inner.yaml

test-playbooks/container/test-build-container-image-release.yaml

@@ -2,51 +2,6 @@
   vars:
     upload_container_image_promote: false
     upload_docker_image_promote: false
-    _arch_docker_images:
-      - context: test-playbooks/container/docker
-        repository: "testrepo"
-        # This is what the Zuul repo uses to tag its releases:
-        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
-        arch: ['linux/amd64', 'linux/arm64']
-    _normal_docker_images:
-      - context: test-playbooks/container/docker
-        repository: "testrepo"
-        # This is what the Zuul repo uses to tag its releases:
-        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
-    container_images:
-      - context: test-playbooks/container/docker
-        registry: 127.0.0.1:5200
-        repository: 127.0.0.1:5200/testrepo
-        # This is what the Zuul repo uses to tag its releases:
-        tags: "{{ zuul.tag is defined | ternary([zuul.get('tag', '').split('.')[0], '.'.join(zuul.get('tag', '').split('.')[:2]), zuul.get('tag', '')], ['latest']) }}"
   tasks:
-    - name: Save zuul variables
+    - name: Include build tasks
-      set_fact:
+      include_tasks: test-build-container-image-inner.yaml
-        old_zuul: "{{ zuul }}"
-    - name: Set simulated zuul variables
-      set_fact:
-        new_zuul:
-          pipeline: "{{ old_zuul.pipeline }}"
-          change_url: "{{ old_zuul.change_url }}"
-          executor: "{{ old_zuul.executor }}"
-          newrev: c12f3fe1defe8b61d59061363c9c04fb520dae18
-          project: "{{ old_zuul.project }}"
-          ref: refs/tags/3.19.0
-          tag: 3.19.0
-
-    - name: "Build a container image"
-      include_role:
-        name: "build-{{ (container_command == 'docker') | ternary('docker', 'container') }}-image"
-      vars:
-        zuul: "{{ new_zuul }}"
-        docker_images: "{{ multiarch | ternary(_arch_docker_images, _normal_docker_images) }}"
-
-    - name: "Upload container image"
-      include_role:
-        name: "upload-{{ (container_command == 'docker') | ternary('docker', 'container') }}-image"
-      vars:
-        zuul: "{{ new_zuul }}"
-        docker_images: "{{ multiarch | ternary(_arch_docker_images, _normal_docker_images) }}"
-
-    - name: "Show local container images for debugging"
-      command: "{{ container_command }} image ls"

zuul-tests.d/container-roles-jobs.yaml

@@ -118,7 +118,7 @@
       - roles/build-container-image/.*
       - roles/upload-container-image/.*
       - test-playbooks/container/.*
-    pre-run: test-playbooks/container/test-build-container-image-release-pre.yaml
+    pre-run: test-playbooks/container/test-build-container-image-pre.yaml
     run: test-playbooks/container/test-build-container-image-release.yaml
     vars:
       container_command: podman
@@ -128,6 +128,33 @@
         - name: builder
           label: ubuntu-jammy
 
+- job:
+    name: zuul-jobs-test-build-container-image-promote
+    description: |
+      Test building a container image in gate and promote pipelines.
+
+      This job tests changes to the build-container-image role,
+      simulating its use within a tag-based release pipeline.  It is
+      not meant to be used directly but rather run on changes to roles
+      in the zuul-jobs repo.
+    files:
+      - roles/ensure-podman/.*
+      - roles/ensure-skopeo/.*
+      - roles/ensure-package-repositories/.*
+      - roles/build-container-image/.*
+      - roles/upload-container-image/.*
+      - roles/promote-container-image/.*
+      - test-playbooks/container/.*
+    pre-run: test-playbooks/container/test-build-container-image-pre.yaml
+    run: test-playbooks/container/test-build-container-image-promote.yaml
+    vars:
+      container_command: podman
+      multiarch: false
+    nodeset:
+      nodes:
+        - name: builder
+          label: ubuntu-jammy
+
 - job:
     name: zuul-jobs-test-build-docker-image-release
     description: |
@@ -143,7 +170,7 @@
       - roles/build-docker-image/.*
       - roles/upload-docker-image/.*
       - test-playbooks/container/.*
-    pre-run: test-playbooks/container/test-build-container-image-release-pre.yaml
+    pre-run: test-playbooks/container/test-build-container-image-pre.yaml
     run: test-playbooks/container/test-build-container-image-release.yaml
     vars:
       container_command: docker
@@ -571,6 +598,7 @@
         - zuul-jobs-test-ensure-docker-ubuntu-focal
         - zuul-jobs-test-ensure-docker-ubuntu-jammy
         - zuul-jobs-test-build-container-image-release
+        - zuul-jobs-test-build-container-image-promote
         - zuul-jobs-test-build-docker-image-release
         - zuul-jobs-test-build-docker-image-release-multiarch
         - zuul-jobs-test-registry-docker
@@ -604,6 +632,7 @@
         - zuul-jobs-test-ensure-docker-ubuntu-focal
         - zuul-jobs-test-ensure-docker-ubuntu-jammy
         - zuul-jobs-test-build-container-image-release
+        - zuul-jobs-test-build-container-image-promote
         - zuul-jobs-test-build-docker-image-release
         - zuul-jobs-test-build-docker-image-release-multiarch
         - zuul-jobs-test-registry-docker