CI: Build default images in experimental jobs

This commit changes seed-images jobs to build seed and overcloud
container images, each with a customisable regex.

The rocky9 job continues to build only the base image, although for both
overcloud and seed instead of seed only.

The other seed-images jobs (centos9s and ubuntu-jammy) now build all
default overcloud images. This takes longer and can fail for various
reasons unrelated to Kayobe, but this is fine since the jobs are marked
as experimental.

This is to attempt to catch bugs in the image build code, such as the
ones fixed by [1].

Note that the ubuntu-jammy job does not build the bifrost-deploy seed
image because the build is currently timing out. The centos9s job builds
it successfully.

Also update release documentation to remind developers to run
experimental jobs.

[1] https://review.opendev.org/c/openstack/kayobe/+/921012

Change-Id: Idf705d9cf41766a897444898ac31c4635b70ec16

doc/source/contributor/releases.rst

@@ -55,6 +55,10 @@ Testing
 
 Test the code and fix at a minimum all critical issues.
 
+Remember to use the ``check experimental`` review comment on a Gerrit change to
+trigger jobs in the experimental pipeline, which provide increased test
+coverage.
+
 Synchronise with Kolla Ansible feature flags
 --------------------------------------------
 

playbooks/kayobe-seed-base/run.yml

@@ -22,10 +22,17 @@
             group: stack
             remote_src: true
 
-        - name: Ensure base container image is built
+        - name: Ensure seed container images are built
           shell:
             cmd: >
               source {{ kayobe_src_dir }}/dev/environment-setup.sh &&
-              kayobe seed container image build ^base &> {{ logs_dir }}/ansible/container-image-build
+              kayobe seed container image build {{ seed_container_image_regex | default('') }} &> {{ logs_dir }}/ansible/seed-container-image-build
             executable: /bin/bash
-      when: build_images
+
+        - name: Ensure overcloud container images are built
+          shell:
+            cmd: >
+              source {{ kayobe_src_dir }}/dev/environment-setup.sh &&
+              kayobe overcloud container image build {{ overcloud_container_image_regex | default('') }} &> {{ logs_dir }}/ansible/overcloud-container-image-build
+            executable: /bin/bash
+      when: build_images | bool

zuul.d/jobs.yaml

@@ -238,15 +238,25 @@
     nodeset: kayobe-centos9s
     voting: false
 
+# Build only the base container image in the kayobe-seed-images-rocky9 job
+# which always runs. Use `check experimental` to run the other jobs which build
+# more images.
 - job:
     name: kayobe-seed-images-rocky9
     parent: kayobe-seed-images-base
     nodeset: kayobe-rocky9
+    vars:
+      overcloud_container_image_regex: "^base"
+      seed_container_image_regex: "^base"
 
+# This job builds default overcloud images but skips the build of bifrost seed
+# images which are currently timing out on Ubuntu Jammy.
 - job:
     name: kayobe-seed-images-ubuntu-jammy
     parent: kayobe-seed-images-base
     nodeset: kayobe-ubuntu-jammy
+    vars:
+      seed_container_image_regex: "^base"
 
 - job:
     name: kayobe-overcloud-host-configure-base