.. _Layered-Build: ======================= Layered Build Reference ======================= What is build layering? ----------------------- Build layering is a feature to accelerate development within StarlingX. A StarlingX build takes hours and produces hundreds of packages. However 90+% of all work within StarlingX occurs within a much smaller set of top level packages. The remaining packages are largely small patches upon an underlying distribution like CentOS, and change only infrequently. If they change infrequently, why make everyone compile them? So we have partitioned the build into layers: 1. compiler = Low level build tools. Compilers, scripting languages, packaging tools. 2. distro = A modified CentOS plus other third party packages, e.g. ceph, openstack ... 3. flock = Packages unique to StarlingX. This is where we expect most folk to work. We have tried to keep the changes to the old build system minimal, particularly for the most common use case of a developer at the flock layer. Things get slightly more complicated when work on a lower layer is required. Do I have to use a layered build? --------------------------------- A qualified no. If you continue to use the default manifest, and don't specify a layer via command line option or environment variable, you will continue to build all packages across all layers in a single step. This will be a tempting option for folk doing work on lower layers, and still needing to build an ISO for test. There is a limitation to this approach. If you are making lst file changes, you might get away with placing your new required packages in the wrong layer's lst file. Building all layers in one pass won't catch this. So when your update includes lst file changes, you should verify with a layered build or builds. What has changed? ----------------- **1) Download the software for a layer via manifest** The manifest repo now has three new manifests in addition to 'default.xml'. They are 'compiler.xml', 'distro.xml' and 'flock.xml'. There is also a 'common.xml', a place for content that must always be included no matter what layer you are working on. the 'default.xml' is retained as a way to download all StarlingX software. Downloading the flock layer for the master branch looks like this... :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml **2) Environment variables** Two new environment variables are available, 'LAYER' and 'STX_CONFIG_DIR'. I strongly encourage you to set the 'LAYER' environment variable. Otherwise you'll need to pass a layer argument into most commands that do any sort of downloading or building. The second environment variable, ``'STX_CONFIG_DIR'``, can safely be left blank for most use cases. The default is to use ``stx-tools/centos-mirror-tools/config``, which is what you want. When might you want to define ``STX_CONFIG_DIR``? Possibly when working on a major change in the lower layer, such as the cutover to a newer OS version. Or perhaps when redefining the set of layers, or the boundary between layers. In these cases it might be very painful to repo sync due to conflicts. It might be desireable to copy ``stx-tools/centos-mirror-tools/config`` outside of git for a time, and make your changes there. More on the config directory below. e.g. :: export LAYER=flock For containerized build, the norm, the environment variables are passed in via your localrc. Add the LAYER value there.:: cat stc-tools/localrc PROJECT=myproject-flock HOST_PREFIX="" LAYER=flock **3) Controlling the download of dependencies. aka yum repos and 'lst' files** The lst files that used to govern the download of rpms and tarballs have been moved and split. The old location was ``stx-tools/centos-mirror-tools/`` with files like: :: rpms_centos.lst rpms_centos3rdparties.lst rpms_3rdparties.lst tarball-dl.lst other_downloads.lst The new location depends on the file type to be downloaded. **a) src.rpm** The lst files are both relocated and renamed with the os name as a prefix, and 'srpm' rather than 'rpm'. e.g. :: rpms_centos.lst -> centos_srpms_centos.lst rpms_centos3rdparties.lst -> centos_srpms_centos3rdparties.lst rpms_3rdparties.lst -> centos_srpms_3rdparties.lst These files are placed in the root of the git where they are referenced. e.g. Within the 'integ' git, we recompile 'libvirt-python' based on a fedora-core src.rpm. Since fedora is not CentOS, nor is it a 3\ :sup:`rd` party package explicitly intended for CentOS, we'll place it in centos_srpms_3rdparties.lst found in the 'integ' subdirectory. :: cat cgcs-root/stx/integ/centos_srpms_3rdparties.lst libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm ... **b) tarballs** The lst file is both relocated and renamed with the os name as a prefix. e.g. tarball-dl.lst -> centos_tarball-dl.lst These files are placed in the root of the git where they are referenced. e.g. Within the 'integ' git, we compile 'blkin' from a tarball. :: cat cgcs-root/stx/integ/centos_tarball-dl.lst blkin-f24ceec055ea236a093988237a9821d145f5f7c8.tar.gz#blkin#https://api.github.com/repos/ceph/blkin/tarball/f24ceec055ea236a093988237a9821d145f5f7c8#https## ... **c) rpm** The lst files for binary rpms remain in the stx-tools git, but are divided based on layer, and are relocated under an os and layer specific directory. The path will be: :: stx-tools/centos-mirror-tools/config// e.g. for the flock layer :: ls stx-tools/centos-mirror-tools/config/centos/flock/*lst other_downloads.lst rpms_3rdparties.lst rpms_centos3rdparties.lst rpms_centos.lst There is one special virtual layer called 'mock' where rpms required to construct a mock build environment are placed. These rpms are automatically included for all layers. Add an rpm to a layer package list if: - It is required to build the layer. - It is required to build the iso Do not add the rpm to a layer package list if: - It is built by a lower layer - It is already listed in the virtual 'mock' layer. When adding a package to a layer package list, check if any other layer is including the same package. It's ok for two layers to require the same package, but they should require the same version of that package. **d) rpm from a lower layer of the STX layered build** These are automatically downloaded for you, based on the configuration found in ``'stx-tools/centos-mirror-tools/config///required_layer_pkgs.cfg'``. The default config is to pull content from the most recent official build. You shouldn't have to touch this file. **e) yum repositories** You should continue to use ``'stx-tools/centos-mirror/yum.repos.d'`` as the place to define new yums repos for downloading non-StarlingX rpms. You may notice that there are also yum directories found at ``'stx-tools/centos-mirror-tools/config///yum.repos.d'``. These are only intended to refer to StarlingX official build rpms. You probably should NOT be touching these unless you are creating a new branch, layer or os. Controlling the package content of the ISO ------------------------------------------ Only the flock layer is capable of building an ISO. ISO image content used to be defined exclusively by the files: :: cgcs-root/build-tools/build_iso/image.inc cgcs-root/build-tools/build_iso/minimal_rpm_list.txt These files continue to be used, but should not include packages that we build. Packages that we build, and supply a top level command or service. should be listed in the ``'_iso_image.inc'`` file. The file is located at the root of the git where the package is found. e.g. qemu-kvm-ev is compiled from the 'integ' git repo, so it is found in ... :: cat cgcs-root/stx/integ/centos_iso_image.inc ... # qemu-kvm-ev qemu-kvm-ev qemu-img-ev qemu-kvm-tools-ev ... Only packages supplying top level commands and services need be listed. Dependencies do NOT need to be listed. They will be resolved automatically. The image inc files of lower layer are automatically pulled in and made available to the flock layer when build-iso is run. This is governed by the ``'stx-tools/centos-mirror-tools/config///required_layer_iso_inc.cfg'`` config file. You shouldn't have to touch this file. How do I use build layering? ---------------------------- Lets address this one scenario at a time. **A flock layer developer ... a simple change ... no packaging changes.** Very little has changed. The populate_download step might be a bit slower as you'll be picking up rpms from lower layer builds, but this is mostly a pain to be suffered on the first build attempt. Once locally cached, subsequent downloads should be fast. The build-pkgs step should be much faster. :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml repo sync ... export LAYER=flock ... echo “LAYER=$LAYER” >> stx-tools/localrc ... cd /stx-tools/centos-mirror-tools download_mirror.sh -c ./yum.conf.sample -n -g ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... build-pkgs build-iso **A distro layer developer ... a simple change ... no packaging changes.** Assuming you can test your changes by patching in new rpms (no ISO build required), then it's just ... :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml repo sync ... export LAYER=distro ... echo “LAYER=$LAYER” >> stx-tools/localrc ... download_mirror.sh ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... build-pkgs build-pkgs --installer # build-iso can't be run from this layer **A compiler layer developer ... a simple change ... no packaging changes.** Assuming you can test your changes by patching in new rpms (no ISO build required), then it's just ... :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml repo sync ... export LAYER=compiler ... echo “LAYER=$LAYER” >> stx-tools/localrc ... download_mirror.sh ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... build-pkgs build-pkgs --installer # build-iso can't be run from this layer **Cross layer development.** e.g. A kernel developer adding a new or updated driver required at install time. This is a cross layer build exercise. The kernel and it's drivers are a distro layer component, but the installer and ISO are built from the flock layer. Set up an independent build environment for each layer. 1) distro environment :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml repo sync ... export LAYER=distro ... echo “LAYER=$LAYER” >> stx-tools/localrc ... download_mirror.sh ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... build-pkgs build-pkgs --installer 2) flock environment :: repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml repo sync ... export LAYER=flock ... echo “LAYER=$LAYER” >> stx-tools/localrc ... At this stage you must point the flock layer to pick up your custom distro layer content. The location of lower layer content is encoded in config files found under ``stx-tools/centos-mirror-tools/config//`` in files ``required_layer_pkgs.cfg`` and ``required_layer_iso_inc.cfg``. Both files use a comma seperated three field lines... ``,,`` e.g. :: cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/rpm.lst distro,rt,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/rpm.lst distro,installer,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/rpm.lst cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc compiler,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image.inc distro,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image-dev.inc To use your lower layer build, you must edit the config in the upper layer build. You must replace the url field for the relevant lines to point to your own build using the ``fill:///`` syntax. e.g. To use a 'distro' build compiled under ``PROJECT=-distro`` :: distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc How to make the changes ... Option a) Edit the config files in place. Do not submit this change!!! Using option 'b' (see below) would be safer. :: vi stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg \\ stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg download_mirror.sh ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ Option b) Use an alternative config directory. Copy the default config to an alternative directory outside of git, but still visible to the build. In the copyied config, edit the config files, replacing the existing 'distro' url's with ``file:///`` urls. Finally instruct the build to use the alternate config. I'll use the environment variable method in the example below. It can also be done with command line arguments. :: cp -r stx-tools/centos-mirror-tools/config config.tmp export STX_CONFIG_DIR=$PWD/config.tmp ... echo “STX_CONFIG_DIR=$STX_CONFIG_DIR” >> stx-tools/localrc ... vi config.tmp/centos/flock/required_layer_pkgs.cfg \\ config.tmp/centos/flock/required_layer_iso_inc.cfg download_mirror.sh ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/ Option c) supply command line arguments to ``populate_downloads.sh`` and ``generate-cgcs-centos-repo.sh`` overriding the urls directly :: download_mirror.sh \\ -L distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ -L distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ -L distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ -I distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ -I distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc ... ln -s /import/mirrors/CentOS/stx-r1/CentOS/downloads/ $MY_REPO/stx/ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/ ... generate-cgcs-centos-repo.sh \\ --layer-pkg-url=distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ --layer-pkg-url=distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ --layer-pkg-url=distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ --layer-inc-url=distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ --layer-inc-url=distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc \\ /import/mirrors/CentOS/stx-r1/CentOS/ Now resume building, but this time we'll roll our own installer :: build-pkgs update-pxe-network-installer This script creates three files on ``/localdisk/loadbuild/-flock/pxe-network-installer/output``. :: new-initrd.img new-squashfs.img new-vmlinuz Rename the files as follows: :: initrd.img squashfs.img vmlinuz Finally ... :: build-pkgs --clean pxe-network-installer build-pkgs pxe-network-installer build-iso Making packaging changes ------------------------ **In what layer should I place my new compiled package ?** If the package is original content, writen for the StarlingX project, it belongs in the 'flock' layer. Yes, envision a flock of starling, might be corny but that is what we named it. All other content is considered third party and goes in either the 'distro' or 'compiler' layer. If it's a core component of a programming or packaging language, a build or packaging tool. It belongs in the the compiler layer. We expect this layer to change only very rarely. All other third party content goes in the 'distro' layer. In it you will find everything from patches CentOS packages, the kernel and drivers, ceph, openstack components and much more. **Location of new repo manifest entries?** If a new git repo is required, add it to BOTH the default and layer specific manifests. **Location of yum repo changes ?** Hopefully we aren't often adding new yum repos. If required, add it to ``'stx-tools/centos-mirror/yum.repos.d'`` and NOT to ``'stx-tools/centos-mirror-tools/config///yum.repos.d'``. **Location of 'lst' file changes ?** If the package to be added is derived from a third party tarball or src.rpm, add it to the lst files at the root of the git where the compile directives (spec file, build_srpm.data etc) will live. You'll be adding to one of .... :: centos_srpms_3rdparties.lst centos_srpms_centos3rdparties.lst centos_srpms_centos.lst centos_tarball-dl.lst ...as appropriate. The package's 'BuildRequires' , as well as the transitive Requires of those BuildRequires, should be added to a lst file under ``stx-tools/centos-mirror-tools/config//``. e.g. one of... :: rpms_3rdparties.lst rpms_centos3rdparties.lst rpms_centos.lst ...as appropriate. If the package will be installed to iso, the package's 'Requires' as well as the transitive Requires of those Requires, should be added to a lst file under ``stx-tools/centos-mirror-tools/config//flock``. Yes I said 'flock, and not , because the ISO is build from the flock layer. Figuring out the transitive list of a package can be a challenge. For centos packages, my suggestion is to fire up a separate centos container of the correct vintage, and try running ... :: repoquery --requires --resolve --recursive \\ --qf='%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}.rpm' ... otherwise you may have to resort to a several build iterations, and fixing what breaks each time. The set of rpms suggested by the above method will likely list packages already in your lst file. If the same version, no action required. If a lower version, then you probably want to update the version to the newer package. Check if the package is listed for a lower layer lst file, and update it there as well. If the rpm suggested by the above method does not exist, check if we are building it within StarlingX. If so, don't list it in a lst file. If not listed, and we don't build it, then add it. **Including a package in the iso?** Add you new compile package to the ``_iso_image.inc`` file at the root of the git where the compile directives for your new package live. Packages obtained from the host os (e.g. CentOS) have traditionally gone in ``'cgcs-root/build-tools/build_iso/image.inc'``, and we can continue with this practice for now, however it will likely become a barrier to an iso build from the distro layer alone. Considering this, I probably wouldn't object to a host os service or binary being listed in a git's ``_iso_image.inc``. A better place for it might be ``stx-tools/centos-mirror-tools/config///image.inc``, but this isn't yet supported.