=================== Layered Build Guide =================== This section describes the steps for layered build and building an ISO image from a StarlingX R4.0 release onwards. For more detailed explanation of layered build, please refer :ref:`Layered-Build`. .. contents:: :local: :depth: 1 ------------------- :ref:`Requirements` ------------------- Layered build has the same requirements as StarlingX R3.0 build. Click the heading above for details. ------------------------------------ :ref:`Development-environment-setup` ------------------------------------ Layered build uses the same development environment setup as StarlingX R3.0 build. Click the heading above for details. -------------------- Build compiler layer -------------------- ***************************** Prepare the base Docker image ***************************** Follow :ref:`Prepare_the_base_Docker_image` for creating the base StarlingX image with changes only in localrc file as below. Since, we are building compiler layer, create and modify only localrc file. Make sure to set the project and layer as compiler and user name as desired. For example: :: # tbuilder localrc MYUNAME= PROJECT=compiler HOST_PREFIX=$HOME/starlingx/workspace HOST_MIRROR_DIR=$HOME/starlingx/mirror LAYER=compiler .. _Build-the-CentOS-mirror-repository-compiler: ********************************** Build the CentOS mirror repository ********************************** The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run building Docker container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Follow the :ref:`building docker container ` for more details. At the end of this procedure, you will be inside the Starlingx container. Quick command summary is as below: :: cd $HOME/tools/ bash tb.sh env bash tb.sh run bash tb.sh exec ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download source code repositories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Inside the building Docker container, start the internal environment: :: eval $(ssh-agent) ssh-add #. Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for compiler layer. :: cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m compiler.xml #. Synchronize code for compiler layer using command below: :: repo sync -j`nproc` ~~~~~~~~~~~~~~~~~ Download packages ~~~~~~~~~~~~~~~~~ #. Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository: :: cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh #. Monitor the download of packages until it is complete. When the download is complete successfully with no missing packages, the following message appears: :: step #5: done successfully sudo rm -rf /tmp/stx_mirror_4d2URF IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz Success *************** Verify packages *************** #. Verify no missing or failed packages exist: :: cat logs/*_missing_*.log cat logs/*_failmoved_*.log #. In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually or re-run the download command given in previous step. Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. ****************** Packages structure ****************** The following is a general overview of the packages structure resulting from downloading the packages for compiler layer: :: /localdisk/designer///stx-tools/centos-mirror-tools/output . └── stx └── CentOS ├── Binary │   ├── noarch │   └── x86_64 ├── downloads │   └── puppet ├── layer_build_info ├── layer_image_inc ├── layer_pkg_lists ├── layer_wheels_inc └── Source ******************************* Copy CentOS mirror repository ******************************* Exit from the building Docker container. Run the following commands: #. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* workspace directory: :: cd $HOME/starlingx/mirror/CentOS/ #. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* workspace directory. Make sure to insert the : :: cp -r $HOME/starlingx/workspace/localdisk/designer//compiler/stx-tools/centos-mirror-tools/output/stx . *************************** Create a tarball repository *************************** #. Enter the StarlingX container using below command: :: cd $HOME/tools/ ./tb.sh exec #. Copy downloaded CentOS tarballs into StarlingX repo :: ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ Alternatively, you can run the "populate_downloads.sh" script to copy the tarballs instead of using a symlink: :: bash populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ ************** Build packages ************** #. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during :ref:`Build-the-CentOS-mirror-repository-compiler`, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages. #. Update the symbolic links: :: bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ #. Build the packages: :: build-pkgs build-pkgs --installer ------------------ Build distro layer ------------------ .. _Build-the-CentOS-mirror-repository-distro: ********************************** Build the CentOS mirror repository ********************************** The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository. If you were building compiler layer, exit from container and enter the tools directory using below commands: :: cd $HOME/tools/ Since, we are building distro layer, modify localrc file. Make sure to set the project and layer as distro. For example: :: # tbuilder localrc MYUNAME= PROJECT=distro HOST_PREFIX=$HOME/starlingx/workspace HOST_MIRROR_DIR=$HOME/starlingx/mirror LAYER=distro ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run building Docker container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Follow the :ref:`building docker container ` for more details. At the end of this procedure, you will be inside the Starlingx container. Quick command summary is as below: :: cd $HOME/tools/ bash tb.sh env bash tb.sh run #Need not give, If the container was created earlier. bash tb.sh exec ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download source code repositories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Inside the building Docker container, start the internal environment: :: eval $(ssh-agent) ssh-add #. Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for compiler layer. :: cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m distro.xml #. Synchronize code for compiler layer using command below: :: repo sync -j`nproc` ~~~~~~~~~~~~~~~~~ Download packages ~~~~~~~~~~~~~~~~~ #. Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository: :: cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh #. Monitor the download of packages until it is complete. When the download is complete, the following message appears: :: step #5: done successfully sudo rm -rf /tmp/stx_mirror_UIQ675 IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz Success *************** Verify packages *************** #. Verify no missing or failed packages exist: :: cat logs/*_missing_*.log cat logs/*_failmoved_*.log #. In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually. Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. ****************** Packages structure ****************** The following is a general overview of the packages structure resulting from downloading the packages for distro layer: :: /localdisk/designer//distro/stx-tools/centos-mirror-tools/output . └── stx └── CentOS ├── Binary │   ├── EFI │   │   └── BOOT │   │   └── fonts │   ├── images │   │   └── pxeboot │   ├── isolinux │   ├── LiveOS │   ├── noarch │   └── x86_64 ├── downloads │   └── puppet │   └── packstack │   └── puppet │   └── modules ├── layer_build_info ├── layer_image_inc ├── layer_pkg_lists ├── layer_repos │   └── compiler │   └── std │   ├── repodata │   └── repodata.upstream ├── layer_wheels_inc └── Source ******************************* Copy CentOS mirror repository ******************************* Exit from the building Docker container. Run the following commands: #. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* workspace directory: :: cd $HOME/starlingx/mirror/CentOS/ #. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* workspace directory. Make sure to insert the : :: cp -r $HOME/starlingx/workspace/localdisk/designer//distro/stx-tools/centos-mirror-tools/output/stx . *************************** Create a tarball repository *************************** #. Enter the StarlingX container using below command: :: cd $HOME/tools/ ./tb.sh exec #. Copy downloaded CentOS tarballs into StarlingX repo :: ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ Alternatively, you can run the "populate_downloads.sh" script to copy the tarballs instead of using a symlink: :: populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ ************** Build packages ************** #. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during :ref:`Build-the-CentOS-mirror-repository-distro`, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages. #. Update the symbolic links: :: bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ This step creates the repo directory, following is the output on the console for a successful repo directory creation for the stx: :: Copying comps.xml file. Createing yum repodata. Directory walk started Directory walk done - 51 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 1450 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Copying mock.cfg.proto file. Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/efiboot.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/boot.msg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/grub.conf Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/initrd.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.bin Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/memtest Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/splash.png Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vmlinuz Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/LiveOS/squashfs.img Creating folder EFI Creating folder EFI/BOOT Creating folder EFI/BOOT/fonts Creating folder images Creating folder images/pxeboot Creating folder isolinux Creating folder LiveOS Done creating repo directory #. Build the packages: :: build-pkgs build-pkgs --installer ----------------- Build flock layer ----------------- .. _Build-the-CentOS-mirror-repository-flock: ********************************** Build the CentOS mirror repository ********************************** The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository. If you were building distro layer, exit from container and enter the tools directory using below commands: :: cd $HOME/tools/ Since, we are building flock layer, modify localrc file. Make sure to set the project and layer as flock. For example: :: # tbuilder localrc MYUNAME= PROJECT=flock HOST_PREFIX=$HOME/starlingx/workspace HOST_MIRROR_DIR=$HOME/starlingx/mirror LAYER=flock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run building Docker container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Follow the :ref:`building docker container ` for more details. At the end of this procedure, you will be inside the Starlingx container. Quick command summary is as below: :: cd $HOME/tools/ bash tb.sh env bash tb.sh run #Need not give, If the container was created earlier. bash tb.sh exec ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download source code repositories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Inside the building Docker container, start the internal environment: :: eval $(ssh-agent) ssh-add #. Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for flock layer. :: cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m flock.xml #. Synchronize code for compiler layer using command below: :: repo sync -j`nproc` ~~~~~~~~~~~~~~~~~ Download packages ~~~~~~~~~~~~~~~~~ #. Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository: :: cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh #. Monitor the download of packages until it is complete. When the download is complete, the following message appears: :: step #5: done successfully sudo rm -rf /tmp/stx_mirror_ievdiA IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz *************** Verify packages *************** When the download is not successful, the following message appears: :: IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz Warning: Not all download steps succeeded. You are likely missing files. You can verify and list missing or failed packages using below command: :: cat logs/*_missing_*.log cat logs/*_failmoved_*.log In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually. Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. ****************** Packages structure ****************** The following is a general overview of the packages structure resulting from downloading the packages for flock layer: :: /localdisk/designer//flock/stx-tools/centos-mirror-tools/output └── stx └── CentOS ├── Binary │   ├── EFI │   │   └── BOOT │   │   └── fonts │   ├── images │   │   └── pxeboot │   ├── isolinux │   ├── LiveOS │   ├── noarch │   └── x86_64 ├── downloads │   └── puppet ├── layer_build_info ├── layer_image_inc ├── layer_pkg_lists ├── layer_repos │   ├── compiler │   │   └── std │   │   ├── repodata │   │   └── repodata.upstream │   └── distro │   ├── installer │   │   ├── repodata │   │   └── repodata.upstream │   ├── rt │   │   ├── repodata │   │   └── repodata.upstream │   └── std │   ├── repodata │   └── repodata.upstream ├── layer_wheels_inc └── Source ******************************* Copy CentOS mirror repository ******************************* Exit from the building Docker container. Run the following commands: #. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* workspace directory: :: cd $HOME/starlingx/mirror/CentOS/ #. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* workspace directory: :: cp -r $HOME/starlingx/workspace/localdisk/designer//flock/stx-tools/centos-mirror-tools/output/stx . *************************** Create a tarball repository *************************** #. Enter the StarlingX container using below command: :: cd $HOME/tools/ ./tb.sh exec #. Copy downloaded CentOS tarballs into StarlingX repo :: ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ Alternatively, you can run the "populate_downloads.sh" script to copy the tarballs instead of using a symlink: :: populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ #. Exit from the container. On the host machine, create mirror binaries: :: mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img ************** Build packages ************** #. Enter the StarlingX container using below command: :: cd $HOME/tools/ ./tb.sh exec #. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during :ref:`Build-the-CentOS-mirror-repository-flock`, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages. #. Update the symbolic links: :: bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ Following is the output: :: Copying comps.xml file. Createing yum repodata. Directory walk started Directory walk done - 1 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 1892 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 40 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Copying mock.cfg.proto file. Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/efiboot.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/boot.msg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/grub.conf Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/initrd.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.bin Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/memtest Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/splash.png Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vmlinuz Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/LiveOS/squashfs.img Creating folder EFI Creating folder EFI/BOOT Creating folder EFI/BOOT/fonts Creating folder images Creating folder images/pxeboot Creating folder isolinux Creating folder LiveOS Done creating repo directory #. Build the packages: :: build-pkgs ------------------- Build StarlingX ISO ------------------- Build the image: :: build-iso ---------------------- :ref:`Build-installer` ---------------------- Layered build has the same procedure for build installer as StarlingX R3.0 build except for the changes in path of files as below. Click the heading above for details. #. The steps covered by the script **update-pxe-network-installer** is detailed in $MY_REPO/stx/stx-metal/installer/initrd/README. This script creates three files on /localdisk/loadbuild/stx/flock/pxe-network-installer/output. #. The path for **build_srpm.data** is $MY_REPO/stx/metal/installer/pxe-network-installer/centos/.