diff --git a/doc/source/developer_resources/Layered_Build.rst b/doc/source/developer_resources/Layered_Build.rst deleted file mode 100644 index d84c98f92..000000000 --- a/doc/source/developer_resources/Layered_Build.rst +++ /dev/null @@ -1,569 +0,0 @@ -.. _Layered-Build: - -======================= -Layered Build Reference -======================= - -What is build layering? ------------------------ - -Build layering is a feature to accelerate development within StarlingX. - -A StarlingX build takes hours and produces hundreds of packages. However -90+% of all work within StarlingX occurs within a much smaller set of -top level packages. The remaining packages are largely small patches -upon an underlying distribution like CentOS, and change only -infrequently. If they change infrequently, why make everyone compile -them? - -So we have partitioned the build into layers: - -1. compiler = Low level build tools. Compilers, scripting languages, packaging tools. -2. distro = A modified CentOS plus other third party packages, e.g. ceph, openstack ... -3. flock = Packages unique to StarlingX. This is where we expect most folk to work. - -We have tried to keep the changes to the old build system minimal, -particularly for the most common use case of a developer at the flock -layer. Things get slightly more complicated when work on a lower layer -is required. - -Do I have to use a layered build? ---------------------------------- - -A qualified no. If you continue to use the default manifest, and don't -specify a layer via command line option or environment variable, you -will continue to build all packages across all layers in a single step. -This will be a tempting option for folk doing work on lower layers, and -still needing to build an ISO for test. - -There is a limitation to this approach. If you are making lst file -changes, you might get away with placing your new required packages in -the wrong layer's lst file. Building all layers in one pass won't catch -this. So when your update includes lst file changes, you should verify -with a layered build or builds. - -What has changed? ------------------ - -**1) Download the software for a layer via manifest** - -The manifest repo now has three new manifests in addition to -'default.xml'. They are 'compiler.xml', 'distro.xml' and 'flock.xml'. -There is also a 'common.xml', a place for content that must always be -included no matter what layer you are working on. the 'default.xml' is -retained as a way to download all StarlingX software. - -Downloading the flock layer for the master branch looks like this... :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - -**2) Environment variables** - -Two new environment variables are available, 'LAYER' and -'STX_CONFIG_DIR'. - -I strongly encourage you to set the 'LAYER' environment variable. -Otherwise you'll need to pass a layer argument into most commands that -do any sort of downloading or building. - -The second environment variable, ``'STX_CONFIG_DIR'``, can safely be left -blank for most use cases. The default is to use -``stx-tools/centos-mirror-tools/config``, which is what you want. When might -you want to define ``STX_CONFIG_DIR``? Possibly when working on a major -change in the lower layer, such as the cutover to a newer OS version. Or -perhaps when redefining the set of layers, or the boundary between layers. In -these cases it might be very painful to repo sync due to conflicts. It -might be desirable to copy ``stx-tools/centos-mirror-tools/config`` outside -of git for a time, and make your changes there. More on the config -directory below. - -e.g. :: - - export LAYER=flock - -For containerized build, the norm, the environment variables are passed -in via your localrc. Add the LAYER value there.:: - - cat stc-tools/localrc - PROJECT=myproject-flock - HOST_PREFIX="" - LAYER=flock - -**3) Controlling the download of dependencies. aka yum repos and 'lst' files** - -The lst files that used to govern the download of rpms and tarballs have -been moved and split. - -The old location was ``stx-tools/centos-mirror-tools/`` with files like: :: - - rpms_centos.lst - rpms_centos3rdparties.lst - rpms_3rdparties.lst - tarball-dl.lst - other_downloads.lst - -The new location depends on the file type to be downloaded. - -**a) src.rpm** - -The lst files are both relocated and renamed with the os name as a -prefix, and 'srpm' rather than 'rpm'. - -e.g. :: - - rpms_centos.lst -> centos_srpms_centos.lst - rpms_centos3rdparties.lst -> centos_srpms_centos3rdparties.lst - rpms_3rdparties.lst -> centos_srpms_3rdparties.lst - -These files are placed in the root of the git where they are referenced. - -e.g. Within the 'integ' git, we recompile 'libvirt-python' based on a -fedora-core src.rpm. Since fedora is not CentOS, nor is it a -3\ :sup:`rd` party package explicitly intended for CentOS, we'll place -it in centos_srpms_3rdparties.lst found in the 'integ' subdirectory. :: - - cat cgcs-root/stx/integ/centos_srpms_3rdparties.lst - libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm - ... - -**b) tarballs** - -The lst file is both relocated and renamed with the os name as a prefix. - -e.g. tarball-dl.lst -> centos_tarball-dl.lst - -These files are placed in the root of the git where they are referenced. - -e.g. Within the 'integ' git, we compile 'blkin' from a tarball. :: - - cat cgcs-root/stx/integ/centos_tarball-dl.lst - blkin-f24ceec055ea236a093988237a9821d145f5f7c8.tar.gz#blkin#https://api.github.com/repos/ceph/blkin/tarball/f24ceec055ea236a093988237a9821d145f5f7c8#https## - ... - -**c) rpm** - -The lst files for binary rpms remain in the stx-tools git, but are -divided based on layer, and are relocated under an os and layer specific -directory. The path will be: :: - - stx-tools/centos-mirror-tools/config// - -e.g. for the flock layer :: - - ls stx-tools/centos-mirror-tools/config/centos/flock/*lst - other_downloads.lst - rpms_3rdparties.lst - rpms_centos3rdparties.lst - rpms_centos.lst - -There is one special virtual layer called 'mock' where rpms required to -construct a mock build environment are placed. These rpms are -automatically included for all layers. - -Add an rpm to a layer package list if: - -- It is required to build the layer. - -- It is required to build the iso - -Do not add the rpm to a layer package list if: - -- It is built by a lower layer - -- It is already listed in the virtual 'mock' layer. - -When adding a package to a layer package list, check if any other layer -is including the same package. It's ok for two layers to require the -same package, but they should require the same version of that package. - -**d) rpm from a lower layer of the STX layered build** - -These are automatically downloaded for you, based on the configuration -found in -``'stx-tools/centos-mirror-tools/config///required_layer_pkgs.cfg'``. -The default config is to pull content from the most recent official -build. You shouldn't have to touch this file. - -**e) yum repositories** - -You should continue to use ``'stx-tools/centos-mirror/yum.repos.d'`` as the -place to define new yums repos for downloading non-StarlingX rpms. - -You may notice that there are also yum directories found at -``'stx-tools/centos-mirror-tools/config///yum.repos.d'``. These -are only intended to refer to StarlingX official build rpms. You -probably should NOT be touching these unless you are creating a new -branch, layer or os. - -Controlling the package content of the ISO ------------------------------------------- - -Only the flock layer is capable of building an ISO. - -ISO image content used to be defined exclusively by the files: :: - - cgcs-root/build-tools/build_iso/image.inc - cgcs-root/build-tools/build_iso/minimal_rpm_list.txt - -These files continue to be used, but should not include packages that we -build. - -Packages that we build, and supply a top level command or service. -should be listed in the ``'_iso_image.inc'`` file. The file is located -at the root of the git where the package is found. - -e.g. qemu-kvm-ev is compiled from the 'integ' git repo, so it is found -in ... :: - - cat cgcs-root/stx/integ/centos_iso_image.inc - - ... - # qemu-kvm-ev - qemu-kvm-ev - qemu-img-ev - qemu-kvm-tools-ev - ... - -Only packages supplying top level commands and services need be listed. -Dependencies do NOT need to be listed. They will be resolved -automatically. - -The image inc files of lower layer are automatically pulled in and made -available to the flock layer when build-iso is run. This is governed by -the -``'stx-tools/centos-mirror-tools/config///required_layer_iso_inc.cfg'`` -config file. You shouldn't have to touch this file. - -How do I use build layering? ----------------------------- - -Lets address this one scenario at a time. - -**A flock layer developer ... a simple change ... no packaging -changes.** - -Very little has changed. The populate_download step might be a bit -slower as you'll be picking up rpms from lower layer builds, but this is -mostly a pain to be suffered on the first build attempt. Once locally -cached, subsequent downloads should be fast. The build-pkgs step should -be much faster. :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - repo sync - ... - export LAYER=flock - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - cd /stx-tools/centos-mirror-tools - download_mirror.sh -c ./yum.conf.sample -n -g - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-iso - -**A distro layer developer ... a simple change ... no packaging -changes.** - -Assuming you can test your changes by patching in new rpms (no ISO build -required), then it's just ... :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml - repo sync - ... - export LAYER=distro - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - # build-iso can't be run from this layer - -**A compiler layer developer ... a simple change ... no packaging -changes.** - -Assuming you can test your changes by patching in new rpms (no ISO build -required), then it's just ... :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml - repo sync - ... - export LAYER=compiler - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - # build-iso can't be run from this layer - -**Cross layer development.** - -e.g. A kernel developer adding a new or updated driver required at -install time. This is a cross layer build exercise. The kernel and it's -drivers are a distro layer component, but the installer and ISO are -built from the flock layer. - -Set up an independent build environment for each layer. - -1) distro environment -:: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml - repo sync - ... - export LAYER=distro - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - -2) flock environment -:: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - repo sync - ... - export LAYER=flock - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - -At this stage you must point the flock layer to pick up your custom -distro layer content. The location of lower layer content is encoded -in config files found under ``stx-tools/centos-mirror-tools/config//`` -in files ``required_layer_pkgs.cfg`` and ``required_layer_iso_inc.cfg``. -Both files use a comma seperated three field lines... ``,,`` -e.g. :: - - cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg - compiler,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst - distro,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/rpm.lst - distro,rt,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/rpm.lst - distro,installer,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/rpm.lst - - cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg - compiler,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc - compiler,dev,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc - distro,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/image.inc - distro,dev,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/image-dev.inc - -To use your lower layer build, you must edit the config in the upper layer build. -You must replace the url field for the relevant lines to point to your own build using the ``fill:///`` syntax. - -e.g. To use a 'distro' build compiled under ``PROJECT=-distro`` -:: - - distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - - distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc - - - -How to make the changes ... - -Option a) Edit the config files in place. Do not submit this change!!! - -Using option 'b' (see below) would be safer. -:: - - vi stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg \\ - stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - -Option b) Use an alternative config directory. - -Copy the default config to an alternative directory outside of git, -but still visible to the build. In the copied config, edit the config files, -replacing the existing 'distro' url's with ``file:///`` urls. Finally -instruct the build to use the alternate config. I'll use the -environment variable method in the example below. It can also be done -with command line arguments. -:: - - cp -r stx-tools/centos-mirror-tools/config config.tmp - export STX_CONFIG_DIR=$PWD/config.tmp - ... - echo “STX_CONFIG_DIR=$STX_CONFIG_DIR” >> stx-tools/localrc - ... - vi config.tmp/centos/flock/required_layer_pkgs.cfg \\ - config.tmp/centos/flock/required_layer_iso_inc.cfg - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - -Option c) supply command line arguments to ``populate_downloads.sh`` and -``generate-local-repo.sh`` overriding the urls directly :: - - download_mirror.sh \\ - -L distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - -L distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - -L distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - -I distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - -I distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh \\ - --layer-pkg-url=distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - --layer-inc-url=distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - --layer-inc-url=distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc \\ - /import/mirrors/CentOS/stx/CentOS/ - -Now resume building, but this time we'll roll our own installer :: - - build-pkgs - update-pxe-network-installer - -This script creates three files on -``/localdisk/loadbuild/-flock/pxe-network-installer/output``. :: - - new-initrd.img - new-squashfs.img - new-vmlinuz - -Rename the files as follows: :: - - initrd.img - squashfs.img - vmlinuz - -Finally ... :: - - build-pkgs --clean pxe-network-installer - build-pkgs pxe-network-installer - build-iso - - -Making packaging changes ------------------------- - -**In what layer should I place my new compiled package ?** - -If the package is original content, written for the StarlingX project, it -belongs in the 'flock' layer. Yes, envision a flock of starling, might -be corny but that is what we named it. All other content is considered -third party and goes in either the 'distro' or 'compiler' layer. - -If it's a core component of a programming or packaging language, a build -or packaging tool. It belongs in the the compiler layer. We expect this -layer to change only very rarely. - -All other third party content goes in the 'distro' layer. In it you will -find everything from patches CentOS packages, the kernel and drivers, -ceph, openstack components and much more. - -**Location of new repo manifest entries?** - -If a new git repo is required, add it to BOTH the default and layer -specific manifests. - -**Location of yum repo changes ?** - -Hopefully we aren't often adding new yum repos. If required, add it to -``'stx-tools/centos-mirror/yum.repos.d'`` and NOT to -``'stx-tools/centos-mirror-tools/config///yum.repos.d'``. - -**Location of 'lst' file changes ?** - -If the package to be added is derived from a third party tarball or -src.rpm, add it to the lst files at the root of the git where the -compile directives (spec file, build_srpm.data etc) will live. You'll be -adding to one of .... :: - - centos_srpms_3rdparties.lst - centos_srpms_centos3rdparties.lst - centos_srpms_centos.lst - centos_tarball-dl.lst - -...as appropriate. - -The package's 'BuildRequires' , as well as the transitive Requires of -those BuildRequires, should be added to a lst file under -``stx-tools/centos-mirror-tools/config//``. e.g. one of... :: - - rpms_3rdparties.lst - rpms_centos3rdparties.lst - rpms_centos.lst - -...as appropriate. - -If the package will be installed to iso, the package's 'Requires' as -well as the transitive Requires of those Requires, should be added to a -lst file under ``stx-tools/centos-mirror-tools/config//flock``. Yes I -said 'flock, and not , because the ISO is build from the flock -layer. - -Figuring out the transitive list of a package can be a challenge. For -centos packages, my suggestion is to fire up a separate centos container -of the correct vintage, and try running ... :: - - repoquery --requires --resolve --recursive \\ - --qf='%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}.rpm' - -... otherwise you may have to resort to a several build iterations, and -fixing what breaks each time. - -The set of rpms suggested by the above method will likely list packages -already in your lst file. If the same version, no action required. If a -lower version, then you probably want to update the version to the newer -package. Check if the package is listed for a lower layer lst file, and -update it there as well. - -If the rpm suggested by the above method does not exist, check if we are -building it within StarlingX. If so, don't list it in a lst file. - -If not listed, and we don't build it, then add it. - -**Including a package in the iso?** - -Add you new compile package to the ``_iso_image.inc`` file at the root -of the git where the compile directives for your new package live. - -Packages obtained from the host os (e.g. CentOS) have traditionally gone -in ``'cgcs-root/build-tools/build_iso/image.inc'``, and we can continue with -this practice for now, however it will likely become a barrier to an iso -build from the distro layer alone. Considering this, I probably wouldn't -object to a host os service or binary being listed in a git's -``_iso_image.inc``. A better place for it might be -``stx-tools/centos-mirror-tools/config///image.inc``, but this -isn't yet supported. diff --git a/doc/source/developer_resources/Layered_Build_cn.rst b/doc/source/developer_resources/Layered_Build_cn.rst deleted file mode 100644 index 66a7c248c..000000000 --- a/doc/source/developer_resources/Layered_Build_cn.rst +++ /dev/null @@ -1,508 +0,0 @@ -========================== -分层构建 "(Layered Build)" -========================== -.. note:: - 由于StarlingX没有翻译组，因此可能无法积极的维护本指导文档。如果您对翻译指导文档感兴趣，请联系 - `StarlingX文档团队 `_。 - - This guide may not be actively maintained because StarlingX does not have a translation project. - If you are interested in translating guides, please contact the - `StarlingX docs team `_. - -什么是分层构建？ ------------------------ - -分层构建是 StarlingX为提高内部研发效率，而设计的一种构建方法。 - -在此之前，构建一个 StarlingX系统需要耗费几个小时，并生成数百个软件包。Starlingx 90%+ 的工作都在于 -对上层的众多package包的维护和开发，剩下的的package包多是发行版(如CentOS)所必须的，而这些package是很少会变化的。 -如果这些不经常更改，那么为什么每个人都要编译它？ - -所以我们把构建分成了几个层级: - -1. compiler = 低层的构建工具层。如：编译器工具，脚本语言工具，打包工具等。 -2. distro = 一个修改过的 CentOS系统 加上其他第三方软件包，比如 ceph，openstack... -3. flock = StarlingX特有的软件包。这是我们希望大多数人工作的地方。 -4. containers = StarlingX特有的容器化软件包。 - -对于在flock层开发的人员来说，构建过程和旧的编译系统(full build)差别不大。而对于在compiler和distro做开发的人员， -layered build构建过程会稍微复杂一些。 - -我必须使用分层的构建吗？ ---------------------------------- - -不一定。如果使用default manifest，且不使用命令行或者环境变量去指定编译特定的层，那么执行编译命令后， -仍然会编译所有的层次中的各个package。对于在compiler和distro layer开发，或者需要获取完整的ISO镜像用于测试， -可以采用这种方式构建。 - -Full build (使用default manifest) 有一些局限性。如对于lst文件，错误的将该层需要的包名填写到了另外一层的lst文件中。 -Full build是可以正确编译的，无法发现这个依赖问题。所以对lst文件的修改，是需要使用layer build构建和验证的。 - -发生了哪些变化？ ------------------ - -**1) 通过 manifest 下载指定层的软件包** - -配置库除了'default.xml'之外还有四个新的配置项。它们是'compiler.xml'，'distro.xml'，'flock.xml'和'containers.xml'。 -还有一个'common.xml' 共通配置表，无论您在那个层上进行构建，这共通配置表都会被包含在其构建中。目前还是保留了'default.xml' -作为下载所有 StarlingX 软件包的方式。 - -在主分支上进行下载flock layer内容，参照以下步骤... :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - -**2) 环境变量** - -新增加了两个环境变量，'LAYER' 和 'STX_CONFIG_DIR'。 - -强烈建议您配置'LAYER'这个环境变量。否则，您在执行任何类型的下载或构建过程中,许多命令操作会提示您需传递一个layer参数。 - -第二个环境变量``'STX_CONFIG_DIR'``,在大多数情况下是可以不进行配置的。默认是使用``stx-tools/centos-mirror-tools/config`` -这个路径， 这是默认的配置路径信息。什么时候需要定义``STX_CONFIG_DIR``？ 可能在主要处理较低层的更改时，比如切换较新的操作系统版本。 -或者重新定义多工作层的集合，或者各工作层的衔接。在这些情况下，由于内容冲突导致repo sync可能会非常痛苦。可能需要复制 -``stx-tools/centos-mirror-tools/config`` 路径的内容到git管理之外一段时间然后进行修改。关于config目录以下有一些详述。 - -例如： :: - - export LAYER=flock - -在容器的构建环境下，标准的配置以及环境变量都是通过 localrc 传入到构建环境里的。比如添加 LAYER 值 :: - - cat stc-tools/localrc - PROJECT=myproject-flock - HOST_PREFIX="" - LAYER=flock - -**3) 控制依赖文件的下载，即 yum repos 和 lst 文件** - -原有的用于管理rpm和tarballs下载的lst文件已经被移动和拆分成多个。 - -旧的位置是 ``stx-tools/centos-mirror-tools/``，其包含的文件如下: :: - - rpms_centos.lst - rpms_centos3rdparties.lst - rpms_3rdparties.lst - tarball-dl.lst - other_downloads.lst - -新的lst文件位置取决于其要下载的文件类型。 - -**a) src.rpm** - -lst文件被重新定位并重命名。其前缀采用了'操作系统名' + 'srpm'，而不再使用'rpm'。 - -例如： :: - - rpms_centos.lst -> centos_srpms_centos.lst - rpms_centos3rdparties.lst -> centos_srpms_centos3rdparties.lst - rpms_3rdparties.lst -> centos_srpms_3rdparties.lst - -Src rpm相关lst文件放在git仓库根目录下。 - -例如，在'integ' git 中，我们基于 fedora-core src.rpm 重新编译'libvirt-python'。 该libvirt-python不是 -CentOS需要的第三方包，所以我们将它记录在'integ'子目录中的 centos_srpms_3rdparties.lst文件中。 :: - - cat cgcs-root/stx/integ/centos_srpms_3rdparties.lst - libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm - ... - -**b) tarballs** - -lst文件同时被重新定位和重命名，并以操作系统名作为前缀。 - -例如，tarball-dl.lst -> centos_tarball-dl.lst - -Tarball相关lst文件放在git仓库根目录下。 - -例如，在'integ' git 中，我们从tarball中编译'blkin'。 :: - - cat cgcs-root/stx/integ/centos_tarball-dl.lst - blkin-f24ceec055ea236a093988237a9821d145f5f7c8.tar.gz#blkin#https://api.github.com/repos/ceph/blkin/tarball/f24ceec055ea236a093988237a9821d145f5f7c8#https## - ... - -**c) rpm** - -对于记录二进制rpm包的lst文件，将其保留在了 stx-tools git 中，但是基于不同的工作层，将其划分并重新定位到 -/特定的目录中，路径如下: :: - - stx-tools/centos-mirror-tools/config// - -例如flock层 :: - - ls stx-tools/centos-mirror-tools/config/centos/flock/*lst - other_downloads.lst - rpms_3rdparties.lst - rpms_centos3rdparties.lst - rpms_centos.lst - -有一个称为'mock'的特殊虚拟层，其中放置了构建模拟构建环境所需的 rpm。 这些rpm自动包含在所有工作层中。 - -什么情况下需要添加一个 rpm 到一个层的包列表中： - -- 这个rpm被构建的层所需要。 - -- 这个rpm被构建的ISO所需要。 - -什么情况下不要将 rpm 添加到工作层的包列表中： - -- 它是由较低的工作层所构建的。 - -- 它已经在虚拟的'mock'层所列出。 - -在向工作层的包列表中添加包时，检查是否有其他工作层包含了相同的包。在两个层中，含有相同的软件包是可以的， -但是他们需要具有相同的软件包版本。 - -**d) 从STX较低的工作层获取 rpm包** - -基于``'stx-tools/centos-mirror-tools/config///required_layer_pkgs.cfg'`` 中的配置， -其可以为您自动下载这些包文件。 默认的配置是从最新的官方版本中获取包文件，您无需修改此文件。 - -**e) yum repositories** - -可以继续使用``'stx-tools/centos-mirror/yum.repos.d'`` 目录，作为yum源去下载非Starlingx的rpm包。 - -您可能也注意到在``'stx-tools/centos-mirror-tools/config///yum.repos.d'`` 中也可以找到一些 yum 目录。 -这些只是为了引用 StarlingX官方构建的 rpms。大部分情况下不用修改这些文件，除非你正在构建一个新的分支，工作层或操作系统。 - -控制构建ISO的包内容 ------------------------------------------- - -只有flock层能够建立一个ISO。 - -ISO镜像内容完全由文件来定义： :: - - cgcs-root/build-tools/build_iso/image.inc - cgcs-root/build-tools/build_iso/minimal_rpm_list.txt - -构建依赖的包信息不在以上文件中。 - -构建依赖的包应该列在``'_iso_image.inc'`` 这个文件，其位于你所看到的git库的根目录中。 - -例如，qemu-kvm-ev 是由'integ' git repo 编译的，所以它可以在 :: - - cat cgcs-root/stx/integ/centos_iso_image.inc - - ... - # qemu-kvm-ev - qemu-kvm-ev - qemu-img-ev - qemu-kvm-tools-ev - ... - -只需要列出构建依赖的包名，包之间的依赖关系不需要列出，依赖会被自动解决。 - -当 build-iso 运行时，底层的镜像文件会自动下载，并提供给flock工作层。 这是由 -``'stx-tools/centos-mirror-tools/config///required_layer_iso_inc.cfg'`` -这个配置文件所控制的，你不应该修改这个配置文件。 - -如何使用分层构建？ ----------------------------- - -让我们依次解决这些问题。 - -**工作在flock layer... 一个简单的改变... 没有包的变化。** - -当非常小的修改时，因为首次您需要从较低的层次构建中获取rpm，下载步骤可能会稍微慢一点，所以说在第一次尝试构建时是最痛苦的。 -一旦本地有了缓存，后续的下载应该很快。构建 pkgs 步骤应该快得多。 :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - repo sync - ... - export LAYER=flock - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - cd /stx-tools/centos-mirror-tools - download_mirror.sh -c ./yum.conf.sample -n -g - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-iso - -**工作在distro layer... 一个简单的改变... 没有包的改变。** - -假设您可以通过打补丁到新的 rpm (不需要 ISO 构建)来测试您的更改，那么..。 :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml - repo sync - ... - export LAYER=distro - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - # build-iso can't be run from this layer - -**工作在compiler layer... 一个简单的修改... 没有包的修改。** - -假设您可以通过打补丁新的 rpm (不需要 ISO 构建)来测试您的更改，那么..。 :: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml - repo sync - ... - export LAYER=compiler - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - # build-iso can't be run from this layer - -**跨层部署构建。** - -例如：内核开发人员在安装时添加新的或更新驱动程序。这就是一个跨层次的构建练习。内核和它的驱动程序是一个发行版层的组件， -但是安装程序和ISO是从 flock layer构建的。 - -为每个工作层设置一个独立的构建环境。 - -1) Distro layer环境 -:: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml - repo sync - ... - export LAYER=distro - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - ... - build-pkgs - build-pkgs --installer - -2) Flock layer环境 -:: - - repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml - repo sync - ... - export LAYER=flock - ... - echo “LAYER=$LAYER” >> stx-tools/localrc - ... - -在这个阶段，需要为flock layer指定你自定义的distro layer的相关内容。这些内容在配置文件中指定，位于 -``stx-tools/centos-mirror-tools/config//``下的``required_layer_pkgs.cfg``和 -``required_layer_iso_inc.cfg``文件中。在这两个配置文件中列出了所依赖的下层描述信息``<依赖层>,<类型>,<依赖内容的路径>``， -其格式使用逗号分隔为三个字段，参照以下： :: - - cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg - compiler,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst - distro,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/rpm.lst - distro,rt,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/rpm.lst - distro,installer,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/rpm.lst - - cat stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg - compiler,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc - compiler,dev,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc - distro,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/image.inc - distro,dev,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/distro/latest_build/outputs/image-dev.inc - -如果需要用到更底层layer所构建的包，需要在当前layer做好配置。使用语法: ``file://`` 将配置文件中的url替换成更底层layer所对应的信息。 - -例如：需要使用到在'distro layer' 编译生成的包(构建时项目名为:``PROJECT=-distro``) :: - - distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst - distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst - distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst - - distro,std,file:///localdisk/loadbuild/-distro/std/image.inc - distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc - - - -如何修改这些配置信息？ - -选项 a)直接修改原始的配置文件。但，请不要提交你的修改! ! - -'b'方案会更加安全 :: - - vi stx-tools/centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg \\ - stx-tools/centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - -选项 b)使用一个替代的配置文件目录 - -拷贝default的配置文件到git仓库以外，但仍需要保证构建系统可见。修改拷贝出的配置文件，使用 ``file://`` url格式修改url。 :: - - cp -r stx-tools/centos-mirror-tools/config config.tmp - export STX_CONFIG_DIR=$PWD/config.tmp - ... - echo “STX_CONFIG_DIR=$STX_CONFIG_DIR” >> stx-tools/localrc - ... - vi config.tmp/centos/flock/required_layer_pkgs.cfg \\ - config.tmp/centos/flock/required_layer_iso_inc.cfg - download_mirror.sh - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh /import/mirrors/CentOS/stx/CentOS/ - -选项 c)提供命令行参数来赋值给 downloads.sh 和 generate-local-repo.sh 脚本文件，并直接覆盖 url :: - - download_mirror.sh \\ - -L distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - -L distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - -L distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - -I distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - -I distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc - ... - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - ... - generate-local-repo.sh \\ - --layer-pkg-url=distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - --layer-inc-url=distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - --layer-inc-url=distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc \\ - /import/mirrors/CentOS/stx/CentOS/ - -然后继续构建，接下来将导出我们自己的安装程序。 :: - - build-pkgs - update-pxe-network-installer - -该脚本在 ``/localdisk/loadbuild/my-project-flock/pxe-network-installer/output`` 上创建三个文件。 :: - - new-initrd.img - new-squashfs.img - new-vmlinuz - -请将文件重命名如下： :: - - initrd.img - squashfs.img - vmlinuz - -最后... :: - - build-pkgs --clean pxe-network-installer - build-pkgs pxe-network-installer - build-iso - - -更换包 ------------------------- - -**我应该把我新编译出的包放在哪一层？** - -如果软件包是你原创的内容，是为 StarlingX 项目所编写的，那么它属于flock layer。所有其他内容都被认为是第三方的， -要么进入distro layer，要么进入compiler layer。 - -用于编译或打包作用的核心组件，属于compiler layer. Compiler layer这一层改动较少，相对稳定。 - -所有其他的第三方的包都属于distro layer。在distro层其中你可以找到CentOS补丁包，内核包和驱动程序包，ceph，openstack 组件等和更多软件包。 - -**如何在repo manifest中添加新的项目？** - -如果需要一个新的 git repo，需要在default manifest和对应layer相关的manifest文件中修改设置。 - -**本地yum配置库如何变更？** - -希望我们不会经常添加新的 yum repos，如果需要，将它添加到``'stx-tools/centos-mirror/yum.reposit.d'``，而不是 -``'stx-tools/centos-mirror-tools/config///repos.yum.d’``。 - -**需要更新哪个'lst'文件？** - -如果要添加的包来自第三方的 tarball 或 src.rpm，那么将这个包添加到git根目录的 lst 文件中，编译指令就会找到它。 -你将内容添加在以下lst文件的其中一个，根据包的原则选择最合适的lst文件。 :: - - centos_srpms_3rdparties.lst - centos_srpms_centos3rdparties.lst - centos_srpms_centos.lst - centos_tarball-dl.lst - -对于编译时所依赖各个包，以及那些编译过程有传递性要求的包，应该添加到 ``stx-tools/centos-mirror-tools/config//`` -下的 lst 文件中。将内容添加在以下lst文件的其中一个，根据包的原则选择最合适的lst文件。 :: - - rpms_3rdparties.lst - rpms_centos3rdparties.lst - rpms_centos.lst - -...as appropriate. - -如果软件包需要安装到iso中，那么应该将这些包所依赖的包，以及有传递性要求的包添加到 ``stx-tools/centos-mirror-tools/config//flock`` -下的 lst 文件中。是flock目录下，而不是其它层，因为 ISO 是从flock层建立的。 - -弄清楚包的传递列表可能是一个挑战。对于centos软件包，我的建议是启动一个单独的centos容器（保持容器版本的正确匹配性），并尝试运行以下命令。 :: - - repoquery --requires --resolve --recursive \\ - --qf='%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}.rpm' - -... 否则，您可能不得不实施几次构建迭代，并修复每次中断的问题。 - -上述方法收集的rpm集合，可能会列出您的lst文件中已经存在的软件包。 如果版本相同，则无需执行任何操作。 如果版本较低， -则您可能需要将lst中版本更新为较新的软件包版本。检查软件包在所依赖的底层lst文件中是否存在，不存在就添加进去。 - -查看以上方法获得的rpm列表，对比lst文件，对于lst文件中尚不存在的rpm包。如果starlingx中已经构建了该rpm包， -则不需要包含在lst文件中。 - -lst中如果没有列出，而且我们也没有构建它，那么就需要添加这个rpm包。 - -**如何在iso中加入一个软件包？** - -将需要编译的软件包添加到git根目录下的``_iso_image.inc``文件下，编译系统会自动识别编译。 - -特定操作系统的(如CentOS)编译时基于的包，以往在 ``'cgcs-root/build-tools/build_iso/image.inc'`` 文件中配置， -可以继续使用这种方式。 -1, layer build以后，需要打包到iso中的package，应该都添加到各自git project里的 ``_iso_image.inc`` 里。 - -2, compile或者distro layer中增加iso中的package，在这两个layer build结束后，执行如下命令生成各自layer的image.inc -:: - - source build-tools/image-utils.sh - image_inc_list iso dev > my__image.inc - -cengna上的各自layer的 image.inc也是这样生成的。 - -在 ``stx-tools/centos-mirror-tools/config///required_layer_iso_inc.cfg`` 文件 -$ cat stx-tools/centos-mirror-tools/config/centos/distro/required_layer_iso_inc.cfg -compiler,std,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc -compiler,dev,https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc - -修改成你的distro 或者compile layer的 image.inc -``file:///localdisk/loadbuild///my__imajge.inc`` - -在build flock layer的时候会去读这个 ``require_layer_iso_inc.cfg`` 或者下面的命令修改layer的image.inc :: - - generate-local-repo.sh \\ - --layer-pkg-url=distro,std,file:///localdisk/loadbuild/-distro/std/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,rt,file:///localdisk/loadbuild/-distro/rt/rpmbuild/RPMS/rpm.lst \\ - --layer-pkg-url=distro,installer,file:///localdisk/loadbuild/-distro/installer/rpmbuild/RPMS/rpm.lst \\ - --layer-inc-url=distro,std,file:///localdisk/loadbuild/-distro/std/image.inc \\ - --layer-inc-url=distro,dev,file:///localdisk/loadbuild/-distro/std/image-dev.inc \\ - /import/mirrors/CentOS/stx/CentOS/ - -编译flock layer时候，会在``cgcs-root/local-repo/layer_image_inc``, 包含distro和compiler layer的image.inc -build-iso的时候会被使用。 diff --git a/doc/source/developer_resources/index.rst b/doc/source/developer_resources/index.rst index af7480b38..17bbc9814 100644 --- a/doc/source/developer_resources/index.rst +++ b/doc/source/developer_resources/index.rst @@ -9,9 +9,6 @@ Developer Resources :maxdepth: 1 build_guide - layered_build_guide - Layered_Build - Layered_Build_cn backup_restore build_docker_image code-submission-guide diff --git a/doc/source/developer_resources/layered_build_guide.rst b/doc/source/developer_resources/layered_build_guide.rst deleted file mode 100644 index 8d7bde36b..000000000 --- a/doc/source/developer_resources/layered_build_guide.rst +++ /dev/null @@ -1,848 +0,0 @@ -=================== -Layered Build Guide -=================== -This section describes the steps for layered build and building an ISO image -from a StarlingX R4.0 release onwards. - -For more detailed explanation of layered build, please refer :ref:`Layered-Build`. - -.. contents:: - :local: - :depth: 1 - -------------------- -:ref:`requirements` -------------------- -Layered build has the same requirements as StarlingX R3.0 build. -Click the heading above for details. - ------------------------------------- -:ref:`starlingx_build_tools` ------------------------------------- -Layered build uses the same development environment setup as StarlingX R3.0 build. -Click the heading above for details. - --------------------- -Build compiler layer --------------------- - -***************************** -Prepare the base Docker image -***************************** - -Since, we are building compiler layer, create and modify only localrc -file. Make sure to set the project and layer as compiler and user name as -desired. For example: - -:: - - # tbuilder localrc - MYUNAME= - PROJECT=compiler - HOST_PREFIX=$HOME/starlingx/workspace - HOST_MIRROR_DIR=$HOME/starlingx/mirror - LAYER=compiler - -.. _Build-the-CentOS-mirror-repository-compiler: - -********************************** -Build the CentOS mirror repository -********************************** - -The creation of the StarlingX ISO relies on a repository of RPM binaries, -RPM sources, and tar compressed files. This section describes how to build -this CentOS mirror repository. - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Run building Docker container -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -At the end of this procedure, you will be inside the Starlingx container. - -Quick command summary is as below: - -:: - - cd $HOME/tools/ - bash tb.sh env - bash tb.sh run - bash tb.sh exec - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Download source code repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Inside the building Docker container, start the internal environment: - - :: - - eval $(ssh-agent) - ssh-add - - -#. Use the repo tool to create a local clone of the manifest git repository - based on the “master” branch for compiler layer. - - :: - - cd $MY_REPO_ROOT_DIR - repo init -u https://opendev.org/starlingx/manifest -b master -m compiler.xml - -#. Synchronize code for compiler layer using command below: - - :: - - repo sync -j`nproc` - -~~~~~~~~~~~~~~~~~ -Download packages -~~~~~~~~~~~~~~~~~ - -#. Inside the Docker container, enter the following commands to download - the required packages to populate the CentOS mirror repository: - - :: - - cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh - -#. Monitor the download of packages until it is complete. When the download - is complete successfully with no missing packages, the following message appears: - - :: - - step #5: done successfully - sudo rm -rf /tmp/stx_mirror_4d2URF - IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images - for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz - - Success - -*************** -Verify packages -*************** - -#. Verify no missing or failed packages exist: - - :: - - cat logs/*_missing_*.log - cat logs/*_failmoved_*.log - -#. In case missing or failed packages do exist, which is usually caused by - network instability (or timeout), you need to download the packages - manually or re-run the download command given in previous step. - Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, - centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. - -****************** -Packages structure -****************** - -The following is a general overview of the packages structure resulting from -downloading the packages for compiler layer: - -:: - - /localdisk/designer///stx-tools/centos-mirror-tools/output - . - └── stx - └── CentOS - ├── Binary - │   ├── noarch - │   └── x86_64 - ├── downloads - │   └── puppet - ├── layer_build_info - ├── layer_image_inc - ├── layer_pkg_lists - ├── layer_wheels_inc - └── Source - -******************************* -Copy CentOS mirror repository -******************************* -Exit from the building Docker container. Run the following commands: - -#. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* - workspace directory: - - :: - - cd $HOME/starlingx/mirror/CentOS/ - -#. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* workspace - directory. Make sure to insert the : - - :: - - cp -r $HOME/starlingx/workspace/localdisk/designer//compiler/stx-tools/centos-mirror-tools/output/stx . - -*************************** -Create a tarball repository -*************************** - -#. Enter the StarlingX container using below command: - - :: - - cd $HOME/tools/ - ./tb.sh exec - -#. Copy downloaded CentOS tarballs into StarlingX repo - - :: - - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - - Alternatively, you can run the "populate_downloads.sh" script to copy the - tarballs instead of using a symlink: - - :: - - bash populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - -************** -Build packages -************** - -#. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / - corrupted rpm and tarball packages generated during - :ref:`Build-the-CentOS-mirror-repository-compiler`, which will cause the next step - to fail. If that step does fail, manually download those missing / - corrupted packages. - -#. Update the symbolic links: - - :: - - bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ - -#. Build the packages: - - :: - - build-pkgs - build-pkgs --installer - ------------------- -Build distro layer ------------------- - -.. _Build-the-CentOS-mirror-repository-distro: - -********************************** -Build the CentOS mirror repository -********************************** - -The creation of the StarlingX ISO relies on a repository of RPM binaries, -RPM sources, and tar compressed files. This section describes how to build -this CentOS mirror repository. - -If you were building compiler layer, exit from container and enter the tools -directory using below commands: - -:: - - cd $HOME/tools/ - -Since, we are building distro layer, modify localrc file. Make sure to set the -project and layer as distro. For example: - -:: - - # tbuilder localrc - MYUNAME= - PROJECT=distro - HOST_PREFIX=$HOME/starlingx/workspace - HOST_MIRROR_DIR=$HOME/starlingx/mirror - LAYER=distro - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Run building Docker container -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -At the end of this procedure, you will be inside the Starlingx container. - -Quick command summary is as below: - -:: - - cd $HOME/tools/ - bash tb.sh env - bash tb.sh run #Need not give, If the container was created earlier. - bash tb.sh exec - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Download source code repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Inside the building Docker container, start the internal environment: - - :: - - eval $(ssh-agent) - ssh-add - - -#. Use the repo tool to create a local clone of the manifest git repository - based on the “master” branch for compiler layer. - - :: - - cd $MY_REPO_ROOT_DIR - repo init -u https://opendev.org/starlingx/manifest -b master -m distro.xml - -#. Synchronize code for compiler layer using command below: - - :: - - repo sync -j`nproc` - -~~~~~~~~~~~~~~~~~ -Download packages -~~~~~~~~~~~~~~~~~ - -#. Inside the Docker container, enter the following commands to download - the required packages to populate the CentOS mirror repository: - - :: - - cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh - -#. Monitor the download of packages until it is complete. When the download - is complete, the following message appears: - - :: - - step #5: done successfully - sudo rm -rf /tmp/stx_mirror_UIQ675 - IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images - for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz - - Success - -*************** -Verify packages -*************** - -#. Verify no missing or failed packages exist: - - :: - - cat logs/*_missing_*.log - cat logs/*_failmoved_*.log - -#. In case missing or failed packages do exist, which is usually caused by - network instability (or timeout), you need to download the packages - manually. - Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, - centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. - -****************** -Packages structure -****************** - -The following is a general overview of the packages structure resulting from -downloading the packages for distro layer: - -:: - - /localdisk/designer//distro/stx-tools/centos-mirror-tools/output - . - └── stx - └── CentOS - ├── Binary - │   ├── EFI - │   │   └── BOOT - │   │   └── fonts - │   ├── images - │   │   └── pxeboot - │   ├── isolinux - │   ├── LiveOS - │   ├── noarch - │   └── x86_64 - ├── downloads - │   └── puppet - │   └── packstack - │   └── puppet - │   └── modules - ├── layer_build_info - ├── layer_image_inc - ├── layer_pkg_lists - ├── layer_repos - │   └── compiler - │   └── std - │   ├── repodata - │   └── repodata.upstream - ├── layer_wheels_inc - └── Source - -******************************* -Copy CentOS mirror repository -******************************* -Exit from the building Docker container. Run the following commands: - -#. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* - workspace directory: - - :: - - cd $HOME/starlingx/mirror/CentOS/ - -#. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* workspace - directory. Make sure to insert the : - - :: - - cp -r $HOME/starlingx/workspace/localdisk/designer//distro/stx-tools/centos-mirror-tools/output/stx . - -*************************** -Create a tarball repository -*************************** - -#. Enter the StarlingX container using below command: - - :: - - cd $HOME/tools/ - ./tb.sh exec - -#. Copy downloaded CentOS tarballs into StarlingX repo - - :: - - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - - Alternatively, you can run the "populate_downloads.sh" script to copy the - tarballs instead of using a symlink: - - - :: - - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - - -************** -Build packages -************** - -#. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / - corrupted rpm and tarball packages generated during - :ref:`Build-the-CentOS-mirror-repository-distro`, which will cause the next step - to fail. If that step does fail, manually download those missing / - corrupted packages. - -#. Update the symbolic links: - - :: - - bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ - - This step creates the repo directory, following is the output on the console - for a successful repo directory creation for the stx: - - :: - - Copying comps.xml file. - Createing yum repodata. - Directory walk started - Directory walk done - 51 packages - Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Source/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 0 packages - Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Source/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 1450 packages - Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 0 packages - Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Binary/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Copying mock.cfg.proto file. - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/efiboot.img - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/boot.msg - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/grub.conf - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/initrd.img - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.bin - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/memtest - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/splash.png - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vmlinuz - Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/LiveOS/squashfs.img - Creating folder EFI - Creating folder EFI/BOOT - Creating folder EFI/BOOT/fonts - Creating folder images - Creating folder images/pxeboot - Creating folder isolinux - Creating folder LiveOS - Done creating repo directory - -#. Build the packages: - - :: - - build-pkgs - build-pkgs --installer - - ------------------ -Build flock layer ------------------ - -.. _Build-the-CentOS-mirror-repository-flock: - -********************************** -Build the CentOS mirror repository -********************************** - -The creation of the StarlingX ISO relies on a repository of RPM binaries, -RPM sources, and tar compressed files. This section describes how to build -this CentOS mirror repository. - -If you were building distro layer, exit from container and enter the tools -directory using below commands: - -:: - - cd $HOME/tools/ - -Since, we are building flock layer, modify localrc file. Make sure to set the -project and layer as flock. For example: - -:: - - # tbuilder localrc - MYUNAME= - PROJECT=flock - HOST_PREFIX=$HOME/starlingx/workspace - HOST_MIRROR_DIR=$HOME/starlingx/mirror - LAYER=flock - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Run building Docker container -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -At the end of this procedure, you will be inside the Starlingx container. - -Quick command summary is as below: - -:: - - cd $HOME/tools/ - bash tb.sh env - bash tb.sh run #Need not give, If the container was created earlier. - bash tb.sh exec - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Download source code repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -#. Inside the building Docker container, start the internal environment: - - :: - - eval $(ssh-agent) - ssh-add - - -#. Use the repo tool to create a local clone of the manifest git repository - based on the “master” branch for flock layer. - - :: - - cd $MY_REPO_ROOT_DIR - repo init -u https://opendev.org/starlingx/manifest -b master -m flock.xml - -#. Synchronize code for compiler layer using command below: - - :: - - repo sync -j`nproc` - -~~~~~~~~~~~~~~~~~ -Download packages -~~~~~~~~~~~~~~~~~ - -#. Inside the Docker container, enter the following commands to download - the required packages to populate the CentOS mirror repository: - - :: - - cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh - -#. Monitor the download of packages until it is complete. When the download - is complete, the following message appears: - - :: - - step #5: done successfully - sudo rm -rf /tmp/stx_mirror_ievdiA - IMPORTANT: The following 3 files are just bootstrap versions. Based - on them, the workable images for StarlingX could be generated by - running "update-pxe-network-installer" command after "build-iso" - - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz - -*************** -Verify packages -*************** - -When the download is not successful, the following message appears: - -:: - - IMPORTANT: The following 3 files are just bootstrap versions. Based - on them, the workable images for StarlingX could be generated by - running "update-pxe-network-installer" command after "build-iso" - - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz - - Warning: Not all download steps succeeded. You are likely missing files. - - -You can verify and list missing or failed packages using below command: - -:: - - cat logs/*_missing_*.log - cat logs/*_failmoved_*.log - -In case missing or failed packages do exist, which is usually caused by -network instability (or timeout), you need to download the packages -manually. -Doing so assures you get all RPMs listed in *centos_srpms_3rdparties.lst, -centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst*. - -****************** -Packages structure -****************** - -The following is a general overview of the packages structure resulting from -downloading the packages for flock layer: - -:: - - /localdisk/designer//flock/stx-tools/centos-mirror-tools/output - - └── stx - └── CentOS - ├── Binary - │   ├── EFI - │   │   └── BOOT - │   │   └── fonts - │   ├── images - │   │   └── pxeboot - │   ├── isolinux - │   ├── LiveOS - │   ├── noarch - │   └── x86_64 - ├── downloads - │   └── puppet - ├── layer_build_info - ├── layer_image_inc - ├── layer_pkg_lists - ├── layer_repos - │   ├── compiler - │   │   └── std - │   │   ├── repodata - │   │   └── repodata.upstream - │   └── distro - │   ├── installer - │   │   ├── repodata - │   │   └── repodata.upstream - │   ├── rt - │   │   ├── repodata - │   │   └── repodata.upstream - │   └── std - │   ├── repodata - │   └── repodata.upstream - ├── layer_wheels_inc - └── Source - - -******************************* -Copy CentOS mirror repository -******************************* -Exit from the building Docker container. Run the following commands: - -#. Navigate to CentOS mirror directory *mirror/CentOS* under your *starlingx* - workspace directory: - - :: - - cd $HOME/starlingx/mirror/CentOS/ - -#. Copy the built CentOS mirror repository *$HOME/starlingx/mirror/* - workspace directory: - - :: - - cp -r $HOME/starlingx/workspace/localdisk/designer//flock/stx-tools/centos-mirror-tools/output/stx . - -*************************** -Create a tarball repository -*************************** - -#. Enter the StarlingX container using below command: - - :: - - cd $HOME/tools/ - ./tb.sh exec - -#. Copy downloaded CentOS tarballs into StarlingX repo - :: - - ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/ - - Alternatively, you can run the "populate_downloads.sh" script to copy the - tarballs instead of using a symlink: - - :: - - populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/ - -#. Exit from the container. On the host machine, create mirror binaries: - - :: - - mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer - cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img - cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz - cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img - -************** -Build packages -************** - -#. Enter the StarlingX container using below command: - - :: - - cd $HOME/tools/ - ./tb.sh exec - -#. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing / - corrupted rpm and tarball packages generated during - :ref:`Build-the-CentOS-mirror-repository-flock`, which will cause the next step - to fail. If that step does fail, manually download those missing / - corrupted packages. - -#. Update the symbolic links: - - :: - - bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/ - - Following is the output: - - :: - - Copying comps.xml file. - Createing yum repodata. - Directory walk started - Directory walk done - 1 packages - Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Source/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 0 packages - Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Source/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 1892 packages - Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Directory walk started - Directory walk done - 40 packages - Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Binary/.repodata/ - Preparing sqlite DBs - Pool started (with 5 workers) - Pool finished - Copying mock.cfg.proto file. - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/efiboot.img - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/boot.msg - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/grub.conf - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/initrd.img - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.bin - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/memtest - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/splash.png - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vmlinuz - Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/LiveOS/squashfs.img - Creating folder EFI - Creating folder EFI/BOOT - Creating folder EFI/BOOT/fonts - Creating folder images - Creating folder images/pxeboot - Creating folder isolinux - Creating folder LiveOS - Done creating repo directory - - -#. Build the packages: - - :: - - build-pkgs - -------------------- -Build StarlingX ISO -------------------- - -Build the image: - -:: - - build-iso - ----------------------- -Build-installer ----------------------- -Layered build has the same procedure for build installer as StarlingX R3.0 build -except for the changes in path of files as below. Click the heading above for details. - -#. The steps covered by the script **update-pxe-network-installer** is detailed in - $MY_REPO/stx/stx-metal/installer/initrd/README. This script creates three files on - /localdisk/loadbuild/stx/flock/pxe-network-installer/output. - -#. The path for **build_srpm.data** is $MY_REPO/stx/metal/installer/pxe-network-installer/centos/. -