diff --git a/doc/source/developer_resources/build_guide.rst b/doc/source/developer_resources/build_guide.rst index d414c263c..cdfa34095 100644 --- a/doc/source/developer_resources/build_guide.rst +++ b/doc/source/developer_resources/build_guide.rst @@ -384,6 +384,7 @@ Exit from the building Docker container. Run the following commands: cp -r $HOME/starlingx/workspace/localdisk/designer///stx-tools/centos-mirror-tools/output/stx-r1 $HOME/starlingx/mirror/CentOS/ +.. _create_stx_pkgs: ------------------------- Create StarlingX packages diff --git a/doc/source/developer_resources/figures/pkging_patch_comp.png b/doc/source/developer_resources/figures/pkging_patch_comp.png new file mode 100644 index 000000000..1bd4a0b4b Binary files /dev/null and b/doc/source/developer_resources/figures/pkging_patch_comp.png differ diff --git a/doc/source/developer_resources/index.rst b/doc/source/developer_resources/index.rst index df111ec44..bf6be58f7 100644 --- a/doc/source/developer_resources/index.rst +++ b/doc/source/developer_resources/index.rst @@ -20,6 +20,7 @@ Developer Resources mirror_repo move_to_new_openstack_version_in_starlingx navigate_source_code + packaging_ref Project Specifications architecture_docs starlingx_patching diff --git a/doc/source/developer_resources/packaging_ref.rst b/doc/source/developer_resources/packaging_ref.rst new file mode 100644 index 000000000..c2c4991b8 --- /dev/null +++ b/doc/source/developer_resources/packaging_ref.rst @@ -0,0 +1,1176 @@ +=================== +Packaging Reference +=================== + +This guide contains information about creating StarlingX packages. + +.. contents:: + :local: + :depth: 1 + +--------------- +Tips and tricks +--------------- + +Here are some things to consider when creating packages: + +* Make sure you have the correct BuildRequires dependencies listed. Mock will + ensure that these packages are installed in the build environment before + trying to build the package in question. Be aware that CentOS often uses + ``-devel`` for developer packages (which typically include headers and the + ``.so`` shared library link) whereas StarlingX generally uses ``-dev``. For + BuildRequires, you can use the following: + + * Instead of ``virtual/x86_64-wrs-linux-gcc``, you can use ``gcc``. + * Instead of ``glib-2.0``, use ``glib2-devel``. + * Instead of ``sqlite3``, use ``sqlite-devel``. + * You can comment out or delete ``virtual/libc`` and + ``virtual/x86_64-wrs-linux-compilerlibs``. + +* Typically, you do not need to manually add dependencies due to shared + libraries to either Requires or Provides in the spec file. RPM will + usually add them automatically. + +* When your package creates a shared library, make sure to install it with 755 + (or other suitably executable) permissions. The automatic dependency + handling only runs if the library file is actually executable. + +* It can be useful to refer to a bitbake-generated spec file, especially for + STX-specific packages. Generally the ``build`` and ``install`` phases are + missing from the bitbake-generated spec file and you must add them. + +* Make sure all necessary files are listed, because the build_srpm phase will + only include source files listed in your spec file. In a simple case, this + can be just the following line: ``Source: %{name}-%{version}.tar.gz`` + +------------------------------------ +STX authored package with a makefile +------------------------------------ + +This section uses ``fm-common`` as a sample package to convert. + +#. Locate the root directory for the package. + + :: + + cd wr-cgcs/layers/cgcs/middleware/fault/recipes-common/fm-common + +#. Create a directory template in the same location as your current bitbake + recipe. The presence of a ``centos`` directory will identify packages to be + compiled for STX on CentOS. + + :: + + mkdir -p centos + mkdir -p centos/files + +#. The makefile requires an ``install`` build target. However, bitbake will try + to use ``install`` if it is introduced when it wasn't previously present, so + we'll use ``install_non_bb`` instead. + + The makefile takes the following parameters: ``DEST_DIR``, ``BIN_DIR``, + ``LIB_DIR``, ``INC_DIR``, ``MAJOR``, ``MINOR``. + + The basic strategy is to construct the directory tree using + ``install -m 755 -d some_dir``, then copy files using + ``install -m 755 from_file some_dir/to_file`` + + For versioned libraries, you must create non- or lesser-versioned symlinks, + for example: + + :: + + ln -s my_lib.so.1.0 some_dir/my_lib.so.1 + + Typically, you do not need the ``.so`` installed on the target (it's + generally only needed for building against). If you do, then it would be + linked as ``ln -s my_lib.so.1.0 some_dir/my_lib.so`` and would usually be + part of a ``-dev`` package. Don't forget tabs!!! + +:: + + vi sources/Makefile + install_non_bb: + install -m 755 -d $(DEST_DIR)$(BIN_DIR) + install -m 755 -d $(DEST_DIR)$(LIB_DIR) + install -m 755 -d $(DEST_DIR)$(INC_DIR) + install -m 755 fmClientCli $(DEST_DIR)$(BIN_DIR) + install -m 644 fmDbAPI.h $(DEST_DIR)$(INC_DIR) + install -m 644 fmAPI.h $(DEST_DIR)$(INC_DIR) + install -m 644 fmThread.h $(DEST_DIR)$(INC_DIR) + install -m 644 fmAlarm.h $(DEST_DIR)$(INC_DIR) + install -m 644 $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR) + ln -s $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR)/$(LIBFMCOMMON_SO).$(MAJOR) + ln -s $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR)/$(LIBFMCOMMON_SO) + +*********************** +Create an RPM spec file +*********************** + +Put the RPM spec file under the ``centos`` directory. This is the rpm version +of a bitbake recipe. + +:: + + vi centos/.spec + +RPM spec files support macros and variable. Reference the macro with the +``%{macro_name}`` syntax. Many macros are already defined to help you, such as, +``%{version}`` refers to the ``Version:`` entry in the description section. You +can also define your own macros, for example ``%define local_dir /usr/local`` +creates the macro ``local_dir``. Fill in the following sections: + +:: + + # A short description of the packaged software + summary: + + # Defines what the package will actually be called + Name: + + # Should be set to the version of the software being packaged + Version: + + # A number that is used to represent the number of times the software, at the present version, has been packaged + Release: + + # Give a list of the source files found in the $BUILD_DIR/SOURCES subdirectory + # If git managed, usually just one, a tarball that we will create via build script e.g. Source0: %{name}-%{version}.tar.gz + # Alternatively name the tarball that is brought in via COPY_LIST from $STX_BASE/downloads directory. + # Many source files can be specified... list any additional files in $BUILD_DIR/SOURCES (likely placed there via COPY_LIST) + Source0: %{name}-%{version}.tar.gz + Source1: %{name}.conf + Source2: %(name}.init + + # List of build requirements + BuildRequires: + + # List of runtime requirements + Requires: + + # Describe the package + %description + + # Describes how to unpack the compressed packages (the setup macro will do it for you) + %prep + %setup + + # invoke 'make' + %build + + # invoke 'make install' + %install + + # Perhaps call 'make clean' or other cleanup scripts + %clean + + # List of files to include in the rpm + %files + + # Apply these permission/ownership overrides and the files that follow + %defattr(-,root,root,-) + + # Send this file to the standard path for documentation + %doc some_file + + # Create a directory when the rpm is installed + %dir some_directory + + # Install this file on this path + %some_file_path + + +One spec file can create multiple related rpm files. One common usage is to +place the ``*.so`` file (but not the ``*.so.*`` files) for a shared library and +its ``.h`` interface files in a ``-dev`` or ``-devel`` package. The Bitbake +naming convention is ``-dev``, while CentOS uses ``-devel``. Most of the +StarlingX packages use the ``-dev`` naming convention. + +You can create additional packages with ``%package -n package_name``, or +``%package suffix`` which creates a package named ``first_package_name-suffix``. + +Following ``%package``, add a unique ``Summary:``, and override any values from the +first package. These additional packages also require a ``%description, %files`` +sections, and may require ``%prep, %setup, %build, %install, %clean`` sections. All +these extra sections use the same ``-n package_name`` or ``suffix`` arguments to +define which package they apply to. Refer to the example below, which uses +``fm-common-dev``. + +:: + + %package -n fm-common-dev Summary: STX Platform Fault Management + Common Package - Development files Group: devel Requires: fm-common = + %{version}-%{release} ... %description -n fm-common-dev ... %files -n + fm-common-dev ... + +:: + + vi centos/fm-common.spec + %define local_dir /usr/local + %define local_bindir %{local_dir}/bin + + Summary: STX Platform Fault Management Common Package + Name: fm-common + Version: 1.0 + Release: r26.0 + License: Apache-2.0 + Group: base + URL: unknown + Source0: %{name}-%{version}.tar.gz + BuildRequires: util-linux + BuildRequires: postgresql-devel + BuildRequires: libuuid-devel + Requires: postgresql + + %package -n fm-common-dev + Summary: STX Platform Fault Management Common Package - Development files + Group: devel + Requires: fm-common = %{version}-%{release} + + %description + STX platform Fault Management Client Library that provides APIs for + applications to raise/clear/update active alarms. + + %description -n fm-common-dev + STX platform Fault Management Client Library that provides APIs for + applications to raise/clear/update active alarms. This package contains + symbolic links, header files, and related items necessary for software + development. + + %prep + %setup + %build + VER=%{version} + MAJOR=`echo $VER | awk -F . '{print $1}'` + MINOR=`echo $VER | awk -F . '{print $2}'` + make MAJOR=$MAJOR MINOR=$MINOR %{?_smp_mflags} + + %install + rm -rf $RPM_BUILD_ROOT + VER=%{version} + MAJOR=`echo $VER | awk -F . '{print $1}'` + MINOR=`echo $VER | awk -F . '{print $2}'` + make DEST_DIR=$RPM_BUILD_ROOT BIN_DIR=%{local_bindir} LIB_DIR=%{_libdir} INC_DIR=%{_includedir} MAJOR=$MAJOR MINOR=$MINOR install_non_bb + + %clean + rm -rf $RPM_BUILD_ROOT + + %files + %defattr(-,root,root,-) + %doc LICENSE + %dir %{local_dir} + %dir %{local_bindir} + %{local_bindir}/* + + %files -n fm-common-dev + %defattr(-,root,root,-) + %{_includedir}/* + %{_libdir}/* + +************************ +Set up srpm build script +************************ + +This section describes two options: you can choose to use the default srpm build +script or create your own srpm build script. + +#. Use the default srpm build script, and just provide a few data definitions, + in bash format. + + :: + + vi centos/build_srpm.data + Variables you can define. All paths are relative to PKG_BASE (see below) unless prefixed by another environment variable, eg. STX_BASE + SRC_DIR=my_src_dir + # Tar everything found in this subdirectory. Define this if source need to be collected into a tarball in $BUILD_DIR/SOURCES. + # Tar file name and version are derived from your .spec file and/or PKG-INFO file. Alternatively you may define TAR_NAME and VERSION in your 'build_srpm.data' file + + COPY_LIST="$STX_BASE/downloads/my_tar.tar.gz my_extra_files_dir my_script_dir/my_script" + # A Space separated list of paths to copy. In the build directory they will by copied into the $BUILD_DIR/SOURCES directory alongside any tarball you may have created via SRC_DIR. + + EXCLUDE_LIST_FROM_TAR=my_omit_dir my_script_dir/my_omit_script" + # Omit these paths under SRC_DIR from the tar + + TAR_NAME=nova + VERSION=1.1 + # The name and version of the package. Assuming SRC_DIR is specified, a tarball with the name "$TAR_NAME-$VERSION.tar.gz" will be created from from the contents of $SRC_DIR. + # NOTE: These values are pulled automatically from your .spec ("%global service" or "Name:" and "Version:") or PKG-INFO ("Name:" and "Version:"). You can override the automatic valuation by supplying your own values. + + COPY_LIST_TO_TAR=my_extra_files_dir my_script_dir/my_script" + # A Space separated list of paths to copy. These are added to the tarball itself. Try to avoid using this... we prefer that tarballs of 3rd party origin remain unaltered. + + TIS_BASE_SRCREV=9af04d8480da4359f766e59c45ac23f199b138b0 + # Last git commit prior to start of TC content + + TIS_PATCH_VER=GITREVCOUNT+3 + # Version is passed into .spec file as rpmmacro to modify package release string. + # Can contain 'GITREVCOUNT', the count of commits since TIS_BASE_SRCREV + + BUILD_IS_BIG=6 + # An estimate of the disk requirements in GB. This package won't build on a tmpfs based build environment small than this value. + # If a build fails on a tmpfs based node due to disk space exhaustion, then at minimum this value must be greater than the tpmfs size + # as seen in the "MOCKCHAIN_RESOURCE_ALLOCATION=0:10:10:5" log. + + BUILD_IS_SLOW5 + # An estimate of the build time on disk in minutes. + # A larger value will tell the build system to schedule this job sooner + # rather than later, to maximize parallelism. + +#. Create your own srpm build script. The goal of the script is to use the + rpmbuild tool to create one or more ``.src.rpm`` files in the ``$BUILD_DIR/SRPMS`` + directory. The directory itself will be created by rpmbuild. + + :: + + vi centos/build_srpm + + A sample build script might look like this: + + :: + + source "$SRC_BASE/build-tools/spec-utils" + + SRC_DIR=$STX_BASE/git/libvirt + + if [ -f PKG-INFO ]: then + VERSION=$(grep '^Version:' PKG-INFO | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//') + TAR_NAME=$(grep '^Name:' PKG-INFO | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//') + else + SPEC_PATH=$(ls $BUILD_DIR/SPECS/*.spec) + VERSION=$(spec_evaluate '%{version}' "$SPEC_PATH" 2>> /dev/null) + NAME=$(spec_evaluate '%{name} "$SPEC_PATH" 2>> /dev/null) + fi + + BUILD_DIR="$RPMBUILD_BASE" + EXTRA_FILES="libvirt-1.2.12/libvirt.logrotate " + EXTRA_FILES+="libvirt-1.2.12/libvirt.lxc " + EXTRA_FILES+="libvirt-1.2.12/libvirt.qemu " + EXTRA_FILES+="libvirt-1.2.12/libvirt.uml " + + mkdir -p $BUILD_DIR/SRPMS + + TAR="$TAR_NAME-$VERSION.tar.gz" + TAR_PATH="$BUILD_DIR/SOURCES/$TAR" + + TAR_NEEDED=0 + if [ -f $TAR_PATH ]; then + n=`find . -cnewer $TAR_PATH -and ! -path './.git*' \ + -and ! -path './build/*' \ + -and ! -path './.pc/*' \ + -and ! -path './patches/*' \ + -and ! -path './$DISTRO/*' \ + -and ! -path './pbr-*.egg/*' \ + | wc -l` + if [ $n -gt 0 ]; then + TAR_NEEDED=1 + fi + else + TAR_NEEDED=1 + fi + + if [ $TAR_NEEDED -gt 0 ]; then + tar czvf $TAR_PATH $EXTRA_FILES \ + -C $(dirname $SRC_DIR) $(basename $SRC_DIR) \ + --exclude '.git/' \ + --exclude '.git*' \ + --transform "s,^libvirt-1.2.12,extra," \ + --transform "s,^$(basename $SRC_DIR),$TAR_NAME-$VERSION," + fi + + + for SPEC in `ls $BUILD_DIR/SPECS`; do + SPEC_PATH="$BUILD_DIR/SPECS/$SPEC" + RELEASE=$(grep '^Release:' $SPEC_PATH | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//') + NAME=`echo $SPEC | sed 's/.spec$//'` + SRPM="$NAME-$VERSION-$RELEASE.src.rpm" + SRPM_PATH="$BUILD_DIR/SRPMS/$SRPM" + + BUILD_NEEDED=0 + if [ -f $SRPM_PATH ]; then + n=`find . -cnewer $SRPM_PATH | wc -l` + if [ $n -gt 0 ]; then + BUILD_NEEDED=1 + fi + else + BUILD_NEEDED=1 + fi + + if [ $BUILD_NEEDED -gt 0 ]; then + rpmbuild -bs $SPEC_PATH --define="%_topdir $BUILD_DIR" + fi + done + +The idea is to tar up any source from git into a tarball into +``$BUILD_DIR/SOURCES`` and/or copy tarballs from a downloads directory. Copy any +supporting files into ``$BUILD_DIR/SOURCES`` also. Finally, use rpmbuild to +create a ``.src.rpm``. DO NOT try to compile the binary ``.rpm``. + +*************************** +Details for build-srpm.data +*************************** + +This section describes the contents of the ``build-srpm.data`` file, which +customizes the setup when you build :abbr:`SRPM (Source RPM)` files. + +**Pre-defined path variables** + +You may use any of the following: + +:: + + SRC_BASE # Usually the same as $MY_REPO + + STX_BASE # $MY_REPO/stx + + PKG_BASE # Base dir for the package. Listed in centos_pkg_dir. + # Directory in which CentOS directory is found. + + PATCHES_BASE # Directory where source patches for the package can be found, typically: $PKG_BASE/$DISTRO/patches + + FILE_BASE # Directory where supplemental files for the package can be found, typically: $PKG_BASE/$DISTRO/files + # Files that can be copied into $BUILD_DIR/SOURCES prior to assembling the .src.rpm + + CGCS_BASE # same as $STX_BASE, being phased out in favour of PKG_BASE + # May be removed in a future release, do not use. + +**User-defined variables** + +You may define any of the following variables: + +:: + + SRC_DIR=my_src_dir + # Tar everything found in this subdirectory. Define this if source need to be collected into a tarball in $BUILD_DIR/SOURCES. + # Tar file name and version are derived from your .spec file and/or PKG-INFO file. Alternatively you may define TAR_NAME and VERSION in your 'build_srpm.data' file + + COPY_LIST="$STX_BASE/downloads/my_tar.tar.gz my_extra_files_dir my_script_dir/my_script" + # A Space separated list of paths to copy. In the build directory they will by copied into the $BUILD_DIR/SOURCES directory alongside any tarball you may have created via SRC_DIR. + + EXCLUDE_LIST_FROM_TAR=my_omit_dir my_script_dir/my_omit_script" + # Omit these paths under SRC_DIR from the tar + + TAR_NAME=nova + VERSION=1.1 + # The name and version of the package. Assuming SRC_DIR is specified, a tarball with the name "$TAR_NAME-$VERSION.tar.gz" will be created from from the contents of $SRC_DIR. + # NOTE: These values are pulled automatically from your .spec ("%global service" or "Name:" and "Version:") or PKG-INFO ("Name:" and "Version:"). You can override the automatic valuation by supplying your own values. + + COPY_LIST_TO_TAR=my_extra_files_dir my_script_dir/my_script" + # A Space separated list of paths to copy. These are added to the tarball itself. Try to avoid using this... we prefer that tarballs of 3rd party origin remain unaltered. + + BUILD_IS_BIG=6 + # An estimate of the disk requirements in GB. This package won't build on a tmpfs based build environment small than this value. + # If a build fails on a tmpfs based node due to disk space exhaustion, then at minimum this value must be greater than the tpmfs size + # as seen in the "MOCKCHAIN_RESOURCE_ALLOCATION=0:10:10:5" log. + + BUILD_IS_SLOW=5 + # An estimate of the build time on disk in minutes. + # A larger value will tell the build system to schedule this job sooner rather than later, to maximize parallelism. + + TIS_PATCH_VER= + # An integer, or one of the supported + # variables listed below, or the sum of + # variables and integers. + # e.g. + # TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5 + + PKG_GITREVCOUNT # Count git revisions relative to PKG_BASE. + # Optionally only count from PKG_BASE_SRCREV + + SRC_GITREVCOUNT # Count git revisions relative to SRC_DIR. + # Optionally only count from SRC_BASE_SRCREV + + GITREVCOUNT # Deprecated, please use SRC_GITREVCOUNT instead. + # Count git revisions relative to SRC_DIR. + # Optionally only count from TIS_BASE_SRCREV + + OTHER_GITREVCOUNT # count git revisions from all sources excluding + # PKG_BASE and SRC_DIR + # Optionally only count from + # BASE_SRCREV_FOR_PATH[] + + PKG_BASE_SRCREV= # Limit PKG_GITREVCOUNT revision count to + # commits since + + SRC_BASE_SRCREV= # Limit SRC_GITREVCOUNT revision count to + # commits since + + TIS_BASE_SRCREV= # Deprecated, please use SRC_BASE_SRCREV + # instead + # Limit GITREVCOUNT revision count to commits + # since + + BASE_SRCREV_FOR_PATH[]=[|OTHER_PKG_BASE_SRCREV] + # Limit OTHER_GITREVCOUNT revision count for + # commits under to commits since . + # If is the PKG_BASE of another package + # (not the current package) then the keyword + # 'OTHER_PKG_BASE_SRCREV' can be used to extract + # the 'PKG_BASE_SRCREV' value of the other + # package. + # + # The can reference variables like + # $STX_BASE and $GIT_BASE. + + OPT_DEP_LIST= # Add a space separated list of paths that + # don't contribute to the content of a src.rpm + # but do contribute to triggering a rebuild, + # and possibly modifying the TIS_PATCH_VER via + # use of OTHER_GITREVCOUNT. + + OPT_DEP_LIST_FOR_BUILD_TYPE[]= + # For a specific build type only, add a space + # separated list of paths that don't + # contribute to the content of src.rpm, + # but do contribute to triggering a + # rebuild, and possibly modifying the + # TIS_PATCH_VER via use of OTHER_GITREVCOUNT. + + +************************************* +Add your package to package list file +************************************* + +The file name is ``centos_pkg_dirs``. There is a separate package list file for +each git repo. When adding your package, use the relative path from the root of +the git to the desired ``PKG_BASE``. + +Using ``package libvirt`` as an example: + +* Spec file: ``$MY_REPO/stx/integ/virt/libvirt/centos/libvirt.spec`` + +* ``PKG_BASE``: ``%BLUE%$MY_REPO/stx%ENDCOLOR%/%RED%integ/virt/libvirt%ENDCOLOR%`` + +* Root of its git tree: ``$MY_REPO/stx`` + +* ``centos_pkg_dirs`` file: ``%BLUE%$MY_REPO/stx%ENDCOLOR%/centos_pkg_dirs`` + +The new entry needs to be: ``%RED%integ/virt/libvirt%ENDCOLOR% +echo "virt/libvirt" >> "$MY_REPO/stx/integ/centos_pkg_dirs"`` + +****************** +Build your package +****************** + +Complete instructions can be found in the +:ref:`Create StarlingX packages section of the Build Guide `. + +:: + + build-rpms fm-common + +******************* +Commit your changes +******************* + +:: + + git add centos/files/* + git add centos/fm-common.spec + git add centos/build_srpm* + + # If you added PKG-INFO + git add PKG-INFO + + git add sources/Makefile + git commit + +-------------------------- +Package without a makefile +-------------------------- + +In this case, the process is generally similar to the above instructions, but +the ``%build`` section has alternative commands or may be omitted altogether. +The install phase is done directly in the RPM spec file. + +For example, the ``collector`` package has no makefile. Its spec file is at +``stx/utilities/tools/collector/centos/collector.spec``. There is no "build" +phase, because the files are scripts. However, the files must be unpacked and +installed into the appropriate locations. The relevant bits look like this: + +:: + + Source0: %{name}-%{version}.tar.gz + + %prep + %setup + + %install + mkdir -p %{buildroot} + + install -d 755 -d %{buildroot}%{_sysconfdir}/collect.d + install -d 755 -d %{buildroot}%{_sysconfdir}/collect + install -d 755 -d %{buildroot}/usr/local/sbin + install -d 755 -d %{buildroot}/usr/local/bin + install -d 755 -d %{buildroot}%{_sbindir} + + install -m 755 collect %{buildroot}/usr/local/sbin/collect + install -m 755 collect_host %{buildroot}/usr/local/sbin/collect_host + install -m 755 collect_date %{buildroot}/usr/local/sbin/collect_date + install -m 755 collect_utils %{buildroot}/usr/local/sbin/collect_utils + install -m 755 collect_parms %{buildroot}/usr/local/sbin/collect_parms + install -m 755 collect_mask_passwords %{buildroot}/usr/local/sbin/collect_mask_passwords + install -m 755 expect_done %{buildroot}/usr/local/sbin/expect_done + + install -m 755 collect_sysinv.sh %{buildroot}%{_sysconfdir}/collect.d/collect_sysinv + install -m 755 collect_psqldb.sh %{buildroot}%{_sysconfdir}/collect.d/collect_psqldb + install -m 755 collect_openstack.sh %{buildroot}%{_sysconfdir}/collect.d/collect_openstack + install -m 755 collect_networking.sh %{buildroot}%{_sysconfdir}/collect.d/collect_networking + install -m 755 collect_ceph.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ceph + install -m 755 collect_sm.sh %{buildroot}%{_sysconfdir}/collect.d/collect_sm + install -m 755 collect_tc.sh %{buildroot}%{_sysconfdir}/collect.d/collect_tc + install -m 755 collect_nfv_vim.sh %{buildroot}%{_sysconfdir}/collect.d/collect_nfv_vim + install -m 755 collect_ovs.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ovs + install -m 755 collect_patching.sh %{buildroot}%{_sysconfdir}/collect.d/collect_patching + install -m 755 collect_coredump.sh %{buildroot}%{_sysconfdir}/collect.d/collect_coredump + install -m 755 collect_crash.sh %{buildroot}%{_sysconfdir}/collect.d/collect_crash + install -m 755 collect_ima.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ima + install -m 755 collect_fm.sh %{buildroot}%{_sysconfdir}/collect.d/collect_fm + install -m 755 collect_containerization.sh %{buildroot}%{_sysconfdir}/collect.d/collect_containerization + install -m 755 collect_dc.sh %{buildroot}%{_sysconfdir}/collect.d/collect_dc + + install -m 755 etc.exclude %{buildroot}%{_sysconfdir}/collect/etc.exclude + install -m 755 run.exclude %{buildroot}%{_sysconfdir}/collect/run.exclude + + ln -sf /usr/local/sbin/collect %{buildroot}/usr/local/bin/collect + ln -sf /usr/local/sbin/collect %{buildroot}%{_sbindir}/collect + +Either the ``setup`` or ``autosetup`` macro can be used to unpack the Source0 +tarball in ``%{_builddir}``. Next, a number of files are explicitly installed +into ``%{buildroot}``. Most of these come from the unpacked tarball. One file +comes from the original source directory, since it wasn't moved over to +``%{_builddir}`` in the prep phase. All the other sections in the spec file are +treated the same way as the above package. + +----------------------------- +OpenStack and Python packages +----------------------------- + +These packages are similar to the above as far as requirements, dependencies, +etc. The build phase is a bit different since there is no compilation step +involved. + +:: + + Source0: %{name}-%{version}.tar.gz + + %prep + %setup -q + + %build + %{__python2} setup.py build + + %install + %{__python2} setup.py install -O1 --skip-build --root %{buildroot} + + mkdir -p %{buildroot}%{_sysconfdir}/bash_completion.d + install -pm 644 tools/nova.bash_completion \ + %{buildroot}%{_sysconfdir}/bash_completion.d/nova + +In this case, the ``setup`` macro unpacks the Source0 tarball in the prep +phase. Next the ``setup.py`` that is part of the Python package handles +the actual process of building and installing the Python code. During the +install phase, a separate file is installed which isn't handled by the Python +installer. + +.. _centos_pkg: + +------------------------------------ +CentOS package with appended patches +------------------------------------ + +This example uses the ``parted`` package from CentOS and appends a few +patches of our own. + +#. Assuming this package has never been modified by StarlingX previously, find + a home for the package. This example uses + ``$MY_REPO/stx/integ/filesystem/parted``. + + :: + + MY_GIT_ROOT=$MY_REPO/stx/integ + MY_PKG_ROOT=$MY_GIT_ROOT/filesystem/parted + mkdir -p $MY_PKG_ROOT + cd $MY_PKG_ROOT + + # You must also add the new package directory to your local centos_pkg_dirs file if not already present + echo filesystem/parted >> $MY_GIT_ROOT/centos_pkg_dirs + +#. Create a directory skeleton. The ``patches`` directory contains patches + applied to the tarball inside the ``src.rpm``. The ``meta_patches`` + directory contains patches applied to the rpmbuild directory of the + ``.src.rpm`` and are used to modify the ``SPECS/*.spec`` file, or + the ``SOURCES/*`` files. + + :: + + mkdir -p centos/meta_patches + mkdir -p centos/patches + touch centos/meta_patches/PATCH_ORDER + echo TIS_PATCH_VER=1 > centos/build_srpm.data + +#. Tell ``build-srpms`` where to find the ``src.rpm`` in the ``srpm_path`` + file. Two locations are supported for source RPMS: the + ``$MY_REPO/cgcs-centos-repo`` directory is for source RPMS provided by + CentOS. Tell the build system to search for a source RPM using + the ``mirror:`` prefix when you specify the path to the source RPM. + + :: + + find -L $MY_REPO/cgcs-centos-repo/ -name 'parted*.src.rpm' + /localdisk//designer/test/cgcs-root/cgcs-centos-repo/Source/parted-3.1-23.el7.src.rpm + + echo "mirror:Source/parted-3.1-23.el7.src.rpm" > centos/srpm_path + + The other location supported by ``build-srpms`` is the downloads subgit of + your layers (found in ``$MY_REPO/stx/downloads`` for CGCS, for example). + This should be used for packages that CentOS does not provide. In this case, + place the source RPM you downloaded in the directory, and use the ``repo:`` + prefix when creating your ``srpm_path`` file. The base path for ``repo:`` + is ``$MY_REPO``, so specify paths from that point. + + :: + + cp ~/Downloads/somepackage-1.2-3.src.rpm $MY_REPO/stx/downloads + echo "repo:stx/downloads/somepackage-1.2-3.src.rpm" > centos/srpm_path + +#. (Optional) Set up a git environment to create your patch. The SRPM will be + extracted, then the source tarball will be extracted and patched. + + :: + + # clean any prior edit environment that might exist + # + build-srpms --edit --clean parted + + # + # Create a working edit environment + # + build-srpms --edit parted + ... + ===== 'parted' has been extracted to for editing. ===== + ===== Metadata can be found at: /localdisk/loadbuild//test/std/srpm_work/parted/rpmbuild + ===== Source code can be found at: /localdisk/loadbuild//test/std/srpm_work/parted/gits/parted.spec/parted-3.1 + + # + # Create you patch + # + cd $MY_WORKSPACE/std/srpm_work/parted/gits/parted.spec/parted-3.1 + + edit + + # create a patch + git add + git commit -m 'foo bar' + git format-patch -n HEAD^ + 0001-foo-bar.patch + + # Copy it to the repo... probabbly best to drop the missleading 0001 prefix + mv 0001-foo-bar.patch $MY_PKG_ROOT/centos/patches/foo-bar.patch + + # undo the commit, if patch failed testing and needs more work + git reset --soft HEAD~ + + # + # Now create a meta_patch to add your patch to the spec file + # + cd $MY_WORKSPACE/std/srpm_work/parted/rpmbuild + + # modify Release to include tis patch version, add our patch to the end of the patch list + vi SPECS/parted.spec + .... + Release: 23.el7%{?_tis_dist}.%{tis_patch_ver} + .... + Patch34: 0034-tests-Use-wait_for_dev_to_-functions.patch + + # WRS + Patch35: syscalls.patch + Patch36: foo-bar.patch + ... + # if the %prep/%setup section does not use 'git am %{patches}' which automatically applies all patches, + # then it should contain a list of %patch macros (one per patch) and you will have to append your patch to the end of the list. + # %patch macros are basically just invocations of the 'patch' command to apply your patches. e.g. like this + %patch35 -p1 + %patch36 -p1 + + + git add SPECS/parted.spec + + # Optionally edit and add files in SOURCES/ + git commit -m 'meta foo bar' + git format-patch -n HEAD^ + 0001-meta-foo-bar.patch + + # Copy it to the repo... probabbly best to drop the missleading 0001 prefix + mv 0001-meta-foo-bar.patch $MY_PKG_ROOT/centos/meta_patches/meta-foo-bar.patch + + # undo the commit, if patch failed testing and needs more work + git reset --soft HEAD~ + + # + # Finally add the meta patch to the patch order file... so that the patches vs the spec file apply in the correct order + # + echo meta-foo-bar.patch >> $MY_PKG_ROOT/centos/meta_patches/PATCH_ORDER + +#. (Optional) Create a ``build_srpm.data`` file and add a ``COPY_LIST`` which + identifies additional files to copy into SOURCES before finalizing the srpm. + + :: + + vi $MY_PKG_ROOT/centos/build_srpm.data + COPY_LIST="files/resizepart.sh" + + + .. note:: + + The order is SRPM extracted first, then copy ``centos/patches/*`` + into SOURCES, then apply meta patches, then copy anything named in + ``COPY_LIST`` into SOURCES. ``COPY_LIST`` items are not available + for meta patching. Just edit them directly. + +#. Build the final ``src.rpm`` with your new changes inserted. + + :: + + build-srpms --clean parted + build-srpms parted + + +#. Build the rpm. + + :: + + build-rpms --clean parted + build-rpms parted + Verify contents of rpm: + build-rpms --no-descendants parted + cd $MY_WORKSPACE/rpmbuild + mkdir TMP; cd TMP + rpm2cpio parted-3.1-23.el7.tis.1.x86_64.rpm | cpio -ivmd + +#. Test. If it fails, go back to step 4. + +#. Commit. + + :: + + cd $MY_PKG_ROOT + git add meta_patches/PATCH_ORDER meta_patches/meta-foo-bar.patch patches/foo-bar.patch + # add these as required + git add centos/srpm_path + git add centos/build_srpm.data + git commit -F + ... rebase/push... + +#. Discard the working directory and its temporary gits created in step 4. + + :: + + build-srpms --clean --edit parted + +------------------------------------------------- +Up-version a CentOS package with appended patches +------------------------------------------------- + +Suppose you have a patched srpm ``resource-agents-3.9.5-54.el7_2.8.src.rpm`` and +you want to migrate to a newer srpm, for example, +``resource-agents-3.9.5-54.el7_2.8.src.rpm``. + +First question: How do I know if there is a newer version available? If the +local CentOS mirror has been updated, you can scan it for new srpms that have +local WRS patches using: + +:: + + find_patched_srpms_needing_upgrade + /localdisk/designer//test/cgcs-root/stx/recipes-cgl/cluster-resource-agents/centos/srpm_path: resource-agents-3.9.5-54.el7_2.8.src.rpm ==> resource-agents-3.9.5-54.el7_2.9.src.rpm + +This tells you that there are patches for +``resource-agents-3.9.5-54.el7_2.8.src.rpm``, but a newer version is available +(``resource-agents-3.9.5-54.el7_2.9.src.rpm``). You should probably migrate the +patches to the new ``resource-agents-3.9.5-54.el7_2.9.src.rpm``. No output +means no migrations are required at this time. + +#. Check out the old version with our patches. + + :: + + build-srpms resource-agents --edit --clean + build-srpms resource-agents --edit + +#. Point the ``srpm_path`` at the new source rpm. + + + :: + + vi $MY_REPO/cgcs-root/stx/recipes-cgl/cluster-resource-agents/centos/srpm_path + mirror:CentOS/vault.centos.org/7.2.1511/updates/Source/SPackages/resource-agents-3.9.5-54.el7_2.9.src.rpm + +#. Check out the new version without our patches. + + :: + + build-srpms resource-agents --edit --no-meta-patch + ... + ===== 'resource-agents' has been extracted for editing. ===== + ===== Metedata can be found at: /localdisk/loadbuild//test/std/srpm_work/resource-agents/rpmbuild + ===== Source code can be found at: /localdisk/loadbuild//teststd//srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96 + +#. You now have two directories, each a git tree with two branches, one based + on the old srpm, one on the new srpm. + + :: + + cd $MY_WORKSPACE/std/srpm_work/resource-agents/rpmbuild + + git branch --all + master + resource-agents-3.9.5-54.el7_2.8 + * resource-agents-3.9.5-54.el7_2.9 + + cd $MY_WORKSPACE/std/srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96 + git branch --all + master + resource-agents-3.9.5-54.el7_2.8 + * resource-agents-3.9.5-54.el7_2.9 + +#. Show the commit histories of the two branches to identify the work to do. + + :: + + cd $MY_WORKSPACE/std/srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96 + + git log resource-agents-3.9.5-54.el7_2.8 --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit + * be18338 - (HEAD, resource-agents-3.9.5-54.el7_2.8) WRS: Patch99: exportfs_accept_ipv6.patch (2016-04-29 01:11:03 -0400) + * 70308b7 - WRS: Patch98: pgsql.patch (2016-04-29 01:11:02 -0400) + * 802a93a - WRS: Patch97: lvm_vg_activation.patch (2016-04-29 01:11:00 -0400) + * 96acac3 - WRS: Patch96: umount-in-namespace.patch (2016-04-29 01:11:00 -0400) + * 24b1bb1 - WRS: Patch95: copyright.patch (2016-04-29 01:10:59 -0400) + * cb5c3a9 - WRS: Patch94: ipaddr2_check_if_state.patch (2016-04-29 01:10:58 -0400) + * 600cda6 - WRS: Patch93: new_ocf_return_codes.patch (2016-04-29 01:10:57 -0400) + * 5f40f0b - WRS: Patch92: filesystem_rmon.patch (2016-04-29 01:10:56 -0400) + * facf036 - (tag: pre_wrs_resource-agents-3.9.5-54.el7_2.8) Patch91: bz1316633-backup-and-restore-rabbitmq-users-during-resource-re.patch (2016-04-29 01:10:32 -0400) + * 48cbdac - Patch90: bz1311180-rabbitmq-cluster-forget-stopped-cluster-nodes.patch (2016-04-29 01:10:31 -0400) + + + git log resource-agents-3.9.5-54.el7_2.9 --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit + * 746eb3d - (HEAD, tag: pre_wrs_resource-agents-3.9.5-54.el7_2.9, resource-agents-3.9.5-54.el7_2.9) Patch94: bz1318744-galera-no-grastate.patch (2016-04-29 01:12:16 -0400) + * f6fc3f0 - Patch93: bz1318744-galera-heuristic-recovered.patch (2016-04-29 01:12:15 -0400) + * dc76b27 - Patch92: bz1318744-galera-crash-recovery.patch (2016-04-29 01:12:14 -0400) + * facf036 - (tag: pre_wrs_resource-agents-3.9.5-54.el7_2.8) Patch91: bz1316633-backup-and-restore-rabbitmq-users-during-resource-re.patch (2016-04-29 01:10:32 -0400) + * 48cbdac - Patch90: bz1311180-rabbitmq-cluster-forget-stopped-cluster-nodes.patch (2016-04-29 01:10:31 -0400) + + The image below shows the color-coded results of the comparison. + + * The patches in green with ``WRS:`` in the patch name must be + ported. + + * The commit in orange noted with ``tag:`` was the last common commit. + + * The patches in blue are new patches from upstream that may interfere with + our patches. + + +.. figure:: figures/pkging_patch_comp.png + :scale: 55% + :alt: Patch comparison + + *Figure 1: Patch comparison* + + +-------------------------------- +Package providing an init script +-------------------------------- + +The packages/recipes that provide an init script (found under ``/etc/init.d/`` +and corresponding ``/etc/rc.d/``) must be converted to systemd services. +Within STX there are two kinds of services: + +* Services that must be started on boot. + +* Services that are monitored by pmond (and automatically restarted in case of + failure). In this case, we don't want systemd to restart a process in case + of failure. + +The general idea about converting those services is to still keep the script +file installed under ``/etc/init.d/`` in the short term and wrap it in a +systemd service. This is needed so that service dependencies can be evaluated by +systemd and start them in the right sequence on boot. Re-using the same init +script at the moment is more convenient for supporting the same code base for +wrlinux and CentOS packaging (no need to support multiple version of the same +script). + +Given an example init script ``/etc/init.d/helloworld``, create a corresponding +``helloworld.service`` file: + +:: + + [Unit] + Description=Helloworld service + + # Configures requirement dependencies on other services. If this service gets activated, the + # services listed here will be activated as well. If one of the other services gets deactivated or + # its activation fails, this service will be deactivated. + Requires= + + # A weaker version of Requires. Services listed in this option will be started if the configuring + # unit is. However, if the listed services fail to start, this has no impact on the current service. + Wants= + + # The next two directives are important for specifying the services dependancies. This will indicate + # to systemd in which order services should be started. This does not imply a dependency + # relationship ('Requires' or 'Wants') and must be used in conjunction with one of the above + # directives if this is desired. + + # The services listed in this directive will be started before starting the current service. + After=syslog.target network.target + + # The services listed in this directive will not be started until the current service is marked as + # started if they are activated at the same time. + Before= + + [Service] + # Usual types are 'oneshot', 'simple' and 'forking'. + # oneshot: The process is not a daemon running in the background. Systemd will wait that the process + # exit before continuing. + # simple: Systemd will fork the process. + # forking: The process will call fork(). It is recommended to use 'PIDfile' in this case so that + # systemd can identify the process. + Type=oneshot + + # Commands with their arguments that are executed when this service is started + ExecStart=/etc/init.d/helloworld start + + # Additional commands that are executed before command in ExecStart + ExecStartPre= + + # Additional commands that are executed after command in ExecStart + ExecStartPost= + + # Commands to execute to stop the service + ExecStop=/etc/init.d/helloworld stop + + # Additional commands that are executed after the service is stopped + ExecStopPost= + + # Commands to execute to trigger a configuration reload in the service + ExecReload= + + # Only useful when 'Type' is 'forking'. PID file is important if the process is monitored by pmond. + PIDFile=/var/run/helloworld.pid + + # Configures whether the service shall be restarted when the service process exits. + # Should be 'always' is systemd should monitor the process. + # Should be 'no' (default) if process is monitored by pmond. + Restart= + + [Install] + # This enables to service to be started on boot. It is equivalent of init runlevel (see below for equivalence). This + # means that the current service will be started when the listed unit is started. All the other services that are + # also dependent on this target will also get started, the order will depends on the 'After' and 'Before' options + # specified above. + WantedBy=multi-user.target + +Once the systemd service file is defined, it needs to be packaged in the rpm. +The ``helloworld.service`` file must be installed in the ``/etc/systemd/system/`` +directory (recommended for STX services) by the rpm spec. It's important to set +file permissions to 664. The systemd services must be enabled on install +and disabled on uninstalled. This is achieved by using the systemd command +systemctl enable (and disable). However, it is recommended to use the macros +already defined by rpm. For example: + +:: + + %install + install -d -m 755 %{buildroot}/etc/systemd/system/ + install -p -D -m 664 helloworld.service %{buildroot}/etc/systemd/system/helloworld.service + + # Enable the service and start it. + %post + %systemd_post helloworld.service + + # Disable the service and stop it. + %preun + %systemd_preun helloworld.service + + # Try to restart the service. Usefull on upgrades. + %postun + %systemd_postun_with_restart helloworld.service + +Systemd searches these directories for services (in order of precedence, the +first item has higher precedence than the one following): + +* ``/etc/systemd/system/`` Recommended and reserved location for system + administrator customized services +* ``/run/systemd/system/`` Mainly used by CentOS. Don't pollute this + directory. +* ``/usr/lib/systemd/system/`` + +CentOS is distributed with a number of predefined targets that are more or less +similar to the standard set of runlevels. Use the following comparison table for +specifying your WantedBy target. + +* runlevel0.target -> poweroff.target +* runlevel1.target -> rescue.target +* runlevel2.target -> multi-user.target +* runlevel3.target -> multi-user.target +* runlevel4.target -> multi-user.target +* runlevel5.target -> graphical.target +* runlevel6.target -> reboot.target + +Convert to systemd service (without intermediate init script): + +* The exact PID file name that is used by pmond can be found by looking on an + installed system under ``/etc/pmon.d/``. Look for ``pidfile=`` in the + corresponding pmond configuration file for the service. + +* Some of the init scripts are using start-stop-daemon to manage processes. + The start-stop-package has been ported to CentOS (it is not supported by + CentOS). Make sure the init script is providing the right ``--pidfile`` + argument that matches the pmond service configuration. A process that is + started by start-stop-daemon with the ``--daemon`` or ``-d`` options + indicates that the processes will be forked, in which case the systemd + service Type should be forking. Else if the script or process doesn't fork, + the Type should be simple which will tell systemd to fork it. + +* Hints for defining service startup order (Before and After): + + * Systemd determines when a service should start using the ``After=`` + and ``Before=`` values found in the ``[Unit]`` section of the ``service`` + file. These values take a space-separated list of other services or + *targets*, where a target is effectively a collection of services. For + example, the log management service has the following values: + + :: + + [Unit] + Description=StarlingX Log Management + After=network.target syslog-ng.service iscsid.service sw-patch.service + Before=config.service pmon.service + + In this example, the log management service can't start until software + is patched, and a network, a disk and the syslog-ng service are all up + and running. + + * The init script might contain some specific information about the order. + Look for Required-Start and Required-Stop in it. + +Troubleshooting (at runtime): + +* Enable a service: ``systemctl enable`` +* Disable a service: ``systemctl disable`` +* Start a service: ``systemctl start`` +* Stop a service: ``systemctl stop`` +* Get service status: ``systemctl status`` +* Get list of running units ordered by time to init: ``systemd-analyze blame`` +* Show systemd logging: ``journalctl /usr/lib/systemd/systemd`` +* Show logging of a specific service: ``journalctl _SYSTEMD_UNIT=`` + +-------------------------------- +After you've converted a package +-------------------------------- + +Each repo has a ``centos_iso_image.inc`` file, which contains the list of +packages to be included in the installation iso relevant to the repo. The +``build-iso`` step merges the individual ``centos_iso_image.inc`` files to +generate the complete list. +