docs/doc/source/developer_resources/build_guide.rst

=====================
StarlingX Build Guide
=====================

This section describes the steps for building an ISO image from a StarlingX
R3.0 and earlier release.

.. contents::
   :local:
   :depth: 1

.. _requirements:

------------
Requirements
------------

*********************
Hardware requirements
*********************

A workstation computer with:

* Processor: x86_64 is the only supported architecture
* Memory: At least 32GB RAM
* Hard Disk: 500GB HDD
* Network: Network adapter with active Internet connection

*********************
Software requirements
*********************

A workstation computer with:

* Operating System: Ubuntu 16.04 LTS 64-bit
* Docker
* Android Repo Tool
* Proxy settings configured, if required (See
  https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/message/FERKPQEAG5YOSV5T5TKEN7HPDCNXBFAL/
  for more details)
* Public SSH key

.. _starlingx_build_tools:

---------------------
StarlingX Build Tools
---------------------

The Debian build is completed using a set of containers designed to run in a Kubernetes environment.

To facilitate this we are currently making use of Minikube and Helm, later on we will provide versions
of the Helm Charts to allow for running builds directly on Kubernetes or StarlingX.

There are five containers required to complete a build:

* stx-builder: main developer build container
* stx-pkgbuilder: Debian package builder (uses sbuild)
* stx-repomgr: Debian local repository archive (uses aptly)
* stx-lat-tool: ISO image builder
* stx-docker: Docker in Docker (build Docker images)

At a high level the StarlingX ISO image creation flow involves the following general steps (assuming you have
already configured Docker on your system):

#. Install Minikube and Helm.
#. Build or download the StarlingX k8s development environment.
#. Enter the stx-builder pod/container to trigger the building task.
#. Build packages/ISO creation.

.. note::

  The build system requires a Linux system with Docker and python 3.x installed. Building on Windows
  is not supported -- please use a Virtual Machine if necessary. The steps on this page have been tested
  on CentOS 7 and Ubuntu Focal.

.. _register_on_docker_hub:

----------------------
Register on Docker Hub
----------------------

The build environment relies on the Docker Hub registry for storing container images used during the build.
Docker Hub puts `limits on the amount of data that can be downloaded <https://docs.docker.com/docker-hub/download-rate-limit>`__
by the same user/IP address. To avoid this limit, we recommend to register an account on Docker Hub and log in to
docker prior to initializing your StarlingX development environment. Note the user ID and password as we will supply
them to the ``stx-init-env`` script below.


.. _configure_build_environment:

---------------------------
Configure build environment
---------------------------

We need to create and start the build containers, which requires some additional configuration described below.

**************
Install Docker
**************

Follow these instructions: https://docs.docker.com/engine/install/ubuntu/

.. warning::

  Do not install or use Docker with `Snap packages <https://en.wikipedia.org/wiki/Snap_(software)>`__. Your machine
  may come with a docker snap package pre-installed (Ubuntu?), please remove the snap package and install a native
  package instead, using the above link. Reason: minikube (see below) doesn't work correctly within docker that
  runs in a snap sandbox/container.

*************************
Install Minikube and Helm
*************************

Install Minikube to support the local k8s framework for building. Install Helm tools to manage the Helm Charts
required to start/stop/upgrade the pods or the deployments for the StarlingX Building system. Before installing
these components please make sure that Docker is available in your environment.

Install minikube (https://minikube.sigs.k8s.io/docs/start/):

::

  curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
  sudo install minikube-linux-amd64 /usr/local/bin/minikube

.. note::

  As of this writing minikube v 1.22.0 is current.

.. note::

  Minikube requires at least 2 CPU cores.

Alternatively, we can also use a third-party Minikube binary:

::

  curl -LO http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.20.0/minikube-linux-amd64
  sudo install minikube-linux-amd64 /usr/local/bin/minikube

Install Helm -- you can select the version listed here or the latest released version:

::

  curl -LO https://get.helm.sh/helm-v3.6.2-linux-amd64.tar.gz
  tar xvf helm-v3.6.2-linux-amd64.tar.gz
  sudo mv linux-amd64/helm /usr/local/bin/

Add your user account to docker group:

::

  sudo usermod -aG docker $(id -un) && newgrp docker

************
Install repo
************

::

  https://gerrit.googlesource.com/git-repo/

****************************
Export environment variables
****************************

These variables should be set every time you start the env. A good practice is to
save them in a text file and then source the file. For more details about the STX environment
variables `click here <https://opendev.org/starlingx/tools/src/branch/master/import-stx.README>`__.


::

  export PROJECT="stx-env"
  export USER_NAME="Firstname Lastname"
  export USER_EMAIL="your@email.org"

  # STX_BUILD_HOME should be set to a directory owned by your userid
  # the build home needs to have at least 200Gb of free space to build all packages and iso
  export STX_BUILD_HOME="/home/${USER}/${PROJECT}"
  export REPO_ROOT="${STX_BUILD_HOME}"/repo
  export REPO_ROOT_SUBDIR="localdisk/designer/${USER}/${PROJECT}"

  # MINIKUBE
  export STX_PLATFORM="minikube"
  export MINIKUBENAME="minikube-${USER}"

  #############################################
  # Manifest/Repo Options:
  #############################################
  # STX MASTER
  export MANIFEST_URL="https://opendev.org/starlingx/manifest.git"
  export MANIFEST_BRANCH="master"
  export MANIFEST="default.xml"

******************
Create directories
******************

Create the ``$STX_BUILD_HOME`` directory, you may need sudo privileges if using /build

::

  e.g:
  sudo mkdir -p /build/${USER}
  sudo chown ${USER}: /build/${USER}

::

  mkdir -p $STX_BUILD_HOME
  cd $STX_BUILD_HOME

***************
Initialize repo
***************

::

  # create REPO_ROOT_SUBDIR and symlink
  # symlink is a helper as minikube mounts the stx_build_home as its workspace
  # so it works as a shortcut to access the repos
  mkdir -p $REPO_ROOT_SUBDIR
  ln -s $REPO_ROOT_SUBDIR $REPO_ROOT

  cd $REPO_ROOT
  # download and sync the repos
  repo init -u ${MANIFEST_URL} -b ${MANIFEST_BRANCH} -m ${MANIFEST}
  repo sync

******************
Init and setup STX
******************

The build tools comes with a script, import-stx, which sets up your PATH and other
environment as necessary. This script must be sourced before attempting to use any tools.

There's a number of environment variables you can set prior to sourcing this file, please
feel free to review the script and import-stx.README for a full list.

.. warning::

  Minikube can't work if your $HOME directory points to an NFS location, we need to point
  it to some other local file system by defining $MINIKUBE_HOME in the environment before
  sourcing import-stx.

.. warning::

  If these variables are misconfigured, the env won't be able to run commands such as
  downloader and build-pkgs

The build expects a configuration file, ``stx.conf`` (`example <https://opendev.org/starlingx/tools/src/branch/master/stx.conf.sample>`__)
to exist at the root of the build tools working directory. It is a key/value file containing
various build options. The ``stx config`` command may be used to add/modify entries in the file.

::

  # Init stx tool
  cd stx-tools
  source import-stx

  # Update stx config
  # Align the builder container to use your user/UID
  stx config --add builder.myuname $(id -un)
  stx config --add builder.uid $(id -u)

  # Embedded in ~/localrc of the build container
  stx config --add project.gituser ${USER}
  stx config --add project.gitemail ${USER_EMAIL}

  # This will be included in the name of your build container and the basename for $MY_REPO_ROOT_DIR  
  stx config --add project.name ${PROJECT}

  stx config --show
  # Show usage information
  stx config --help

*****************************
Start/Create build containers
*****************************

The ``stx-init-env`` script will download or re-create build (docker) containers, and start them:

::

  cd repo/stx-tools
  # Type in DockerHub username & password if prompted
  ./stx-init-env --dockerhub-login
  # Monitor the status until they are running:
  stx control status
  # You should see 5 containers on Running state

Once docker images are available locally, you can start & stop them using the ``stx`` tool:

::

  stx control start         # start builder PODs if not running
  stx control status        # display POD status
  stx control stop          # stop PODs
  stx control is-started    # check if Helm release is installed


.. warning::

  Any changes to ``stx.conf`` or (``stx config add`` etc) requires that the PODs are re-started.
  If you want to make changes to the environment in the build container, use ``stx control stop``,
  then ``stx config`` to adjust the variables, and re-start the containers.

  ::

    stx control stop
    stx config add <...>
    stx control start

The script pulls build containers from DockerHub by default, where a new version is
built once per day (i.e default container images may be slightly out of date when you
pull them). You can force a local re-build as follows:

::

  stx control stop
  cd repo/stx-tools
  ./stx-init-env --dockerhub-login --rebuild

.. _entering_and_controlling_pods:

---------------------------
Entering & controlling Pods
---------------------------

Once the containers are running, one can enter them (think ``docker exec <...> /bin/bash``).
While there are 5 containers, most build tasks are driven from the "builder" container,
which is the default when using the ``stx`` tool:

::

  # enter the "builder" container
  stx shell

You can enter other containers as follows

::

  stx shell --container [builder|pkgbuilder|lat|repomgr|docker]

Use ``exit`` command to exit from the node to host environment.

You can use the ``stx control`` command to start/stop & monitor builder POD status:

::

  # control the Pods
  stx control start
  stx control stop
  stx control status

  # more info
  stx control --help

The ``status`` command will include Helm status, including deployments and the pods.
You can use that information to manually enter or troubleshoot PODs using munikube or kubectl.

*********************************
Every time you start/restart Pods
*********************************

Every time you start/restart Pods, execute these mandatory steps inside the builder:

::

  sudo apt-get update
  git config --global user.name "Firstname Lastname"
  git config --global user.email "Your email"

.. note::

  You may see the following errors from apt. You can ignore this and continue.

  ::

    E: Failed to fetch http://stx-stx-repomgr:80/deb-local-source/dists/bullseye/main/source/Sources 404 Not Found [IP: 10.102.135.193 80]
    E: Some index files failed to download. They have been ignored, or old ones used instead.

.. _build_packages_iso_creation:

---------------------------
Build packages/ISO creation
---------------------------

The builder is the container where you will perform most of the actions,
such as launching the task of building packages and images.


::

  stx shell

.. _refresh_the_source_tree:

-----------------------
Refresh the source tree
-----------------------

The StarlingX source tree consists of multiple git repositories. The tool ``repo``
is used to sync these repositories, if required you can sync the repos from inside the builder:

::

  cd $MY_REPO_ROOT_DIR
  repo sync

After the ``repo sync`` is done, check the below directory:

::

  ls $MY_REPO
  ls $MY_REPO/stx
  ls $MY_REPO_ROOT_DIR/stx-tools

Before running ``build-pkgs``:

Run below command to download the sources of all buildable packages by scanning
the repo root ``$MY_REPO/stx`` the download directory is: ``$STX_MIRROR``

::

  downloader -s -B std,rt

All the below lists with build types will be scanned in the repo root ``$MY_REPO/stx``:

* debian_pkg_dirs
* debian_pkg_dirs_rt
* debian_pkg_dirs_installer

**********************************
Download 3rd-party tar & deb files
**********************************

Run below command to download the debian binary packages (distribution: bullseye)
into directory ``$STX_MIRROR/binaries``:

::

  downloader -b -B std,rt

All the below lists of binary packages will be downloaded:

::

  $MY_REPO_ROOT_DIR/stx-tools/debian-mirror-tools/config/debian/common/base-bullseye.lst
  $MY_REPO_ROOT_DIR/stx-tools/debian-mirror-tools/config/debian/<layer>/os-std.lst
  $MY_REPO_ROOT_DIR/stx-tools/debian-mirror-tools/config/debian/<layer>/os-rt.lst

You can also run below command to download both sources and binaries:

::

  downloader -B std,rt

  # To check all options:
  downloader --help

Currently, the apt sources used to download packages are in the '/etc/apt/sources.list'
file in the builder container.

***************************************
Verify that the local repos are created
***************************************

::

  repo_manage.py list
  INFO:repo_manage:No remote repo
  INFO:repo_manage:3 local repos:
  INFO:repo_manage:deb-local-build : bullseye : main
  INFO:repo_manage:deb-local-binary : bullseye : main
  INFO:repo_manage:deb-local-source : bullseye : main

.. note::

  All 3 repos should be seen only after ``build-pkgs [-p <package>]`` is done at a later time.

.. _build_packages:

--------------
Build packages
--------------

The ``build-pkgs`` has two phases based on the Debian package's build:

1) Check the digest of package's source meta data, for example:

::

  if package 'dhcp' in cache:
    if sha256 digest of the folder (/path/to/stx/integ/base/dhcp/debian) have not changed:
      if the dsc file of package dhcp exists:
          reuse the existing dsc file
          return
      create dsc file for package 'dhcp' and add the checksum to cache 

2) Build avoidance is enabled by default for this phase, the build option '-c' will turn 'build avoidance' off.

::

  if build avoidance is enabled:
      check whether there is build stamp for this package:
          if Yes, skip the build and return
  Send the build request for the package to pkgbuilder container

To Build packages:


::

  # Build all packages
  # this should rebuild all packages (std and rt)
  build-pkgs

  # If you want to clean and build all:
  build-pkgs --clean 

  # If you just want to resume the previous build, please run without '-c':
  # build-pkgs

  # Build packages in parallel
  build-pkgs --parallel <number of parallel tasks, the default number is 10, the maximum number of parallel tasks is 30>

  # To define the interval to poll the packages build status during parallel build:
  --poll_interval <the default interval is 10 seconds>

  # To limit the number of make jobs for a package:
  --max_make_jobs <the default number of jobs is equal to the environment variable 'STX_BUILD_CPUS' or 'MAX_CPUS' inside the container>

  # Build single package
  build-pkgs -p <package name>

  # Build single package cleaning previous build
  build-pkgs -c -p <package name>

.. warning::

  Be careful, ``--clean`` not only cleans the whole build directory
  ``/localdisk/loadbuild/<user>/<project>/{std,rt}`` but also cleans the local
  repository "deb-local-build". This means all the starlingX packages will be
  built from scratch and this will take time. 

.. _build_iso:

---------
Build ISO
---------

Once you have built all of the packages you can build the iso by running the following command:

::

  build-image
  ls -al /localdisk/deploy/*.iso


.. _teardown_env:

------------
Teardown env
------------

Delete minikube profile:

::

  ./stx-init-env --delete-minikube-profile

Delete chroots + restart pods:

::

  ./stx-init-env --reset

Stop pods, delete local workspaces, chroots, aptly, docker & minikube profile:

::

  ./stx-init-env --reset-hard

.. _log_files:

---------
Log files
---------

While inside the build container, log files may be found here:

* Top-level build controller log files

  * ``/localdisk/builder.log``

  * ``/localdisk/pkgbuilder.log``

* Individual package build logs

  * ``${MY_WORKSPACE}/<std or rt>/<package name>/*.build``