.. _virtual-create-and-bootstrap-from-a-private-docker-registry:

==================================================================
Create and Bootstrap from a Local External Private Docker Registry
==================================================================

You can bootstrap controller-0 from a private local external Docker registry.
This is useful in case you plan to perform multiple installations on your own
machine and do not want to be rate-limited by public registries.

This will also speed up the bootstrap process as images will be downloaded only
once. This guide assumes that you are installing this local external docker
registry on a Linux system with Docker installed and configured.

Optionally, you can also create multiple registries with pull-through cache,
skipping the need to pre-populate the registry and keeping the images and tags
up-to-date.

.. rubric:: |proc|

.. tabs::
  .. group-tab:: Private Registry

    .. note::
      In this section, you will pre-populate a local registry with multiple
      images. Currently, the generated images list will not contain all
      images, making it necessary to manually pull some images used by the
      kubeadm tool. Alternatively, you can follow the "pull-through cache"
      section, allowing docker to get the image from remote, if it doesn't
      exists locally.

    #.  Create folders to store your local registry images (``storage``) and to
        place setup files that will be used later on (``images``):

        .. code-block:: shell

            mkdir -p $HOME/docker-registry/storage
            mkdir -p $HOME/docker-registry/images

    #.  Create a configuration file that will be used by Docker's official
        Registry image later on:

        .. code-block:: shell

            cat > $HOME/docker-registry/config.yml << EOF
            version: 0.1
            log:
              fields:
                service: registry
            storage:
              cache:
                blobdescriptor: inmemory
              filesystem:
                rootdirectory: /var/lib/registry
            http:
              addr: :5000
              headers:
                X-Content-Type-Options: [nosniff]
            health:
              storagedriver:
                enabled: true
                interval: 10s
                threshold: 3
            EOF

    #.  Run the docker container registry:

        .. code-block:: shell

            export LOCAL_REG=$HOME/docker-registry
            docker run -d \
                --restart=always \
                --name registry \
                -v "$LOCAL_REG"/storage:/var/lib/registry \
                -v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \
                -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
                -p 5000:5000 \
                registry:2

        .. note::
           *Optional*: the ``-p`` parameter configures a mapping between the
           host port and the container port. If you want to listen on another
           port on your host, say 9000, change from ``-p 5000:5000 \`` to
           ``-p 9000:5000 \``.

    #.  Create the list of images that will populate the registry.

        Obtain the Kubernetes version your |prod| uses. This can be found in
        the ``fresh_install_k8s_version`` value of the `Kubernetes versions`_
        file. Use the branches and tags to find the value for your version.

        With the Kubernetes version, you can find the corresponding folder in
        `system images`_ and locate the ``system-images.yml`` file. This file
        contains the list of images to be loaded into your registry.

        To make the list of images for |prod| 8.0, take the raw address of the
        corresponding ``system-images.yml`` file and set a variable with it:

        .. code-block:: shell

            export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml"

        Use the command to generate a ``list.lst`` file:

        .. code-block:: shell

            curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst

        The expected image list will be presented in this format:

        .. code-block:: text

            <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
            <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
            <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>...

        .. important::
           Due to ``kubeadm`` dynamically pulling the necessary images for
           creating the k8s cluster, based on the ``Kubernetes version`` used,
           this list doesn't contain all necessary images.
           You'll need to manually add the extra images to the ``list.lst``
           file, using `kubeadm`_. You can check the official k8s docs on how
           to get them.

        .. note::
           *Optional*: If you have a running |prod| setup, you can run the
           following to create an Ansible Playbook to get the exact images you
           will need instead:

           .. code-block:: shell

              cat > list-images.yml << EOF
              ---
              - hosts: localhost
                gather_facts: true
                tasks:
                  - name: Load image info
                    include_role:
                      name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information

                  - name: Print image list
                    debug:
                      msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}"
              EOF

           Then, run the following to execute the Ansible Playbook:

           .. code-block:: shell

              K8S_VERSION=<version>
              ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}"

           You will find the Kubernetes version to assign to the `K8S_VERSION`
           variable on the aforementioned `Kubernetes versions`_ file.

    #.  Create and run a script that will populate the registry based on the
        list of images:

        .. code-block:: shell

            export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh
            cat > $REG_SCRIPT <<'EOF'
            #!/bin/bash

            if [[ -z $1 ]]; then
                    echo "Please provide a file with a list of Docker images."
                exit 1
            fi

            TAGS_FILE=$1
            LOCAL_REGISTRY=localhost:5000

            while read DOCKER_IMAGE;
            do
                echo ""
                echo -n "--- ${DOCKER_IMAGE}: ";

                IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " "))
                REPO=${IMAGE_ARRAY[0]}
                TAG=${IMAGE_ARRAY[1]}
                REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list"
                if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then
                    echo -n "Skipping..."
                    continue
                fi

                echo "Pulling..."

                set -x
                docker pull ${DOCKER_IMAGE};
                REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE}
                docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
                docker push ${REGISTRY_IMAGE};
                docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
                set +x

            done < $TAGS_FILE
            EOF
            chmod +x $REG_SCRIPT
            $REG_SCRIPT $HOME/docker-registry/images/list.lst

        .. note::
            The ``populate_registry.sh`` script checks if each image in the
            list is already present, which means you can update the list and
            re-run the script to get new images whenever necessary.

  .. group-tab:: With Pull-through cache

      #.  Create folders to store your local registry images (``storage``) and to place setup files that will be used later on (``images``):

          .. code-block:: shell

              mkdir -p $HOME/docker-registry/storage
              mkdir -p $HOME/docker-registry/images
              mkdir -p $HOME/docker-registry/config

      #.  Create a configuration file that will be used by Docker's official
          registry image later on. The script will replace the PORT and
          REGISTRY values:

          .. code-block:: shell

              cat > $HOME/docker-registry/config.yml << EOF
              version: 0.1
              log:
                fields:
                  service: registry
              storage:
                cache:
                  blobdescriptor: inmemory
                filesystem:
                  rootdirectory: /var/lib/registry
              http:
                addr: <PORT>
                headers:
                  X-Content-Type-Options: [nosniff]
              health:
                storagedriver:
                  enabled: true
                  interval: 10s
                  threshold: 3
              proxy:
                remoteurl: https://<REGISTRY>
              EOF

      #.  Create and run a script that will create several private docker
          registries:

          .. code-block:: shell

              export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh
              cat > $REG_CACHE_SCRIPT <<'EOF'
              #!/bin/bash

              if [ -z "$1" ]; then
                  echo "No argument supplied"
                  exit 1
              fi

              export LOCAL_REG=$HOME/docker-registry
              INSECURE_REGISTRY_LIST=()
              LOCALHOST_IP=$1
              [ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2
              REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io"
                  "docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io")
              CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml)

              for cur_registry in "${REGISTRY_LIST[@]}"; do
                  CONTAINER_NAME="registry_$cur_registry"
                  NEW_CONFIG_FILE_NAME="config_$cur_registry.yml"
                  NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME"
                  NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s/<PORT>/$BASE_PORT/" | sed "s/<REGISTRY>/$cur_registry/")
                  echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE"

                  docker run -d \
                      --restart=always \
                      --name "$CONTAINER_NAME" \
                      -v "$LOCAL_REG"/storage:/var/lib/registry \
                      -v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \
                      -e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \
                      -p "$BASE_PORT":"$BASE_PORT" \
                      registry:2.6.2

                  INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT")
                  ((BASE_PORT++))

              done

              for reg in "${INSECURE_REGISTRY_LIST[@]}"; do
                  echo "$reg"
              done
              EOF

              chmod +x $REG_CACHE_SCRIPT
              $REG_CACHE_SCRIPT <your ip address> <starting port (default 5000)>

          .. important::
              The ``create_registries.sh`` script create 8 registry
              containers. You'll need 8 free ports, in sequence, starting from
              the specified starting port.
              Ex.: Passing port ``6000`` as the second argument, will create containers on ``6000-6007`` port range.

          .. note::
              On it's current version, Docker CLI throws a warning when
              pulling images that are still using the v1 manifest format.
              The official Docker image registry, on it's latest version, does
              not allow images that still uses them.
              A few images used by the StarlingX OS are outdated, so, as of
              now, it'll be necessary to use a older version of the registry
              image (v2.6.2).
              Check the official `registry as a pull through cache`_ docker
              guide for more info.


.. note::
    The Docker CLI exclusively permits insecure (HTTP) registries when on
    the local host. When executing the provided script remotely, in
    addition to modifying the ``LOCAL_REGISTRY`` variable to match the IP
    address of the registry's location, it is necessary to insert an entry
    in the ``insecure-registries:`` section within the
    ``etc/docker/daemon.json`` file. Following this adjustment, you must
    restart the Docker service.

.. rubric:: |result|

Your registry is ready! On your next |prod| installation, update your
``/home/sysadmin/localhost.yml`` bootstrap overrides file with the
following lines to use it:

.. tabs::
  .. group-tab:: Private Registry

    .. code-block:: yaml

        docker_registries:
          quay.io:
            url: <your IP address>:5000/quay.io
          gcr.io:
            url: <your IP address>:5000/gcr.io
          k8s.gcr.io:
            url: <your IP address>:5000/k8s.gcr.io
          docker.io:
            url: <your IP address>:5000/docker.io
          docker.elastic.co:
            url: <your IP address>:5000/docker.elastic.co
          ghcr.io:
            url: <your IP address>:5000/ghcr.io
          registry.k8s.io:
            url: <your IP address>:5000/registry.k8s.io
          icr.io:
            url: <your IP address>:5000/icr.io
          defaults:
            type: docker
            secure: false

  .. group-tab:: With Pull-through cache

    .. code-block:: yaml

        docker_registries:
          quay.io:
            url: <your IP address>:<port_1>
          gcr.io:
            url: <your IP address>:<port_2>
          k8s.gcr.io:
            url: <your IP address>:<port_3>
          docker.io:
            url: <your IP address>:<port_4>
          docker.elastic.co:
            url: <your IP address>:<port_5>
          ghcr.io:
            url: <your IP address>:<port_6>
          registry.k8s.io:
            url: <your IP address>:<port_7>
          icr.io:
            url: <your IP address>:<port_8>
          defaults:
            type: docker
            secure: false

.. note::
    This procedure configured |prod| to use an insecure registry via the
    ``docker_registries.defaults.secure`` parameter set to ``false`` in the
    excerpt above. Make sure you only use this on your own development
    environment.

.. _Kubernetes versions: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/bootstrap/validate-config/vars/main.yml
.. _system images: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars
.. _registry as a pull through cache: https://docs.docker.com/docker-hub/mirror/
.. _kubeadm: https://kubernetes.io/pt-br/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-list