docs/doc/source/developer_resources/local_registry.rst

Create and Bootstrap from a Local External Private Docker Registry

You can bootstrap controller-0 from a private local external Docker registry. This is useful in case you plan to perform multiple installations on your own machine and do not want to be rate-limited by public registries.

This will also speed up the bootstrap process as images will be downloaded only once. This guide assumes that you are installing this local external docker registry on a Linux system with Docker installed and configured.

Optionally, you can also create multiple registries with pull-through cache, skipping the need to pre-populate the registry and keeping the images and tags up-to-date.

.. group-tab:: Private Registry

Note

In this section, you will pre-populate a local registry with multiple images. Currently, the generated images list will not contain all images, making it necessary to manually pull some images used by the kubeadm tool. Alternatively, you can follow the "pull-through cache" section, allowing docker to get the image from remote, if it doesn't exists locally.

1. Create folders to store your local registry images (storage) and to place setup files that will be used later on (images):

   mkdir -p $HOME/docker-registry/storage
   mkdir -p $HOME/docker-registry/images

2. Create a configuration file that will be used by Docker's official Registry image later on:

   cat > $HOME/docker-registry/config.yml << EOF
   version: 0.1
   log:
     fields:
       service: registry
   storage:
     cache:
       blobdescriptor: inmemory
     filesystem:
       rootdirectory: /var/lib/registry
   http:
     addr: :5000
     headers:
       X-Content-Type-Options: [nosniff]
   health:
     storagedriver:
       enabled: true
       interval: 10s
       threshold: 3
   EOF

3. Run the docker container registry:

   export LOCAL_REG=$HOME/docker-registry
   docker run -d \
       --restart=always \
       --name registry \
       -v "$LOCAL_REG"/storage:/var/lib/registry \
       -v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \
       -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
       -p 5000:5000 \
       registry:2

   Note

   Optional: the -p parameter configures a mapping between the host port and the container port. If you want to listen on another port on your host, say 9000, change from -p 5000:5000 \ to -p 9000:5000 \.

4. Create the list of images that will populate the registry.

   Obtain the Kubernetes version your uses. This can be found in the fresh_install_k8s_version value of the Kubernetes versions file. Use the branches and tags to find the value for your version.

   With the Kubernetes version, you can find the corresponding folder in system images and locate the system-images.yml file. This file contains the list of images to be loaded into your registry.

   To make the list of images for 8.0, take the raw address of the corresponding system-images.yml file and set a variable with it:

   export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml"

   Use the command to generate a list.lst file:

   curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst

   The expected image list will be presented in this format:

   <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
   <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
   <REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>...

   Important

   Due to kubeadm dynamically pulling the necessary images for creating the k8s cluster, based on the Kubernetes version used, this list doesn't contain all necessary images. You'll need to manually add the extra images to the list.lst file, using kubeadm. You can check the official k8s docs on how to get them.

   Note

   Optional: If you have a running setup, you can run the following to create an Ansible Playbook to get the exact images you will need instead:

   cat > list-images.yml << EOF
   ---
   - hosts: localhost
     gather_facts: true
     tasks:
       - name: Load image info
         include_role:
           name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information
   
       - name: Print image list
         debug:
           msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}"
   EOF

   Then, run the following to execute the Ansible Playbook:

   K8S_VERSION=<version>
   ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}"

   You will find the Kubernetes version to assign to the K8S_VERSION variable on the aforementioned Kubernetes versions file.

5. Create and run a script that will populate the registry based on the list of images:

   export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh
   cat > $REG_SCRIPT <<'EOF'
   #!/bin/bash
   
   if [[ -z $1 ]]; then
           echo "Please provide a file with a list of Docker images."
       exit 1
   fi
   
   TAGS_FILE=$1
   LOCAL_REGISTRY=localhost:5000
   
   while read DOCKER_IMAGE;
   do
       echo ""
       echo -n "--- ${DOCKER_IMAGE}: ";
   
       IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " "))
       REPO=${IMAGE_ARRAY[0]}
       TAG=${IMAGE_ARRAY[1]}
       REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list"
       if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then
           echo -n "Skipping..."
           continue
       fi
   
       echo "Pulling..."
   
       set -x
       docker pull ${DOCKER_IMAGE};
       REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE}
       docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
       docker push ${REGISTRY_IMAGE};
       docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
       set +x
   
   done < $TAGS_FILE
   EOF
   chmod +x $REG_SCRIPT
   $REG_SCRIPT $HOME/docker-registry/images/list.lst

   Note

   The populate_registry.sh script checks if each image in the list is already present, which means you can update the list and re-run the script to get new images whenever necessary.

With Pull-through cache

1. Create folders to store your local registry images (storage) and to place setup files that will be used later on (images):

   mkdir -p $HOME/docker-registry/storage
   mkdir -p $HOME/docker-registry/images
   mkdir -p $HOME/docker-registry/config

2. Create a configuration file that will be used by Docker's official registry image later on. The script will replace the PORT and REGISTRY values:

   cat > $HOME/docker-registry/config.yml << EOF
   version: 0.1
   log:
     fields:
       service: registry
   storage:
     cache:
       blobdescriptor: inmemory
     filesystem:
       rootdirectory: /var/lib/registry
   http:
     addr: <PORT>
     headers:
       X-Content-Type-Options: [nosniff]
   health:
     storagedriver:
       enabled: true
       interval: 10s
       threshold: 3
   proxy:
     remoteurl: https://<REGISTRY>
   EOF

3. Create and run a script that will create several private docker registries:

   export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh
   cat > $REG_CACHE_SCRIPT <<'EOF'
   #!/bin/bash
   
   if [ -z "$1" ]; then
       echo "No argument supplied"
       exit 1
   fi
   
   export LOCAL_REG=$HOME/docker-registry
   INSECURE_REGISTRY_LIST=()
   LOCALHOST_IP=$1
   [ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2
   REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io"
       "docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io")
   CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml)
   
   for cur_registry in "${REGISTRY_LIST[@]}"; do
       CONTAINER_NAME="registry_$cur_registry"
       NEW_CONFIG_FILE_NAME="config_$cur_registry.yml"
       NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME"
       NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s/<PORT>/$BASE_PORT/" | sed "s/<REGISTRY>/$cur_registry/")
       echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE"
   
       docker run -d \
           --restart=always \
           --name "$CONTAINER_NAME" \
           -v "$LOCAL_REG"/storage:/var/lib/registry \
           -v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \
           -e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \
           -p "$BASE_PORT":"$BASE_PORT" \
           registry:2.6.2
   
       INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT")
       ((BASE_PORT++))
   
   done
   
   for reg in "${INSECURE_REGISTRY_LIST[@]}"; do
       echo "$reg"
   done
   EOF
   
   chmod +x $REG_CACHE_SCRIPT
   $REG_CACHE_SCRIPT <your ip address> <starting port (default 5000)>

   Important

   The create_registries.sh script create 8 registry containers. You'll need 8 free ports, in sequence, starting from the specified starting port. Ex.: Passing port 6000 as the second argument, will create containers on 6000-6007 port range.

   Note

   On it's current version, Docker CLI throws a warning when pulling images that are still using the v1 manifest format. The official Docker image registry, on it's latest version, does not allow images that still uses them. A few images used by the StarlingX OS are outdated, so, as of now, it'll be necessary to use a older version of the registry image (v2.6.2). Check the official registry as a pull through cache docker guide for more info.

Note

The Docker CLI exclusively permits insecure (HTTP) registries when on the local host. When executing the provided script remotely, in addition to modifying the LOCAL_REGISTRY variable to match the IP address of the registry's location, it is necessary to insert an entry in the insecure-registries: section within the etc/docker/daemon.json file. Following this adjustment, you must restart the Docker service.

Your registry is ready! On your next installation, update your /home/sysadmin/localhost.yml bootstrap overrides file with the following lines to use it:

.. group-tab:: Private Registry

docker_registries:
  quay.io:
    url: <your IP address>:5000/quay.io
  gcr.io:
    url: <your IP address>:5000/gcr.io
  k8s.gcr.io:
    url: <your IP address>:5000/k8s.gcr.io
  docker.io:
    url: <your IP address>:5000/docker.io
  docker.elastic.co:
    url: <your IP address>:5000/docker.elastic.co
  ghcr.io:
    url: <your IP address>:5000/ghcr.io
  registry.k8s.io:
    url: <your IP address>:5000/registry.k8s.io
  icr.io:
    url: <your IP address>:5000/icr.io
  defaults:
    type: docker
    secure: false

With Pull-through cache

docker_registries:
  quay.io:
    url: <your IP address>:<port_1>
  gcr.io:
    url: <your IP address>:<port_2>
  k8s.gcr.io:
    url: <your IP address>:<port_3>
  docker.io:
    url: <your IP address>:<port_4>
  docker.elastic.co:
    url: <your IP address>:<port_5>
  ghcr.io:
    url: <your IP address>:<port_6>
  registry.k8s.io:
    url: <your IP address>:<port_7>
  icr.io:
    url: <your IP address>:<port_8>
  defaults:
    type: docker
    secure: false

Note

This procedure configured to use an insecure registry via the docker_registries.defaults.secure parameter set to false in the excerpt above. Make sure you only use this on your own development environment.