ae68691bf0
StarlingX needs to download a variety of rpms and tarballs from various upstream sources. Unfortunately the upstream sources are not always dependable. Either servers go down, are unreachable, or drop older content that we still depend on. Our proposed solution is to run our own mirror to capture an independent copy of the content needed by StarlingX. For this purpose, a server has been set up at http://mirror.starlingx.cengn.ca/mirror/centos The mirror will use deterministic paths derived from the upstream urls. Scripts will be used to convert an upstream url to the mirror's equivalent url (see function url_to_stx_mirror_url in url_utils.sh) The mirror will be refreshed daily. New .lst entries will be processed at that time. Processing of changes under yum.repos.d is not automated by this update. Expect a follow-up update to address this issue soon. These scripts are found under the 'stx_mirror_scripts' subdirectory. Changes are made to the download_mirror.sh script, and it's supporting scripts. New arguments have been added to each script to select the download source. -s StarlingX mirror only -S StarlingX mirror, with upstream source as backup -u Upstream source only -U Upstream source, with StarlingX mirror as backup You do not need to provide any of these flags. Continue to us download_mirror.sh as you always have. The default behavior is currently set to '-S', i.e. first try the StarlingX mirror, with upstream source as backup. If this proves to place to heavy a load on the existing server, we might switch the default to '-U', i.e. first try the upstream source, with StarlingX mirror as backup. If poor download performance is seen, you might want to try explicitly adding -U as an argument. The remaining two options are not recommended for regular use. Upstream only, i.e. '-u', restores original behaviour, but you may once again encounter rpms that have aged out, and been removed from their original repos. StarlingX only, i.e. '-s', is vulnerable if a .lst file has been updated, but the mirror has not yet processed it. Change-Id: I7e0f3d9fb99253662f9f4bf12457d39250408c0b Story: 2003906 Task: 26785 Signed-off-by: Scott Little <scott.little@windriver.com>
Create mirror for StarlingX
Step 0 - Build the container
Build the docker image on your Linux host (with Docker supported). NOTE: if necessary you might have to set http/https proxy in your Dockerfile before building the docker image below.
$ docker build -t <your_docker_image_name>:<your_image_version> -f Dockerfile .
Step 1 - Run the container
The container shall be run from the same directory where the other scripts are stored.
$ docker run -it -v $(pwd):/localdisk <your_docker_image_name>:<your_image_version>
The container can also be run the following way, so the
download_mirror.sh runs automatically without having to
enter the container and the step 2can be simplified.
$ docker run -it -v $(pwd):/localdisk <your_docker_image_name>:<your_image_version> download_mirror.sh
As /localdisk is defined as the workdir of the
container, the same folder name should be used to define the volume. The
container will start to run and populate a logs and
output folders in this directory. The container shall be
run from the same directory where the other scripts are stored.
step 2 - Run the
download_mirror.sh script
Once inside the container run the downloader script
$ ./download_mirror.sh
NOTE: in case there are some downloading failures due to network instability (or timeout), you should download them manually, to assure you get all RPMs listed in "rpms_from_3rd_parties.lst" and "rpms_from_centos_repo.lst".
step 3 - Copy the files to the mirror
After all downloading complete, copy the download files to mirror.
$ find ./output -name "*.i686.rpm" | xargs rm -f $ chown 751:751 -R ./output $ cp -rf output/stx-r1/ <your_mirror_folder>/
In this case <your_mirror_folder> can be whatever
folder you want to use as mirror.
step 4 - Tweaks in the StarlingX build system.
NOTE: step below is not needed if you've synced the latest codebase.
Go into StarlingX build system (another container which hosts cgcs build system), and follow up below steps:
Debugging issues
The download_mirror.sh script will create log files in
the form of centos_rpms_*.txt. After the download is
complete, it's recommended to check the content of these files to see if
everything was downloaded correctly.
A quick look into these files could be:
$ cd output/ $ cat missing
In this case, there shoudn't be any package in the "missing" files.