From 1b8a1018fedac0e3f2d5ca37801feec18849427c Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Wed, 30 May 2018 09:56:14 +1000 Subject: [PATCH] Add some rationale for reprepro An interesting discussion this morning brought up some details of reprepro that maybe aren't widely understood. Add some rationale to the docs. Change-Id: I371cc67a10d29881b8df03f09830776bf5639ac8 --- doc/source/reprepro.rst | 57 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 5 deletions(-) diff --git a/doc/source/reprepro.rst b/doc/source/reprepro.rst index f47f5255db..e95e4ac831 100644 --- a/doc/source/reprepro.rst +++ b/doc/source/reprepro.rst @@ -25,15 +25,62 @@ At a Glance Overview ======== -reprepro is the tool we use to mirror Debian repositories (including -Ubuntu) to the AFS mirrors. +``reprepro`` is the tool we use to mirror Debian repositories +(including Ubuntu) to the AFS mirrors. + +When updating package mirrors, it is undesirable for the index to be +out of sync with the actual packages on disk, and vice-versa. This is +generally achieved by syncing in two stages -- firstly obtaining new +files in the mirror and then secondly updating the indexes (and +removing no-longer referenced files). + +Problems occur if the upstream mirror updates itself during this +process (which may happen up to 4 times a day). Debian, for example +runs a "push" model where first-tier mirrors are notified of +in-progress updates (see +``__) and can restart +any in-progress syncs to maintain consistency. OpenStack Infra is not +suitable to apply for these notifications, as our mirror is not +intended to be public and may be incomplete (we may not mirror all +suites, or architectures, etc. as our needs dictate). This means if +using other tools like `ftpsync +`__ primarily intended +for full replication we are very likely to have periods where our +mirror gets out of sync (with subsequent job failures). + +``reprepro`` is more commonly used to build and manage private +repositories, but has a number of features making it suitable for our +use. + +Rather than sync upstream indexes, it recreates them based upon files +gathered from the upstream mirror. Since the upstream mirror remains +consistent, ``reprepro`` will always download a consistent set of +files. Then thanks to the release of the AFS mirror volume being +atomic, we do not have any period where the repository package index +doens't match the set of packages in the filesystem. + +Since this does not require coordination with upstream, the same +pattern is suitable across Ubuntu, Debian and other various apt +repositories that may require integration (or perhaps do not provide +facilities) for correct mirroring. Although ``reprepro`` can be more +complicated to configure, it is consistent across these different +distributions. + +``reprepro`` also makes it fairly easy to mirror only certain suites +or architectures for a given repository, and to modify these +configurations as required. Repository signing ================== -Note our repositories are not signed. ``apt`` will require -``--no-check-gpg`` or similar settings in configuration to use -OpenStack mirrors. +Note our repositories are not signed since ``reprepro`` recreates the +indexes from scratch. This is actually somewhat helpful in avoiding +the infra mirrors becoming de facto mirrors for a range of unrelated +jobs (since we really do not guarantee contents for anything other +than infra jobs). + +``apt`` will require ``--no-check-gpg`` or similar settings in +configuration to use OpenStack mirrors. Normal operation ================