upload-afs: rename to upload-afs-roots; add afs-upload-synchronize

The upload-afs role is actually fairly specialised for uploading docs
with its root-marker scheme.  As we're moving more parts of our
infrastructure to publish via AFS I think it's worth making this quite
explicit by renaming the upload-afs role to upload-afs-roots.  This
should be transparent as we call the renamed role, and once callers
have been migrated we can use the usual deprecation process.

Add a partner role, upload-afs-synchronize, which is more of a
straight copy with options to synchronize.  This will be suitable for
uploading tarball and release artifacts.  This is similar in concept
to the promotion job used by zuul [1], but in this case will be
focused on the extant publishing jobs in project-config which don't
currently use the artifact-reporting/promote process.

[1] https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/artifacts/promote.yaml#L39

Change-Id: I493d8829e3fd98e84f1b8f0e776e7ba41abf16c2
Story: #2006598
Task: #38600

doc/source/afs-roles.rst

@@ -3,5 +3,7 @@ AFS Roles
 
 .. zuul:autorole:: destroy-afs-token
 .. zuul:autorole:: upload-afs
+.. zuul:autorole:: upload-afs-roots
+.. zuul:autorole:: upload-afs-synchronize
 .. zuul:autorole:: create-afs-token
 .. zuul:autorole:: release-afs-volume

roles/upload-afs-roots/README.rst

@@ -0,0 +1,26 @@
+Copy contents from ``{{ zuul.executor.work_root }}/artifacts/`` to AFS
+
+This is intented for documentation publishing, it deletes files that
+do not exist in the content from the source.
+
+Before the job rsyncs the build into its final location, it must first
+create a list of directories that should not be deleted. This way if
+an entire directory is removed from a document, it will still be
+removed from the website, but directories which are themselves roots
+of other documents (for example, the stein branch) are not removed. A
+marker file, called `.root-marker`, at the root of each such directory
+will accomplish this; therefore each build job should also ensure that
+it leaves such a marker file at the root of its build. The job will
+find each of those in the destination hierarchy and add their
+containing directories to a list of directories to exclude from
+rsyncing.
+
+**Role Variables**
+
+.. zuul:rolevar:: afs_source
+
+  Path to local source directory.
+
+.. zuul:rolevar:: afs_target
+
+  Target path in AFS (should begin with '/afs/...').

roles/upload-afs-roots/tasks/main.yaml

@@ -0,0 +1,4 @@
+- name: Synchronize files to AFS
+  zuul_afs:
+    source: "{{ afs_source }}"
+    target: "{{ afs_target }}"

roles/upload-afs-synchronize/README.rst

@@ -0,0 +1,19 @@
+Copy contents from ``{{ zuul.executor.work_root }}/artifacts/`` to AFS
+
+**Role Variables**
+
+.. zuul:rolevar:: afs_source
+
+  Path to local source directory.
+
+.. zuul:rolevar:: afs_target
+
+  Target path in AFS (should begin with '/afs/...').
+
+.. zuul:rolevar:: afs_copy_only
+   :default: True
+
+   If set to `false`, this will specify `--delete-after` to remove
+   files on the remote side that do not exist on the copying side.
+   When set to `true` will act as a regular additive copy process and
+   will not remove any remote files.

roles/upload-afs-synchronize/defaults/main.yaml

@@ -0,0 +1,2 @@
+afs_source: "{{ zuul.executor.work_root }}/artifacts/"
+afs_copy_only: true

roles/upload-afs-synchronize/tasks/main.yaml

@@ -0,0 +1,12 @@
+- name: Upload contents to AFS
+  synchronize:
+    src: "{{ afs_source }}"
+    dest: "{{ afs_target }}"
+    # NOTE(ianw): you can't set group permissions on AFS, hence we
+    # don't set owner specifically.
+    archive: false
+    perms: true
+    times: true
+    recursive: true
+    rsync_opts: '{{ ["--safe-links"] + ["--delete-after"] if not afs_copy_only else [] }}'
+

roles/upload-afs/README.rst

@@ -1,26 +1,3 @@
 Copy contents from ``{{ zuul.executor.work_root }}/artifacts/`` to AFS
 
-This is intented for documentation publishing, it deletes files that
+This role is deprecated for :zuul:role:`upload-afs-roots`
-do not exist in the content from the source.
-
-Before the job rsyncs the build into its final location, it must first
-create a list of directories that should not be deleted. This way if
-an entire directory is removed from a document, it will still be
-removed from the website, but directories which are themselves roots
-of other documents (for example, the stein branch) are not removed. A
-marker file, called `.root-marker`, at the root of each such directory
-will accomplish this; therefore each build job should also ensure that
-it leaves such a marker file at the root of its build. The job will
-find each of those in the destination hierarchy and add their
-containing directories to a list of directories to exclude from
-rsyncing.
-
-**Role Variables**
-
-.. zuul:rolevar:: afs_source
-
-  Path to local source directory.
-
-.. zuul:rolevar:: afs_target
-
-  Target path in AFS (should begin with '/afs/...').

roles/upload-afs/tasks/main.yaml

@@ -1,4 +1,3 @@
-- name: Synchronize files to AFS
+- name: Call upload-afs-roots
-  zuul_afs:
+  include_role:
-    source: "{{ afs_source }}"
+    name: upload-afs-roots
-    target: "{{ afs_target }}"