Set up robots.txt on lists servers

This sets up a robots.txt on our lists servers. To start this file
prevents SEMrush bot from indexing our lists as that has been causing
lists.openstack.org to OOM with many listinfo processes started by
Apache.

We've avoided this OOM by manually configuring this robots.txt. Other
things we have ruled out are bup and input email causes qrunner's to
grow unexpectedly large. Fairly confident this bot is the trigger.

Note this fixes testing by adding 'hieradata' to set listpassword var.

Depends-On: https://review.opendev.org/724389
Change-Id: Id4f6739a8cf6a01f9796fa54c86ba1af3e31fecf

modules/openstack_project/files/mailman/robots.txt

@@ -0,0 +1,2 @@
+User-agent: SemrushBot
+Disallow: /

modules/openstack_project/manifests/lists.pp

@@ -8,6 +8,15 @@ class openstack_project::lists(
     multihost => true,
   }
 
+  file { '/var/www/robots.txt':
+    ensure  => present,
+    source  => 'puppet:///modules/openstack_project/mailman/robots.txt',
+    owner   => 'root',
+    group   => 'root',
+    replace => true,
+    mode    => '0444',
+  }
+
   # Disable inactive admins
   user::virtual::disable { 'oubiwann': }
   user::virtual::disable { 'rockstar': }

playbooks/zuul/run-base.yaml

@@ -67,6 +67,7 @@
         - host_vars/etherpad01.opendev.org.yaml
         - host_vars/letsencrypt01.opendev.org.yaml
         - host_vars/letsencrypt02.opendev.org.yaml
+        - host_vars/lists.openstack.org.yaml
         - host_vars/gitea99.opendev.org.yaml
         - host_vars/mirror01.openafs.provider.opendev.org.yaml
         - host_vars/mirror-update01.opendev.org.yaml

playbooks/zuul/templates/host_vars/lists.openstack.org.yaml.j2

@@ -0,0 +1 @@
+listpassword: notarealpassword