eeac5467c3
This sets up a robots.txt on our lists servers. To start this file prevents SEMrush bot from indexing our lists as that has been causing lists.openstack.org to OOM with many listinfo processes started by Apache. We've avoided this OOM by manually configuring this robots.txt. Other things we have ruled out are bup and input email causes qrunner's to grow unexpectedly large. Fairly confident this bot is the trigger. Note this fixes testing by adding 'hieradata' to set listpassword var. Depends-On: https://review.opendev.org/724389 Change-Id: Id4f6739a8cf6a01f9796fa54c86ba1af3e31fecf
3 lines
35 B
Plaintext
3 lines
35 B
Plaintext
User-agent: SemrushBot
|
|
Disallow: /
|