opendev/system-config
Code Issues Proposed changes

Merge "Remove the old mailing list server"

Browse Source
This commit is contained in:
Zuul 2023-10-20 23:04:26 +00:00 committed by Gerrit Code Review
commit bd3fd30462
14 changed files with 23 additions and 416 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

81
doc/source/lists.rst
View File

@ -5,33 +5,33 @@
Mailing Lists
#############
`Mailman <http://www.gnu.org/software/mailman/>`_ is installed on
lists.openstack.org to run OpenStack related mailing lists, as well as
host list archives.
`The Mailman Suite <https://www.list.org/>`_ (Mailman Core, Postorius,
Hyperkitty) is installed on lists.opendev.org to run mailing lists for the
OpenDev Collaboratory and projects it hosts, as well as archives of those
lists.
At a Glance
===========
:Hosts:
* http://lists.openstack.org
* https://lists.opendev.org (and numerous other aliases)
:Ansible:
* :git_file:`playbooks/service-lists.yaml`
* :git_file:`playbooks/roles/mailman`
* :git_file:`playbooks/roles/mailman-site`
* :git_file:`playbooks/roles/mailman-list`
* :git_file:`playbooks/service-lists3.yaml`
* :git_file:`playbooks/roles/mailman3`
:Projects:
* http://www.gnu.org/software/mailman/
* https://www.list.org/
* https://gitlab.com/mailman/
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* https://bugs.launchpad.net/mailman
* https://gitlab.com/mailman/
:Resources:
* `Mailman Documentation <http://www.gnu.org/software/mailman/docs.html>`_
* `Mailman Documentation <https://docs.mailman3.org/>`_
Adding a List
=============
A list may be added by adding it to the ``openstack-infra/system-config``
repository in :git_file:`inventory/service/host_vars/lists.openstack.org.yaml`.
repository in :git_file:`inventory/service/host_vars/lists01.opendev.org.yaml`.
For example:
.. code-block:: yaml
@ -41,61 +41,14 @@ For example:
admin: 'admin@example.com'
password: "{{ mailman_list_password }}"
Scripted Changes to Lists
=========================
Scripted Interaction with Lists
===============================
This may only be performed with root access to the list server.
Mailman supports running a python code snippet in the context of
individual lists or every list on the system. The following example
adds an address to the list of banned addresses for every list. This
has proved useful in the case of attackers abusing the HTTP
subscription interface to subscribe a target's address to multiple
mailing lists.
Banning an Address from All Lists
---------------------------------
Create the file `/usr/lib/mailman/bin/ban.py` with the following
content:
.. code-block:: python
def ban(m, address):
try:
m.Lock()
if address not in m.ban_list:
m.ban_list.append(address)
m.Save()
finally:
m.Unlock()
And then run the withlist script as:
.. code-block:: bash
sudo -u list /usr/lib/mailman/bin/withlist -a -r ban "<address to ban>"
sudo docker-compose -f /etc/mailman-compose/docker-compose.yaml exec -T -u mailman mailman-core mailman help
Because the script itself handles locking, do not use the `-l`
argument to withlist. To run the same script on a single list, use:
.. code-block:: bash
sudo -u list /usr/lib/mailman/bin/withlist -r ban listname "<address to ban>"
Note that the ban list accepts regular expressions, so to ban an
address and all suffixes, use '^address.*@example.com' as the "address
to ban".
Lock Files
----------
If a list stops handling traffic for some time, it may be due to a
stale lock file. Mailman locks are in /srv/mailman/openstack/locks.
If a lock is held for a list, then ``listname.lock`` will exist. The
contents of the file will be the name of the lock sequence file which
was used to obtain the lock. That file is in the form
``listname.lock.hostname.pid.sequence``. If the process id in that
string no longer exists, it's safe to assume the process died without
cleaning up the lock. It should generally be safe to remove the
lockfile in that case.
Use context help for the CLI's many subcommands, or see the Mailman v3
documentation for more details.

6
doc/source/project.rst
View File

@ -39,7 +39,11 @@ Contributing
this list to get smaller over time.
We welcome contributions from new contributors. Reading this
documentation is the first step. You should also join our `mailing list <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>`_.
documentation is the first step. You should also join our `announcements
<https://lists.opendev.org/mailman3/lists/service-announce.lists.opendev.org/>`_
and `discussion
<https://lists.opendev.org/mailman3/lists/service-discuss.lists.opendev.org/>`_
mailing lists.
We are most active on IRC, so please join the **#opendev**
channel on OFTC.

1
hiera/common.yaml
View File

@ -29,7 +29,6 @@ cacti_hosts:
- kdc03.openstack.org
- kdc04.openstack.org
- keycloak01.opendev.org
- lists.openstack.org
- lists01.opendev.org
- nb01.opendev.org
- nb02.opendev.org

11
inventory/base/hosts.yaml
View File

@ -308,17 +308,6 @@ all:
- 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKUtLplUhod5VnjVoTY5WHhjOHrRM6puFpFpcr9iJmOKkbnJ5V2SA8U0thFEne4XUoa/eZ3SiQ9Yt923+1MAcKQ='
- 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5qje1++4tUZ1U4sQ2Jsju/S4BdpCeiauSxZ2uMdQSegtjZ4GclxRjP4zJjL6P/iixTwjsu4dOEnvPt8B9JZGEaYERzKiqjIRT3I80mTjI0wsx+bN38Z2xg5Tm1O5xrOxT0rjA2zGJDRtMhk6IwmUg4DELlxUfalsWgpoZV0fYxUFneOgVuG8XY841b1igh2ScyOuSfu8RQFF3YTulzoT7o8QzgdKiliciLAWujy+4okN8wln5/atqiDuN7oi+9WYLt/HW2YZTUHd2/u+ZghgvbVVJ8xsB2gQ+BESS3P4YZsWMqM/7lz/7GVUQfolRnC5dyPOa9cwuoBW9ru6VGYH/'
- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDSUpspKrIHEXRkP9xIa/hyKkauDvuPX0nVwWpUzQkIh'
lists.openstack.org:
ansible_host: 50.56.173.222
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 50.56.173.222
public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736
host_keys:
- 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJXPszbNKhny/BiR9JAC/yvV5Knqh6ZgUlk21XzJlk7D3kCBcyuMCSlWskvQU/59pYM5pDSBf8+a//pvxoR3Cm0MBPrMEYGT3LcGY/lxsgHoHaHhMgRvdHKVNPsO852CuecCB31HnUZ5c0cZJQ3V5UzZzld5FRvp9tV6vxKdaLrkA8HEuktMj59Tgm3zUv7a6Ou6JV78NC8zFAw73ZUZYNgRZ118MSd3ZogBoHOLY0FEWF6vwYHOC3qp3655+SEHEzJOVxknntxeeaopVLqxuIOkfjDCAc8DIBhbFr/IevK34XWSlRJjXvQnYKpzA1/urwugsLvQKqFwjeVQZOn5Bb'
- 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH7wzqKuzV5f6N5lCoOMqdXE0j/LpGAU5/vfPadqzadQCY4xApfpbJgCnGWJ6dESAKhv8HRIZ7nVKuns8NfNQIU='
- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL4fR5POWlUrnu8JGSKLCxPFY5VCOwveqsDgCDWimWCD'
lists01.opendev.org:
ansible_host: 162.209.78.70
location:

4
inventory/service/groups.yaml
View File

@ -39,7 +39,6 @@ groups:
- lists99.opendev.org
# All these servers are "special-cased" in specifically
# as they are puppet and should be replaced "soon"
- lists.openstack.org
- storyboard01.opendev.org
- translate01.openstack.org
borg-backup-server:
@ -92,7 +91,6 @@ groups:
- graphite[0-9]*.opendev.org
- insecure-ci-registry[0-9]*.opendev.org
- keycloak[0-9]*.opendev.org
- lists.openstack.org
- lists[0-9]*.opendev.org
- meetpad[0-9]*.opendev.org
- mirror[0-9]*.opendev.org
@ -105,8 +103,6 @@ groups:
- tracing[0-9]*.opendev.org
- translate[0-9]*.open*.org
- zuul[0-9]*.opendev.org
mailman:
- lists.openstack.org
mailman3:
- lists[0-9]*.opendev.org
meetpad:

168
inventory/service/host_vars/lists.openstack.org.yaml
View File

@ -1,168 +0,0 @@
mm_domains: 'lists.openstack.org'
exim_local_domains: "@:{{ mm_domains }}"
exim_enable_spf: true
exim_aliases:
root: "{{ ','.join(listadmins|default([])) }}"
interop-wg: openstack-discuss
openstack: openstack-discuss
openstack-dev: openstack-discuss
openstack-infra: openstack-discuss
openstack-operators: openstack-discuss
openstack-security: openstack-discuss
openstack-sigs: openstack-discuss
openstack-tc: openstack-discuss
user-committee: openstack-discuss
airship-discuss-owner: spam
community-owner: spam
edge-computing-owner: spam
foundation-board-confidential-owner: spam
foundation-board-owner: spam
foundation-owner: spam
legal-discuss-owner: spam
mailman-owner: spam
marketing-owner: spam
openstack-announce-owner: spam
openstack-docs-owner: spam
openstack-fr-owner: spam
openstack-i18n-owner: spam
openstack-infra-owner: spam
openstack-ko-owner: spam
openstack-qa-owner: spam
product-wg-owner: spam
user-committee-owner: spam
spam: ':fail: delivery temporarily disabled due to ongoing spam flood'
exim_domain_aliases:
community@lists.openstack.org: community@lists.openinfra.dev
edge-computing@lists.openstack.org: edge-computing@lists.opendev.org
foundation@lists.openstack.org: foundation@lists.openinfra.dev
foundation-board@lists.openstack.org: foundation-board@lists.openinfra.dev
foundation-board-confidential@lists.openstack.org: foundation-board-confidential@lists.openinfra.dev
goldmembers@lists.openstack.org: goldmembers@lists.openinfra.dev
marketing@lists.openstack.org: marketing@lists.openinfra.dev
staff@lists.openstack.org: staff@lists.openinfra.dev
summit-programming-committee@lists.openinfra.dev: summit-track-chairs@lists.openinfra.dev
summitsponsors@lists.openstack.org: summitsponsors@lists.openinfra.dev
exim_routers:
- mailman_verp_router: |
{% raw -%}
driver = dnslookup
condition = ${if or{{eq{$sender_host_address}{127.0.0.1}}\
{eq{$sender_host_address}{::1}}}{yes}{no}}
{% endraw %}
domains = !+local_domains
ignore_target_hosts = <; 0.0.0.0; \
127.0.0.0/8; \
::1/128;fe80::/10;fe \
c0::/10;ff00::/8
senders = "*-bounces@*"
transport = mailman_verp_smtp
- dnslookup: '{{ exim_dnslookup_router }}'
- system_aliases: '{{ exim_system_aliases_router }}'
- domain_aliases: |
driver = redirect
allow_fail
allow_defer
data = ${lookup{$local_part@$domain}lsearch{/etc/aliases.domain}}
file_transport = address_file
pipe_transport = address_pipe
- localuser: '{{ exim_localuser_router }}'
- mailman_copy: |
driver = accept
domains = lists.openstack.org
local_parts = openstack-discuss
transport = local_copy
unseen
- mailman_router: |
driver = accept
domains = {{ mm_domains }}
local_part_suffix = -admin : \
-bounces : -bounces+* : \
-confirm : -confirm+* : \
-join : -leave : \
-owner : -request : \
-subscribe : -unsubscribe
local_part_suffix_optional
require_files = ${lookup{${lc::$domain}}lsearch{/etc/mailman/sites}}/lists/${lc::$local_part}/config.pck
transport = mailman_transport
exim_transports:
- local_copy: |
driver = appendfile
file = /var/mail/$local_part
group = mail
mode = 0660
- mailman_transport: |
driver = pipe
command = /var/lib/mailman/mail/mailman \
'${if def:local_part_suffix \
{${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
{post}}' \
$local_part
current_directory = /var/lib/mailman
environment = HOST=${lc:$domain}
group = list
home_directory = /var/lib/mailman
user = list
- mailman_verp_smtp: |
driver = smtp
headers_add = Errors-To: ${return_path}
headers_remove = Errors-To
max_rcpt = 1
return_path = ${local_part:$return_path}+$local_part=$domain@${domain:$return_path}
# We put lists.openstack.org first as it's the current servername
letsencrypt_certs:
lists-openstack-org-main:
- lists.openstack.org
mailman_multihost: true
mailman_sites:
- name: openstack
listdomain: lists.openstack.org
install_languages: ['de', 'fr', 'it', 'ko', 'ru', 'vi', 'zh_TW']
lists:
- name: mailman
description: 'The mailman site list'
admin: 'nobody@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-es
description: 'Lista de correo acerca de OpenStack en español'
admin: 'flavio@redhat.com'
password: "{{ mailman_list_password }}"
- name: openstack-fr
description: 'List of the OpenStack french user group'
admin: 'erwan@erwan.com'
password: "{{ mailman_list_password }}"
- name: openstack-i18n
description: 'List of the OpenStack Internationalization team.'
admin: 'guoyingc@cn.ibm.com'
password: "{{ mailman_list_password }}"
- name: openstack-it
description: 'Discussioni su OpenStack in italiano'
admin: 'stefano@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-ko
description: 'OpenStack Korea Community Discussions in Korean (오픈스택 한국 커뮤니티 메일링리스트)'
admin: 'ianyrchoi@gmail.com'
password: "{{ mailman_list_password }}"
- name: openstack-zh
description: 'OpenStack社区中文讨论群组'
admin: 'yeluaiesec@gmail.com'
password: "{{ mailman_list_password }}"
- name: release-job-failures
description: 'Notification messages for failures from release-related build jobs.'
admin: 'doug@doughellmann.com'
password: "{{ mailman_list_password }}"
- name: embargo-notice
description: 'Announcements to stakeholders for embargoed security vulnerabilities.'
admin: 'jeremy@openstack.org'
password: "{{ mailman_list_password }}"
- name: release-announce
description: 'Announcement of official OpenStack releases.'
admin: 'thierry@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-mentoring
description: 'List to coordinate interactions between mentors and mentees of the OpenStack mentoring program. Also for questions about the mentoring program (i.e. how to get involved, how it works, etc.'
admin: 'amy@demarco.com'
password: "{{ mailman_list_password }}"
- name: openstack-discuss
description: 'Discussion of OpenStack usage and development.'
admin: 'fungi@yuggoth.org'
password: "{{ mailman_list_password }}"

3
playbooks/roles/static/files/ask.openstack.org/index.html
View File

@ -18,7 +18,7 @@
<p>The following options are available for community-based support.</p>
<ul>
<li><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss">openstack-discuss</a> mailing list.</li>
<li><a href="https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/">openstack-discuss</a> mailing list.</li>
<li><a href="https://stackoverflow.com/questions/tagged/openstack">stackoverflow.com</a> for code questions.</li>
<li><a href="https://serverfault.com/tags/openstack/info">serverfault.com</a> for operations.</li>
<li>The <a href="https://web.archive.org/web/*/ask.openstack.org">Internet
@ -27,4 +27,3 @@
</body>
</html>

26
playbooks/test-lists.yaml
View File

@ -1,26 +0,0 @@
- hosts: "lists.openstack.org"
tasks:
# Make sure Mailman services for each site are running so that they
# will attempt to deliver any pending list admin notifications and
# we can capture that activity in the Exim logs.
- name: Restart Mailman services
service:
name: "mailman-{{ zuul_mailman_site.name }}"
state: restarted
loop: "{{ mailman_sites }}"
loop_control:
loop_var: zuul_mailman_site
- hosts: "localhost"
tasks:
# Pause briefly so that mail deliveries are attempted, otherwise the
# job may finish too quickly and we won't have a complete picture in
# the MTA logs by the time we start collecting them.
#
# TODO(fungi): We could improve this to wait for specific log entries
# instead, I suppose.
- name: Wait for delivery attempts
wait_for:
timeout: 10

1
playbooks/zuul/run-base.yaml
View File

@ -142,7 +142,6 @@
- host_vars/etherpad99.opendev.org.yaml
- host_vars/letsencrypt01.opendev.org.yaml
- host_vars/letsencrypt02.opendev.org.yaml
- host_vars/lists.openstack.org.yaml
- host_vars/gitea99.opendev.org.yaml
- host_vars/grafana01.opendev.org.yaml
- host_vars/mirror01.openafs.provider.opendev.org.yaml

11
playbooks/zuul/run-lists-post.yaml
View File

@ -1,11 +0,0 @@
- hosts: lists.openstack.org
tasks:
- name: Collect per-site mailman log directories
shell:
cmd: |
mkdir -p /var/log/mailman
for logs in /srv/mailman/*/logs ; do
cp -a $logs /var/log/mailman/$(basename $(dirname $logs))
done
become: yes
ignore_errors: true

62
testinfra/test_lists_o_o.py
View File

@ -1,62 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
testinfra_hosts = ['lists.openstack.org']
def test_mm_list_is_present(host):
cmd = host.run('HOST=lists.openstack.org list_lists --bare')
assert 'openstack-discuss' in cmd.stdout
def test_mm_list_site(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/cgi-bin/mailman/listinfo')
assert '<TITLE>lists.openstack.org Mailing Lists</TITLE>' in cmd.stdout
def test_mm_list_site_redirect_http(host):
cmd = host.run('curl '
'--resolve lists.openstack.org:80:127.0.0.1 '
'http://lists.openstack.org/cgi-bin/mailman/listinfo')
assert ('The document has moved <a href="'
'https://lists.openstack.org/cgi-bin/mailman/listinfo'
'">here</a>') in cmd.stdout
def test_mm_list_site_redirect_listinfo(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/cgi-bin/mailman/listinfo/staff')
assert ('The document has moved <a href="'
'https://lists.openinfra.dev/cgi-bin/mailman/listinfo/staff'
'">here</a>') in cmd.stdout
def test_mm_list_site_redirect_archives(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/pipermail/staff/')
assert ('The document has moved <a href="'
'https://lists.openinfra.dev/pipermail/staff/'
'">here</a>') in cmd.stdout
def test_mm_list_site_static_files(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/archives.yaml')
assert 'openstack-discuss' in cmd.stdout
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/robots.txt')
assert 'Disallow: /' in cmd.stdout
def test_domain_aliases(host):
domain_aliases = host.file('/etc/aliases.domain')
assert domain_aliases.exists
assert domain_aliases.contains('staff@lists.openstack.org: staff@lists.openinfra.dev')

14
zuul.d/infra-prod.yaml
View File

@ -561,20 +561,6 @@
- playbooks/roles/graphite/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-lists
parent: infra-prod-service-base
description: Run service-lists.yaml playbook.
vars:
playbook_name: service-lists.yaml
files:
- inventory/base
- inventory/service/host_vars/lists.openstack.org.yaml
- playbooks/roles/iptables/
- playbooks/roles/base/exim
- playbooks/roles/mailman/
- playbooks/service-lists.yaml
- job:
name: infra-prod-service-lists3
parent: infra-prod-service-base

9
zuul.d/project.yaml
View File

@ -29,7 +29,6 @@
- name: system-config-build-image-hound
soft: true
- system-config-run-kerberos
- system-config-run-lists
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
@ -196,7 +195,6 @@
- name: system-config-upload-image-hound
soft: true
- system-config-run-kerberos
- system-config-run-lists
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
@ -509,12 +507,6 @@
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-lists: &infra-prod-service-lists
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-lists3: &infra-prod-service-lists3
dependencies:
- name: infra-prod-service-borg-backup
@ -665,7 +657,6 @@
- infra-prod-service-graphite: *infra-prod-service-graphite
- infra-prod-service-keycloak: *infra-prod-service-keycloak
- infra-prod-service-meetpad: *infra-prod-service-meetpad
- infra-prod-service-lists: *infra-prod-service-lists
- infra-prod-service-lists3: *infra-prod-service-lists3
- infra-prod-service-mirror: *infra-prod-service-mirror
- infra-prod-service-nodepool: *infra-prod-service-nodepool

42
zuul.d/system-config-run.yaml
View File

@ -277,48 +277,6 @@
- playbooks/roles/letsencrypt-install-txt-record
- playbooks/roles/letsencrypt-request-certs
- job:
name: system-config-run-lists
parent: system-config-run
description: |
Run the playbook for a list server.
post-run: playbooks/zuul/run-lists-post.yaml
nodeset:
nodes:
- <<: *bridge_node_x86
- name: lists.openstack.org
label: ubuntu-focal
groups:
- <<: *bastion_group
required-projects:
- opendev/system-config
files:
- playbooks/bootstrap-bridge.yaml
- inventory/service/host_vars/lists.openstack.org.yaml
- inventory/service/group_vars/mailman.yaml
- playbooks/roles/base/exim
- playbooks/roles/mailman/
- playbooks/service-lists.yaml
- playbooks/test-lists.yaml
- playbooks/zuul/templates/host_vars/lists.openstack.org.yaml.j2
- testinfra/test_lists_o_o.py
- playbooks/zuul/run-lists-post.yaml
vars:
run_playbooks:
- playbooks/letsencrypt.yaml
- playbooks/service-lists.yaml
# Run this twice to check idempotency
- playbooks/service-lists.yaml
run_test_playbook: playbooks/test-lists.yaml
host-vars:
lists.openstack.org:
host_copy_output:
'/etc/aliases.domain': logs_txt
'/var/log/acme.sh': logs
'/var/log/apache2': logs
'/var/log/mailman': logs
'/etc/apache2/sites-enabled': logs
- job:
name: system-config-run-lists3
# We don't use the system-config-run-containers base job because we