Merge "Add a specification for Mailman 3"
This commit is contained in:
Zuul
Gerrit Code Review
commit
56c1288bc5
@ -47,6 +47,7 @@ permits.
|
||||
specs/wiki_modernization
|
||||
specs/website-stats
|
||||
specs/xwiki-farm
|
||||
specs/mailman3
|
||||
|
||||
Help Wanted
|
||||
===========
|
||||
|
||||
258
specs/mailman3.rst
Normal file
258
specs/mailman3.rst
Normal file
@ -0,0 +1,258 @@
|
||||
::
|
||||
|
||||
Copyright 2021 Jeremy Stanley
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0
|
||||
Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
===================
|
||||
Mailman 3 Migration
|
||||
===================
|
||||
|
||||
https://storyboard.openstack.org/#!/story/2009249
|
||||
|
||||
This specification covers the implementation of a new Mailman 3
|
||||
based listserv in OpenDev, and subsequent migration of the existing
|
||||
Mailman 2 based mailing lists we provide to the new server,
|
||||
concluding with taking down the old server once all lists have been
|
||||
migrated. Care will be taken to preserve existing list
|
||||
configurations as much as the existing migration tooling allows, as
|
||||
well as import all historical list archives and ensure old archive
|
||||
hyperlinks still take people to the same content.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
For years now, the Mailman community has been producing two series
|
||||
of mailing list software:
|
||||
|
||||
- Mailman 2, which we currently use and have since circa 2010,
|
||||
offers a stable and battle-hardened mailing list service focused
|
||||
on exchanging information via E-mail, with an optional public Web
|
||||
archive called Pipermail.
|
||||
|
||||
- Mailman 3, which was experimental for a very long time but has
|
||||
become more stable and essentially reached feature parity with its
|
||||
predecessor some years ago now, is functionally the same
|
||||
internally as far as exchanging E-mail messages is concerned, but
|
||||
has expanded substantially on its Web interface with a new
|
||||
Postorius list management interface and Hyperkitty interactive Web
|
||||
archive.
|
||||
|
||||
In the decade since we started running Mailman, significant shifts
|
||||
have occurred in the ways people interface with E-mail and hold
|
||||
group discussions. The mass freemail providers have become
|
||||
entrenched, ubiquitous, and emboldened to begin breaking
|
||||
long-established protocol standards. Portable handheld devices are
|
||||
becoming the primary way many people interact. Web forums have
|
||||
morphed into social media platforms, and their popularity has
|
||||
further altered expectations of the next generation of computer
|
||||
users. At the same time, these shifts in behavior have driven
|
||||
changes in how Web applications are developed and delivered.
|
||||
|
||||
Mailman 3's Hyperkitty archive interface is essentially a bolt-on
|
||||
Web forum frontend, with more advanced authentication and some
|
||||
(mis?)features borrowed from popular social media. Its Postorius
|
||||
list management interface uses the same new authentication
|
||||
subsystems, having distinct per-user accounts and role-based access
|
||||
controls instead of the shared password model used by MM2. Both of
|
||||
these are now Django based applications, so can take advantage of
|
||||
the Django ecosystem of additional features for further
|
||||
customization such as social authentication and integration with
|
||||
external identity providers.
|
||||
|
||||
Among relevant improvements in Mailman 3 is proper virtual domain
|
||||
support, which should allow us to greatly simplify our current
|
||||
implementation by hosting all our various list sites in a single
|
||||
deployment without fragile workarounds, and more easily add new
|
||||
sites in the future.
|
||||
|
||||
Proposed Change
|
||||
===============
|
||||
|
||||
Since there are semi-official `Mailman 3 containers
|
||||
<https://asynchronous.in/docker-mailman/>`_ already available on
|
||||
GHCR/Quay/DockerHub, this seems like the easiest path forward and
|
||||
most compatible with our present model of configuration management
|
||||
and orchestration. They publish three applications as individual
|
||||
containers: mailman-core (the listserv), mailman-web (a.k.a.
|
||||
Hyperkitty), and postorius. In addition, it's assumed that the
|
||||
system(s) where these run will provide a Web server to act as a
|
||||
proxying front-end, an SMTP server, and an RDBMS.
|
||||
|
||||
We'll write Ansible playbooks and Zuul jobs to deploy the three
|
||||
Mailman service containers along with an accompanying MariaDB
|
||||
container, and then configure system packages of Apache and Exim to
|
||||
support these (similar to how we do with our existing services, e.g.
|
||||
Gerrit). The Mailman container documentation provides examples for
|
||||
configuring `Exim <https://asynchronous.in/docker-mailman/#exim4>`_
|
||||
and `Nginx
|
||||
<https://asynchronous.in/docker-mailman/#setting-up-your-web-server>`_
|
||||
but the latter can be easily adapted to Apache for increased
|
||||
consistency with our other Web-based services, and the Mailman
|
||||
documentation also has some `tips for this
|
||||
<https://docs.list.org/projects/hyperkitty/en/latest/install.html?highlight=apache#running-hyperkitty-on-apache-with-mod-wsgi>`_.
|
||||
|
||||
Following our usual test-driven development methodology, once basic
|
||||
functionality is working and tested successfully, we'll hold one or
|
||||
more test nodes and perform direct interactive testing, first with
|
||||
sample mailing lists, but later with imported production
|
||||
configuration and archives (taking sufficient measures to avoid
|
||||
annoying people with unwanted tests or duplicate messages). The
|
||||
Mailman developers provide a convenient `migration tool
|
||||
<https://docs.list.org/en/latest/migration.html>`_ which we can use
|
||||
for this purpose. We should also be sure to note any significant
|
||||
changes in behavior for things like bounce processing or which
|
||||
mailing-list-specific headers get included in messages, so that we
|
||||
can provide users with better guidance around the migration.
|
||||
|
||||
Once we're confident the deployment and config/archive imports are
|
||||
working as expected, we'll boot a new production mailing list
|
||||
server, then schedule and announce a migration window when we'll be
|
||||
moving all the existing sites and their lists to the new server. A
|
||||
full copy of the old pre-migration Pipermail archives will also be
|
||||
served from the new server in order to preserve the functionality of
|
||||
any existing bookmarks or historical hyperlinks. The
|
||||
lists.opendev.org virtual domain will migrate first in order to test
|
||||
the waters, and then the remaining virtual domains will be migrated
|
||||
individually as is convenient for their communities.
|
||||
|
||||
After all migrations are completed and any remaining issues sorted,
|
||||
the old mailing list servers will be imaged for posterity and then
|
||||
deleted.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
We could consider using distro packaged Mailman 3 like we presently
|
||||
do for Mailman 2, the packages for it in Ubuntu 20.04 LTS are quite
|
||||
usable at this point, though slightly outdated compared to what we
|
||||
get from the Mailman 3 container images. The down-side here is that
|
||||
we only consume new features when upgrading the operating system
|
||||
itself to a new major release, and this tight coupling tends to
|
||||
complicate things as MM3 is still a bit of a moving target and two
|
||||
years of deferred upgrades could mean many significant changes in
|
||||
behavior to juggle and untangle at the same time. On the other hand,
|
||||
relying on distribution packages in theory shields us from
|
||||
unexpected changes outside operating system upgrades, and (if the
|
||||
packages are well maintained) means we get to skip bugs impacting
|
||||
upstream releases which may be ironed out in packaging before they
|
||||
ever trickle down to us.
|
||||
|
||||
We could also of course consider switching to a different mailing
|
||||
list software entirely, or sticking with Mailman 2 (it's still
|
||||
maintained), or migrate everyone to a SaaS list provider, or decide
|
||||
to stop running mailing lists entirely. Any of these would require a
|
||||
much different spec however.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
- fungi
|
||||
- clarkb
|
||||
|
||||
Gerrit Topic
|
||||
------------
|
||||
|
||||
Use Gerrit topic "mailman3" for all patches related to this spec.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git-review -t mailman3
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Create deployment playbooks and related testing.
|
||||
|
||||
2. Test config migration and archive import on a held test node.
|
||||
|
||||
3. Create new production list server.
|
||||
|
||||
4. Schedule and announce virtual domain migrations.
|
||||
|
||||
5. Migrate all Mailman virtual domains to the new server.
|
||||
|
||||
6. Stop the old servers, image them for posterity, and then delete.
|
||||
|
||||
Repositories
|
||||
------------
|
||||
|
||||
No new git repositories need to be created.
|
||||
|
||||
Servers
|
||||
-------
|
||||
|
||||
A new lists01.opendev.org server will be created using our default
|
||||
image (currently Ubuntu 20.04 LTS), and the old
|
||||
lists.katacontainers.io and lists.openstack.org server will be
|
||||
removed.
|
||||
|
||||
DNS Entries
|
||||
-----------
|
||||
|
||||
The lists.airshipit.org, lists.katacontainers.io, lists.opendev.org,
|
||||
lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org DNS
|
||||
records will need updating to the addresses of the new server during
|
||||
migration, as will the DNS records for any other virtual domains
|
||||
added to the old servers prior to migration. New lists01.opendev.org
|
||||
records will be added, as will ACME records to support HTTPS for all
|
||||
the virtual domains.
|
||||
|
||||
Backups
|
||||
-------
|
||||
|
||||
Borg backups will be configured for the new lists01.opendev.org
|
||||
server prior to migration of any virtual domains, and backups of the
|
||||
old server will be retained after migration for as long as is
|
||||
reasonable.
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
The system-config `mailing lists
|
||||
<https://docs.opendev.org/opendev/system-config/latest/lists.html>`_
|
||||
document will be updated to reflect the details of the new
|
||||
deployment. Additionally, user-facing documentation in the
|
||||
Infrastructure Manual or maintained by individual project
|
||||
communities may need updating if it mentions details of list
|
||||
subscription or includes screenshots.
|
||||
|
||||
Security
|
||||
--------
|
||||
|
||||
This is overall a net improvement in security of the service, since
|
||||
Mailman 3 uses stronger (effectively non-brute-forcible) password
|
||||
confirmation tokens, a proper password reset option rather than
|
||||
simple reminders, stores password hashes internally instead of
|
||||
plaintext copies, provides role-based access control rather than
|
||||
relying on shared passwords, can integrate with other authentication
|
||||
and identity providers if we want, and will be deployed with HTTPS
|
||||
in order to reinforce the stronger security model.
|
||||
|
||||
The only significant security risk is that we operate a handful of
|
||||
private mailing lists, and so extra care should be taken during
|
||||
migration to avoid exposing or otherwise leaking the archives for
|
||||
these. We should also make sure to double-check lists with moderated
|
||||
subscription settings to be sure unauthorized users can't subscribe
|
||||
after the list has been moved to the new system.
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
Basic functional testing of deployment and Web interfaces (including
|
||||
some screen capture artifacts) will be added as part of creating the
|
||||
configuration management and orchestration for the service.
|
||||
Additional on-demand config and archive migration testing will be
|
||||
performed with held job nodes. In future, end-to-end testing of
|
||||
message delivery and archiving can easily be added to help prevent
|
||||
regressions when evaluating new versions of the container images.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
There are no dependencies outside of the spec itself.
|
||||
Loading…
Reference in New Issue
Block a user