Merge "Add a specification for Mailman 3"
This commit is contained in:
Zuul
Gerrit Code Review
commit
56c1288bc5
@ -47,6 +47,7 @@ permits.
|
|||||||
specs/wiki_modernization
|
specs/wiki_modernization
|
||||||
specs/website-stats
|
specs/website-stats
|
||||||
specs/xwiki-farm
|
specs/xwiki-farm
|
||||||
|
specs/mailman3
|
||||||
|
|
||||||
Help Wanted
|
Help Wanted
|
||||||
===========
|
===========
|
||||||
|
|||||||
258
specs/mailman3.rst
Normal file
258
specs/mailman3.rst
Normal file
@ -0,0 +1,258 @@
|
|||||||
|
::
|
||||||
|
|
||||||
|
Copyright 2021 Jeremy Stanley
|
||||||
|
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0
|
||||||
|
Unported License.
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
===================
|
||||||
|
Mailman 3 Migration
|
||||||
|
===================
|
||||||
|
|
||||||
|
https://storyboard.openstack.org/#!/story/2009249
|
||||||
|
|
||||||
|
This specification covers the implementation of a new Mailman 3
|
||||||
|
based listserv in OpenDev, and subsequent migration of the existing
|
||||||
|
Mailman 2 based mailing lists we provide to the new server,
|
||||||
|
concluding with taking down the old server once all lists have been
|
||||||
|
migrated. Care will be taken to preserve existing list
|
||||||
|
configurations as much as the existing migration tooling allows, as
|
||||||
|
well as import all historical list archives and ensure old archive
|
||||||
|
hyperlinks still take people to the same content.
|
||||||
|
|
||||||
|
Problem Description
|
||||||
|
===================
|
||||||
|
|
||||||
|
For years now, the Mailman community has been producing two series
|
||||||
|
of mailing list software:
|
||||||
|
|
||||||
|
- Mailman 2, which we currently use and have since circa 2010,
|
||||||
|
offers a stable and battle-hardened mailing list service focused
|
||||||
|
on exchanging information via E-mail, with an optional public Web
|
||||||
|
archive called Pipermail.
|
||||||
|
|
||||||
|
- Mailman 3, which was experimental for a very long time but has
|
||||||
|
become more stable and essentially reached feature parity with its
|
||||||
|
predecessor some years ago now, is functionally the same
|
||||||
|
internally as far as exchanging E-mail messages is concerned, but
|
||||||
|
has expanded substantially on its Web interface with a new
|
||||||
|
Postorius list management interface and Hyperkitty interactive Web
|
||||||
|
archive.
|
||||||
|
|
||||||
|
In the decade since we started running Mailman, significant shifts
|
||||||
|
have occurred in the ways people interface with E-mail and hold
|
||||||
|
group discussions. The mass freemail providers have become
|
||||||
|
entrenched, ubiquitous, and emboldened to begin breaking
|
||||||
|
long-established protocol standards. Portable handheld devices are
|
||||||
|
becoming the primary way many people interact. Web forums have
|
||||||
|
morphed into social media platforms, and their popularity has
|
||||||
|
further altered expectations of the next generation of computer
|
||||||
|
users. At the same time, these shifts in behavior have driven
|
||||||
|
changes in how Web applications are developed and delivered.
|
||||||
|
|
||||||
|
Mailman 3's Hyperkitty archive interface is essentially a bolt-on
|
||||||
|
Web forum frontend, with more advanced authentication and some
|
||||||
|
(mis?)features borrowed from popular social media. Its Postorius
|
||||||
|
list management interface uses the same new authentication
|
||||||
|
subsystems, having distinct per-user accounts and role-based access
|
||||||
|
controls instead of the shared password model used by MM2. Both of
|
||||||
|
these are now Django based applications, so can take advantage of
|
||||||
|
the Django ecosystem of additional features for further
|
||||||
|
customization such as social authentication and integration with
|
||||||
|
external identity providers.
|
||||||
|
|
||||||
|
Among relevant improvements in Mailman 3 is proper virtual domain
|
||||||
|
support, which should allow us to greatly simplify our current
|
||||||
|
implementation by hosting all our various list sites in a single
|
||||||
|
deployment without fragile workarounds, and more easily add new
|
||||||
|
sites in the future.
|
||||||
|
|
||||||
|
Proposed Change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Since there are semi-official `Mailman 3 containers
|
||||||
|
<https://asynchronous.in/docker-mailman/>`_ already available on
|
||||||
|
GHCR/Quay/DockerHub, this seems like the easiest path forward and
|
||||||
|
most compatible with our present model of configuration management
|
||||||
|
and orchestration. They publish three applications as individual
|
||||||
|
containers: mailman-core (the listserv), mailman-web (a.k.a.
|
||||||
|
Hyperkitty), and postorius. In addition, it's assumed that the
|
||||||
|
system(s) where these run will provide a Web server to act as a
|
||||||
|
proxying front-end, an SMTP server, and an RDBMS.
|
||||||
|
|
||||||
|
We'll write Ansible playbooks and Zuul jobs to deploy the three
|
||||||
|
Mailman service containers along with an accompanying MariaDB
|
||||||
|
container, and then configure system packages of Apache and Exim to
|
||||||
|
support these (similar to how we do with our existing services, e.g.
|
||||||
|
Gerrit). The Mailman container documentation provides examples for
|
||||||
|
configuring `Exim <https://asynchronous.in/docker-mailman/#exim4>`_
|
||||||
|
and `Nginx
|
||||||
|
<https://asynchronous.in/docker-mailman/#setting-up-your-web-server>`_
|
||||||
|
but the latter can be easily adapted to Apache for increased
|
||||||
|
consistency with our other Web-based services, and the Mailman
|
||||||
|
documentation also has some `tips for this
|
||||||
|
<https://docs.list.org/projects/hyperkitty/en/latest/install.html?highlight=apache#running-hyperkitty-on-apache-with-mod-wsgi>`_.
|
||||||
|
|
||||||
|
Following our usual test-driven development methodology, once basic
|
||||||
|
functionality is working and tested successfully, we'll hold one or
|
||||||
|
more test nodes and perform direct interactive testing, first with
|
||||||
|
sample mailing lists, but later with imported production
|
||||||
|
configuration and archives (taking sufficient measures to avoid
|
||||||
|
annoying people with unwanted tests or duplicate messages). The
|
||||||
|
Mailman developers provide a convenient `migration tool
|
||||||
|
<https://docs.list.org/en/latest/migration.html>`_ which we can use
|
||||||
|
for this purpose. We should also be sure to note any significant
|
||||||
|
changes in behavior for things like bounce processing or which
|
||||||
|
mailing-list-specific headers get included in messages, so that we
|
||||||
|
can provide users with better guidance around the migration.
|
||||||
|
|
||||||
|
Once we're confident the deployment and config/archive imports are
|
||||||
|
working as expected, we'll boot a new production mailing list
|
||||||
|
server, then schedule and announce a migration window when we'll be
|
||||||
|
moving all the existing sites and their lists to the new server. A
|
||||||
|
full copy of the old pre-migration Pipermail archives will also be
|
||||||
|
served from the new server in order to preserve the functionality of
|
||||||
|
any existing bookmarks or historical hyperlinks. The
|
||||||
|
lists.opendev.org virtual domain will migrate first in order to test
|
||||||
|
the waters, and then the remaining virtual domains will be migrated
|
||||||
|
individually as is convenient for their communities.
|
||||||
|
|
||||||
|
After all migrations are completed and any remaining issues sorted,
|
||||||
|
the old mailing list servers will be imaged for posterity and then
|
||||||
|
deleted.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
We could consider using distro packaged Mailman 3 like we presently
|
||||||
|
do for Mailman 2, the packages for it in Ubuntu 20.04 LTS are quite
|
||||||
|
usable at this point, though slightly outdated compared to what we
|
||||||
|
get from the Mailman 3 container images. The down-side here is that
|
||||||
|
we only consume new features when upgrading the operating system
|
||||||
|
itself to a new major release, and this tight coupling tends to
|
||||||
|
complicate things as MM3 is still a bit of a moving target and two
|
||||||
|
years of deferred upgrades could mean many significant changes in
|
||||||
|
behavior to juggle and untangle at the same time. On the other hand,
|
||||||
|
relying on distribution packages in theory shields us from
|
||||||
|
unexpected changes outside operating system upgrades, and (if the
|
||||||
|
packages are well maintained) means we get to skip bugs impacting
|
||||||
|
upstream releases which may be ironed out in packaging before they
|
||||||
|
ever trickle down to us.
|
||||||
|
|
||||||
|
We could also of course consider switching to a different mailing
|
||||||
|
list software entirely, or sticking with Mailman 2 (it's still
|
||||||
|
maintained), or migrate everyone to a SaaS list provider, or decide
|
||||||
|
to stop running mailing lists entirely. Any of these would require a
|
||||||
|
much different spec however.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
- fungi
|
||||||
|
- clarkb
|
||||||
|
|
||||||
|
Gerrit Topic
|
||||||
|
------------
|
||||||
|
|
||||||
|
Use Gerrit topic "mailman3" for all patches related to this spec.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
git-review -t mailman3
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. Create deployment playbooks and related testing.
|
||||||
|
|
||||||
|
2. Test config migration and archive import on a held test node.
|
||||||
|
|
||||||
|
3. Create new production list server.
|
||||||
|
|
||||||
|
4. Schedule and announce virtual domain migrations.
|
||||||
|
|
||||||
|
5. Migrate all Mailman virtual domains to the new server.
|
||||||
|
|
||||||
|
6. Stop the old servers, image them for posterity, and then delete.
|
||||||
|
|
||||||
|
Repositories
|
||||||
|
------------
|
||||||
|
|
||||||
|
No new git repositories need to be created.
|
||||||
|
|
||||||
|
Servers
|
||||||
|
-------
|
||||||
|
|
||||||
|
A new lists01.opendev.org server will be created using our default
|
||||||
|
image (currently Ubuntu 20.04 LTS), and the old
|
||||||
|
lists.katacontainers.io and lists.openstack.org server will be
|
||||||
|
removed.
|
||||||
|
|
||||||
|
DNS Entries
|
||||||
|
-----------
|
||||||
|
|
||||||
|
The lists.airshipit.org, lists.katacontainers.io, lists.opendev.org,
|
||||||
|
lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org DNS
|
||||||
|
records will need updating to the addresses of the new server during
|
||||||
|
migration, as will the DNS records for any other virtual domains
|
||||||
|
added to the old servers prior to migration. New lists01.opendev.org
|
||||||
|
records will be added, as will ACME records to support HTTPS for all
|
||||||
|
the virtual domains.
|
||||||
|
|
||||||
|
Backups
|
||||||
|
-------
|
||||||
|
|
||||||
|
Borg backups will be configured for the new lists01.opendev.org
|
||||||
|
server prior to migration of any virtual domains, and backups of the
|
||||||
|
old server will be retained after migration for as long as is
|
||||||
|
reasonable.
|
||||||
|
|
||||||
|
Documentation
|
||||||
|
-------------
|
||||||
|
|
||||||
|
The system-config `mailing lists
|
||||||
|
<https://docs.opendev.org/opendev/system-config/latest/lists.html>`_
|
||||||
|
document will be updated to reflect the details of the new
|
||||||
|
deployment. Additionally, user-facing documentation in the
|
||||||
|
Infrastructure Manual or maintained by individual project
|
||||||
|
communities may need updating if it mentions details of list
|
||||||
|
subscription or includes screenshots.
|
||||||
|
|
||||||
|
Security
|
||||||
|
--------
|
||||||
|
|
||||||
|
This is overall a net improvement in security of the service, since
|
||||||
|
Mailman 3 uses stronger (effectively non-brute-forcible) password
|
||||||
|
confirmation tokens, a proper password reset option rather than
|
||||||
|
simple reminders, stores password hashes internally instead of
|
||||||
|
plaintext copies, provides role-based access control rather than
|
||||||
|
relying on shared passwords, can integrate with other authentication
|
||||||
|
and identity providers if we want, and will be deployed with HTTPS
|
||||||
|
in order to reinforce the stronger security model.
|
||||||
|
|
||||||
|
The only significant security risk is that we operate a handful of
|
||||||
|
private mailing lists, and so extra care should be taken during
|
||||||
|
migration to avoid exposing or otherwise leaking the archives for
|
||||||
|
these. We should also make sure to double-check lists with moderated
|
||||||
|
subscription settings to be sure unauthorized users can't subscribe
|
||||||
|
after the list has been moved to the new system.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
-------
|
||||||
|
|
||||||
|
Basic functional testing of deployment and Web interfaces (including
|
||||||
|
some screen capture artifacts) will be added as part of creating the
|
||||||
|
configuration management and orchestration for the service.
|
||||||
|
Additional on-demand config and archive migration testing will be
|
||||||
|
performed with held job nodes. In future, end-to-end testing of
|
||||||
|
message delivery and archiving can easily be added to help prevent
|
||||||
|
regressions when evaluating new versions of the container images.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
There are no dependencies outside of the spec itself.
|
||||||
Loading…
Reference in New Issue
Block a user