Browse Source

add Pipermail archiver

master
Max Erenberg 1 year ago
parent
commit
fd9ed1dfa1
  1. 1
      .gitignore
  2. 67
      README.md
  3. 239
      pipermail/newlist-pipermail
  4. 2
      setup.cfg
  5. 2
      src/extra_mailman_archivers/monthly_archiver.py
  6. 131
      src/extra_mailman_archivers/pipermail.py

1
.gitignore vendored

@ -1,2 +1,3 @@
/src/extra_mailman_archivers.egg-info/
__pycache__/
*.swp

67
README.md

@ -1,10 +1,28 @@
# Extra Mailman Archivers
This contains some extra archivers to be used with Mailman 3.
Currently there is only one archiver, `MonthlyArchiver`, which simply gzips
Currently there are two archivers, `MonthlyArchiver`, and `Pipermail`.
## Installation
Make sure to have the following packages installed first:
```sh
apt install python3-pip python3-setuptools python3-wheel
```
Then:
```sh
pip3 install .
```
## MonthlyArchiver
MonthlyArchiver simply gzips
each message and stores it in a folder named by the year and month. Each
message's file name is a Unix epoch timestamp.
The motivation behind `MonthlyArchiver` was to avoid storing thousands of
messages in a single directory, which the `Prototype` archiver currently
does since it uses a maildir.
Example directory structure:
```
/var/lib/mailman3/archives
@ -17,16 +35,43 @@ Example directory structure:
\_ ...
```
The motivation behind `MonthlyArchiver` was to avoid storing thousands of
messages in a single directory, which the `Prototype` archiver currently
does since it uses a maildir.
To enable this archiver, add the following to your mailman.cfg:
```
[archiver.monthly_archiver]
class: extra_mailman_archivers.monthly_archiver.MonthlyArchiver
enable: yes
```
## Installation
Make sure to have the following packages installed first:
```sh
apt install python3-pip python3-setuptools python3-wheel
## Pipermail
This archiver attempts to add messages to Pipermail, the archiver for
Mailman 2. An existing Mailman 2 installation is required.
If the mailing list does not exist in the Mailman 2 directory, you
must create it first. Use the `newlist-pipermail` script in the `pipermail`
directory after copying it to the `bin` directory where Mailman 2
is installed. It works the same way as the traditional `newlist` script
except that it does not communicate with the MTA. **Note**: by default,
this will create a public archive; to make it private, unlink the appropriate
folder in the `archives/public` directory where Mailman 2 is installed.
To enable this archiver, add the following to your mailman.cfg:
```
Then:
```sh
pip3 install .
[archiver.pipermail]
class: extra_mailman_archivers.pipermail.Pipermail
enable: yes
```
The default list URL for this archiver is `http://$domain/pipermail/$short_listname`.
The default Mailman 2 directory is `/var/lib/mailman`.
To change either of these values, add the following to the pipermail
section in mailman.cfg:
```
configuration: /etc/mailman3/mailman-pipermail.cfg
```
Then, in `/etc/mailman3/mailman-pipermail.cfg`, set `base_url` and/or `mailman2_dir`
appropriately, e.g.
```
[general]
base_url: http://$domain/old/pipermail/$short_listname
```

239
pipermail/newlist-pipermail

@ -0,0 +1,239 @@
#! /usr/bin/python
#
# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
# USA.
"""Create a new, unpopulated mailing list.
Usage: %(PROGRAM)s [options] [listname [listadmin-addr [admin-password]]]
Options:
-l language
--language=language
Make the list's preferred language `language', which must be a two
letter language code.
-u urlhost
--urlhost=urlhost
Gives the list's web interface host name.
-e emailhost
--emailhost=emailhost
Gives the list's email domain name.
-q/--quiet
Normally the administrator is notified by email (after a prompt) that
their list has been created. This option suppresses the prompt and
notification.
-a/--automate
This option suppresses the prompt prior to administrator notification
but still sends the notification. It can be used to make newlist
totally non-interactive but still send the notification, assuming
listname, listadmin-addr and admin-password are all specified on the
command line.
-h/--help
Print this help text and exit.
You can specify as many of the arguments as you want on the command line:
you will be prompted for the missing ones.
Every Mailman list has two parameters which define the default host name for
outgoing email, and the default URL for all web interfaces. When you
configured Mailman, certain defaults were calculated, but if you are running
multiple virtual Mailman sites, then the defaults may not be appropriate for
the list you are creating.
You also specify the domain to create your new list in by typing the command
like so:
newlist --urlhost=www.mydom.ain mylist
where `www.mydom.ain' should be the base hostname for the URL to this virtual
hosts's lists. E.g. with this setting people will view the general list
overviews at http://www.mydom.ain/mailman/listinfo. Also, www.mydom.ain
should be a key in the VIRTUAL_HOSTS mapping in mm_cfg.py/Defaults.py if
the email hostname to be automatically determined.
If you want the email hostname to be different from the one looked up by the
VIRTUAL_HOSTS or if urlhost is not registered in VIRTUAL_HOSTS, you can specify
`emailhost' like so:
newlist --urlhost=www.mydom.ain --emailhost=mydom.ain mylist
where `mydom.ain' is the mail domain name. If you don't specify emailhost but
urlhost is not in the virtual host list, then mm_cfg.DEFAULT_EMAIL_HOST will
be used for the email interface.
For backward compatibility, you can also specify the domain to create your
new list in by spelling the listname like so:
mylist@www.mydom.ain
where www.mydom.ain is used for `urlhost' but it will also be used for
`emailhost' if it is not found in the virtual host table. Note that
'--urlhost' and '--emailhost' have precedence to this notation.
If you spell the list name as just `mylist', then the email hostname will be
taken from DEFAULT_EMAIL_HOST and the url will be taken from DEFAULT_URL_HOST
interpolated into DEFAULT_URL_PATTERN (as defined in your Defaults.py file or
overridden by settings in mm_cfg.py).
Note that listnames are forced to lowercase.
"""
import sys
import os
import getpass
import getopt
import paths
from Mailman import mm_cfg
from Mailman import MailList
from Mailman import Utils
from Mailman import Errors
from Mailman import Message
from Mailman import i18n
_ = i18n._
C_ = i18n.C_
PROGRAM = sys.argv[0]
def usage(code, msg=''):
if code:
fd = sys.stderr
else:
fd = sys.stdout
print >> fd, C_(__doc__)
if msg:
print >> fd, msg
sys.exit(code)
def main():
try:
opts, args = getopt.getopt(sys.argv[1:], 'hqal:u:e:',
['help', 'quiet', 'automate', 'language=',
'urlhost=', 'emailhost='])
except getopt.error, msg:
usage(1, msg)
lang = mm_cfg.DEFAULT_SERVER_LANGUAGE
quiet = False
automate = False
urlhost = None
emailhost = None
for opt, arg in opts:
if opt in ('-h', '--help'):
usage(0)
if opt in ('-q', '--quiet'):
quiet = True
if opt in ('-a', '--automate'):
automate = True
if opt in ('-l', '--language'):
lang = arg
if opt in ('-u', '--urlhost'):
urlhost = arg
if opt in ('-e', '--emailhost'):
emailhost = arg
# Is the language known?
if lang not in mm_cfg.LC_DESCRIPTIONS.keys():
usage(1, C_('Unknown language: %(lang)s'))
if len(args) > 0:
listname = args[0]
else:
listname = raw_input(C_('Enter the name of the list: '))
listname = listname.lower()
if '@' in listname:
# note that --urlhost and --emailhost have precedence
listname, domain = listname.split('@', 1)
urlhost = urlhost or domain
emailhost = emailhost or mm_cfg.VIRTUAL_HOSTS.get(domain, domain)
urlhost = urlhost or mm_cfg.DEFAULT_URL_HOST
host_name = emailhost or \
mm_cfg.VIRTUAL_HOSTS.get(urlhost, mm_cfg.DEFAULT_EMAIL_HOST)
web_page_url = mm_cfg.DEFAULT_URL_PATTERN % urlhost
if Utils.list_exists(listname):
usage(1, C_('List already exists: %(listname)s'))
if len(args) > 1:
owner_mail = args[1]
else:
owner_mail = raw_input(
C_('Enter the email of the person running the list: '))
if len(args) > 2:
listpasswd = args[2]
else:
listpasswd = getpass.getpass(C_('Initial %(listname)s password: '))
# List passwords cannot be empty
listpasswd = listpasswd.strip()
if not listpasswd:
usage(1, C_('The list password cannot be empty'))
mlist = MailList.MailList()
try:
pw = Utils.sha_new(listpasswd).hexdigest()
# Guarantee that all newly created files have the proper permission.
# proper group ownership should be assured by the autoconf script
# enforcing that all directories have the group sticky bit set
oldmask = os.umask(002)
try:
try:
if lang == mm_cfg.DEFAULT_SERVER_LANGUAGE:
langs = [lang]
else:
langs = [lang, mm_cfg.DEFAULT_SERVER_LANGUAGE]
mlist.Create(listname, owner_mail, pw, langs=langs,
emailhost=host_name, urlhost=urlhost)
finally:
os.umask(oldmask)
except Errors.BadListNameError, s:
usage(1, C_('Illegal list name: %(s)s'))
except Errors.EmailAddressError, s:
usage(1, C_('Bad owner email address: %(s)s') +
C_(' - owner addresses need to be fully-qualified names'
' like "owner@example.com", not just "owner".'))
except Errors.MMListAlreadyExistsError:
usage(1, C_('List already exists: %(listname)s'))
# Assign domain-specific attributes
mlist.host_name = host_name
mlist.web_page_url = web_page_url
# And assign the preferred language
mlist.preferred_language = lang
mlist.Save()
finally:
mlist.Unlock()
if __name__ == '__main__':
main()

2
setup.cfg

@ -1,6 +1,6 @@
[metadata]
name = extra-mailman-archivers
version = 0.0.1
version = 0.0.2
author = Max Erenberg
author_email = merenber@csclub.uwaterloo.ca
description = Some extra archivers for Mailman 3

2
src/extra_mailman_archivers/monthly_archiver.py

@ -37,7 +37,7 @@ from public import public
from zope.interface import implementer
log = logging.getLogger('mailman.error')
log = logging.getLogger('mailman.archiver')
@public

131
src/extra_mailman_archivers/pipermail.py

@ -0,0 +1,131 @@
# Copyright (C) 2021 Max Erenberg
# This file is adapted from GNU Mailman. The original copyright notice
# is preserved below.
#
# Copyright (C) 2008-2019 by the Free Software Foundation, Inc.
#
# This file is part of GNU Mailman.
#
# GNU Mailman is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
"""
This archiver passes messages to Pipermail. Mailman 2 must already be
installed.
"""
from contextlib import suppress
from datetime import timedelta
from email.generator import BytesGenerator
from io import BytesIO
import logging
import os
import tempfile
from flufl.lock import Lock
from mailman.config import config
from mailman.config.config import external_configuration
from mailman.interfaces.archiver import IArchiver
from mailman.interfaces.configuration import MissingConfigurationFileError
from mailman.utilities.string import expand
from public import public
from subprocess import PIPE, DEVNULL, Popen
from urllib.parse import urljoin
from zope.interface import implementer
log = logging.getLogger('mailman.archiver')
@public
@implementer(IArchiver)
class Pipermail:
"""Local archiver which passes messages to Pipermail."""
name = 'pipermail'
is_enabled = False
def __init__(self):
default_base_url = 'http://$domain/pipermail/$short_listname'
default_mailman2_dir = '/var/lib/mailman'
# Read our specific configuration file
try:
archiver_config = external_configuration(
config.archiver.pipermail.configuration)
self.base_url = archiver_config.get(
'general', 'base_url', fallback=default_base_url)
self.mailman2_dir = archiver_config.get(
'general', 'mailman2_dir', fallback=default_mailman2_dir)
except MissingConfigurationFileError:
self.base_url = default_base_url
self.mailman2_dir = default_mailman2_dir
def list_url(self, mlist):
"""See `IArchiver`."""
return expand(self.base_url, mlist)
def permalink(self, mlist, msg):
"""See `IArchiver`."""
# Unfortunately Pipermail URLs are not guaranteed to be stable
# since the mbox file can be modified and the archives regenerated.
return None
def archive_message(self, mlist, msg):
"""See `IArchiver`."""
# Mangle the 'From ' lines and add the Unix envelope header.
fp = BytesIO()
g = BytesGenerator(fp, mangle_from_=True)
g.flatten(msg, unixfrom=True)
msg_bytes = fp.getvalue()
mbox_path = os.path.join(
self.mailman2_dir, 'archives', 'private',
mlist.list_name + '.mbox', mlist.list_name + '.mbox')
lock_path = os.path.join(
config.LOCK_DIR, '{0}-pipermail.lock'.format(mlist.list_name))
lock = Lock(lock_path)
# Unfortunately the `bin/arch` script needs to seek/tell the mbox
# file, so we need to write the message to a real file.
fd, tempfile_name = tempfile.mkstemp()
fo = os.fdopen(fd, 'wb')
fo.write(msg_bytes)
fo.close()
try:
lock.lock(timeout=timedelta(seconds=3))
# Append the message to the mbox file.
with open(mbox_path, 'ab') as fo:
# Add a newline before and after to be absolutely sure we don't
# corrupt the mbox file.
fo.write(b'\n')
fo.write(msg_bytes)
fo.write(b'\n')
# Send the message to Pipermail.
args = ['bin/arch', mlist.list_name, tempfile_name]
proc = Popen(
args, cwd=self.mailman2_dir,
stdin=DEVNULL, stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
if proc.returncode != 0:
log.error('%s: bin/arch subprocess had non-zero exit code: %s' %
(msg.get('message-id'), proc.returncode))
log.info(stdout.decode())
log.error(stderr.decode())
finally:
lock.unlock(unconditionally=True)
os.unlink(tempfile_name)
return None
Loading…
Cancel
Save