DAIMI Mail System Upgraded
I am storing all my mail since 1999 on DAIMI’s IMAP server — around 750 MiB in total for 26,726 mails. It is very nice of them to provide such a server, but unfortunately it is slow… so slow that I have finally taken the time to switch to something better.
It was actually quite simple to switch. DAIMI uses the UW-IMAP server configured to store mails in mbx files which work similar to mbox files except that they contain some extra meta data that should speed up access to them. In my experience it still took the IMAP server 10-30 seconds to look at my 35 mailboxes only to determine that all but one contained no new messages!
I have had great success with the Dovecot IMAP server in the past, so that is what I switched to. It is very fast, easy to configure, and robust. Dovecot can use the excellent Maildir mailbox format which I like because it is simple and fast. Each mail sits in its own file which makes it fast to delete a single message or more it from one folder to another since you don’t have to rewrite a 200 MiB mbx file…
Converting the mbx files to Maildir directories was a two-step process: first I used mailutil to convert from mbx to mbox files:
for f in *; do mailutil -verbose copy $f "#driver.unix/../mboxes/$f"; done
That was done in the root IMAP folder used by UW-IMAP. The result is a bunch of mbox files in a directory called mboxes besides the IMAP directory. Converting these into Maildir directories was done using the mb2md tool. I first converted /var/spool/mail/mg into ~/Maildir:
mb2md -m
I then converted the mbx files in mboxes:
mb2md -s mboxes
It might have been better to use the -f flag which seems to strip out the dummy messages inserted by UW-IMAP. But deleting them afterwards was easy:
rm ~/Maildir/.*/cur/*.000000.*
I first looked at the files with cat to verify that they were all dummy messages. A final problem was the INBOX file which ended up as ~/Maildir/.INBOX where it was shadowed by the normal ~/Maildir/{cur,new,tmp} directories (which represent INBOX in Dovecot). Moving the contents of ~/Maildir/.INBOX/{cur,new,tmp} to ~/Maildir/{cur,new,tmp} makes the old inbox messages visible again.
After the conversion I had a nice ~/Maildir directory which can be read directly by Gnus and other mail clients. But I also want to read it remotely from my home machine. This is where Dovecot comes into the picture — I wanted to use it to present the Maildir as an IMAP server to my Gnus at home. This was very easy:
(setq imap-shell-program "ssh mail.daimi.au.dk dovecot --exec-mail imap")
(setq gnus-secondary-select-methods '((nnimap "daimi" (nnimap-stream shell))))
This makes Gnus connect to DAIMI via SSH and launch dovecot as an IMAP server. Dovecot defaults to look at ~/Maildir so this works out of the box. And it is much faster than the old UW-IMAP server, scanning for new mail now takes about 1 second.
Getting new mail into the Maildir instead of the usual place is a simple matter of making a .forward file:
"|/mail/mg/opt/libexec/dovecot/deliver"
I compiled Dovecot using --prefix=$HOME/opt so its local delivery agent ends up at $HOME/opt/libexec/dovecot/deliver. I actually use maildrop to sort and SPAM check my mail (the DAIMI filter is way to lenient) before passing it on to deliver. But the principle is the same.
What I am missing out on here is the ability to use the webmail interface offered by DAIMI. I mostly used this to check that I really saw all message in Gnus since the old IMAP server would often fail to show me new messages in INBOX. That should no longer be necessary with Dovecot… I guess I could also install my own webmail if needed — it just has to be able to manipulate a Maildir.