DAIMI Mail System Upgraded
I am storing all my mail since 1999 on DAIMI’s IMAP server — around 750 MiB in total for 26,726 mails. It is very nice of them to provide such a server, but unfortunately it is slow… so slow that I have finally taken the time to switch to something better.
It was actually quite simple to switch. DAIMI uses the UW-IMAP server configured to store mails in mbx files which work similar to mbox files except that they contain some extra meta data that should speed up access to them. In my experience it still took the IMAP server 10–30 seconds to look at my 35 mailboxes only to determine that all but one contained no new messages!
I have had great success with the Dovecot IMAP server in the past, so that is what I switched to. It is very fast, easy to configure, and robust. Dovecot can use the excellent Maildir mailbox format which I like because it is simple and fast. Each mail sits in its own file which makes it fast to delete a single message or more it from one folder to another since you don’t have to rewrite a 200 MiB mbx file…
Converting the mbx files to Maildir directories was a two-step process: first I used mailutil to convert from mbx to mbox files:
for f in *; do mailutil -verbose copy $f "#driver.unix/../mboxes/$f"; done
That was done in the root IMAP folder used by UW-IMAP. The result is a bunch of mbox files in a directory called mboxes besides the IMAP directory. Converting these into Maildir directories was done using the mb2md tool. I first converted /var/spool/mail/mg into ~/Maildir:
mb2md -m
I then converted the mbx files in mboxes:
mb2md -s mboxes
It might have been better to use the -f flag which seems to strip out the dummy messages inserted by UW-IMAP. But deleting them afterwards was easy:
rm ~/Maildir/.*/cur/*.000000.*
I first looked at the files with cat to verify that they were all dummy messages. A final problem was the INBOX file which ended up as ~/Maildir/.INBOX where it was shadowed by the normal ~/Maildir/{cur,new,tmp} directories (which represent INBOX in Dovecot). Moving the contents of ~/Maildir/.INBOX/{cur,new,tmp} to ~/Maildir/{cur,new,tmp} makes the old inbox messages visible again.
After the conversion I had a nice ~/Maildir directory which can be read directly by Gnus and other mail clients. But I also want to read it remotely from my home machine. This is where Dovecot comes into the picture — I wanted to use it to present the Maildir as an IMAP server to my Gnus at home. This was very easy:
(setq imap-shell-program "ssh mail.daimi.au.dk dovecot --exec-mail imap")
(setq gnus-secondary-select-methods '((nnimap "daimi" (nnimap-stream shell))))
This makes Gnus connect to DAIMI via SSH and launch dovecot as an IMAP server. Dovecot defaults to look at ~/Maildir so this works out of the box. And it is much faster than the old UW-IMAP server, scanning for new mail now takes about 1 second.
Getting new mail into the Maildir instead of the usual place is a simple matter of making a .forward file:
"|/mail/mg/opt/libexec/dovecot/deliver"
I compiled Dovecot using --prefix=$HOME/opt so its local delivery agent ends up at $HOME/opt/libexec/dovecot/deliver. I actually use maildrop to sort and SPAM check my mail (the DAIMI filter is way to lenient) before passing it on to deliver. But the principle is the same.
What I am missing out on here is the ability to use the webmail interface offered by DAIMI. I mostly used this to check that I really saw all message in Gnus since the old IMAP server would often fail to show me new messages in INBOX. That should no longer be necessary with Dovecot… I guess I could also install my own webmail if needed — it just has to be able to manipulate a Maildir.
Thomas:
Note that you cannot access your home directory from the DAIMI webserver, so you would need to move the directory to setup a webmail system (on DAIMI at least)
2 May 2008, 4:44 pmMartin Geisler:
Yeah, that might be a show-stopper. The Maildir is actually at
/mail/mg/Maildiron all regular DAIMI machines, but I don’t know if that includes the webserver.Another problem is that DAIMI runs all PHP processes using the same user (gopher, I believe). That is a quite poor setup — I wont be able to manipulate the Maildir from PHP.
But I can easily setup a IMAP server on my office machine. Nobody minds me using that for a server.
2 May 2008, 6:09 pmThomas:
Probably not, but it’s probably not visible from the outside so you’ll have to setup some PROXY/forwarding stuff - but I guess that’s easy enough to do (I’ve done it before)
3 May 2008, 1:38 amMartin Geisler:
Right, my office machine is not reachable from outside the firewall, except indirectly via Apache which directs traffic for Roundup and BuildBot to it.
So if I setup some webmail system on the DAIMI webserver, then it should work nicely since the webserver can talk with my office machine behind the firewall. I can even secure the webmail since DAIMI has a SSL certificate for http://www.daimi.au.dk. I don’t think I’ll play with this right now, but all in all it sounds quite easy :-)
3 May 2008, 10:50 amRune B. Broberg:
This sounds … interesting. While I am sure we would probably give you the freedom to do what you like with your email, you should be aware that 1) it is certainly not supported by the staff and 2) we will be switching to a new email server setup, including new software, within a reasonably short time interval.
I can’t tell you the exact date, since I don’t know it yet, but don’t spend too much energy on this - it will most likely break when we switch, and we won’t be able to transfer your existing email to the new server.
My recommendation to people at DAIMI: Avoid. ;-)
8 May 2008, 6:45 amMartin Geisler:
Hi Rune, cool that you stopped by! Can you tell what kind of mail backend will the new system use? For as you can see, I have had terrible performance with the current system. Have you guys evaluated the use of Dovecot as IMAP server?
When you switch, will you no longer support custom
8 May 2008, 8:36 am~/.forwardfiles? That is all I require to be able to direct the mail where I need it. Of course~/.forwardfiles wont matter if the new system is as fast as my home-brew system.Rune B. Broberg:
Lots of questions there ;-) Answers follow:
The new server will be a Cyrus/Postfix setup.
I don’t know what amount of evaluation went on with Dovecot, but I know we did some tests of it at some point.
~/.forward-files won’t be supported, in the sense that you won’t have access to the file system any more :-) You can set up mail-forwarding if you like, but the only access to the mail system will be IMAP(S).
I hope the performance will be satisfactory; I haven’t heard of the performance problems you describe before, so I haven’t actually tested it on the new system. It seems to be widely used, and I have heard no complaints about it.
8 May 2008, 10:29 amMartin Geisler:
Thanks for the answers! I think Cyrus will be fine — from what I could find, people compare it with Dovecot in terms of performance, so that is cool.
However, I’m very sorry to hear that you will stop supporting
~/.forwardfiles. How are we to (asynchronously) sort our mails then?I also spam filter everything a second time since the default filter fails to catch about 10 SPAM mails a day for me. Unless the new mail system is way better at catching SPAM, I guess I will have to move to some other system.
8 May 2008, 11:45 amRune B. Broberg:
Short reply: RFC3028. :-)
8 May 2008, 11:57 amMartin:
Okay, that would be great! I really hate the procmail language, and the maildrop language (which I use now) is also quite fragile. The examples I have seen of Sieve looks better plus there are many tools for it. So I’m looking forward to it!
8 May 2008, 1:36 pmBrowder:
Really great info here… I found your site via Yahoo and I will bookmark it and come back again!
5 July 2009, 8:44 pmDave Fetrow:
For what it’s worth, the last version of UW-IMAP (via the c-client library), or at least its Panda Fork (still Mark Crispin) has a newish fast format (mx) designed to deal with
29 December 2009, 11:36 pmlarge amounts and sizes of mail; worth a look if coming across limits in earlier IMAP servers.