Where to put the mail?

DAIMI has decided to rename itself to CS, and as part of that process my current address of mg@daimi.au.dk will change to mg@cs.au.dk. Okay, I can live with that, except that they apparently want to disable the old address at some point next year! That is stupid if you ask me — there are many, many papers out there that reference a @daimi.au.dk address, and I have been using it for all my software development in the last couple of years.

So instead of just going with mg@cs.au.dk as my new address I want a new address, probably mg@mgsys.dk, which is a not-so-well-known address I’ve had for a while. Please let me know what do you think of that address.

Three days ago I got a mail from the DAIMI, ehmm…, CS staff that informed me that they will be changing their mail system next Saturday. I’m affected by this since I’ve already hacked their system by upgrading it to Dovecot instead of whatever old and slow IMAP server they were using. So I have until Saturday to either migrate my mail back to their slow server or to move to somewhere else.

The mgsys.dk domain is hosted at DreamHost, and they provide all the space I could ever want. Except that they only allow 2000 mails in the INBOX folder! I have 3207 mails at the moment for the year 2008, so this will be sort of annoying for me. Also, their SPAM filter seems a bit crude: it will only allow me to see the messages in the quarantine folder by logging into their webmail system. And I hate such systems.

Another option is to move everything to Gmail — and access it through IMAP. A number of people seem to use Gnus with Gmail, so I guess it will work okay. There is even a Gmail Lab thingy that will let you make Gmail behave more like a normal IMAP server. I don’t know how good Gmail’s SPAM filter is, but I assume it is good? Though, after hearing from Claudio that it has flagged two of my mails to viff-devel as SPAM for no apparent reason I am not really sure.

Right now CS does some graylisting for me and run the mails through SpamAssassin. I run them through SA a second time with my own settings. During the last week I caught about 200 SPAMs by this second filter. By moving to Gmail I would have a much harder time doing such custom filtering (at least if it should be done asynchronously and before the mail hits my folders). Moving to DreamHost would allow me to setup any wacky scheme I want for now, but they too want to consolidate their mail on special servers with no support for .forward files.

Hmm… It sounds like Gmail would be the simplest solution. There is of course the issue about privacy, but I guess Google knows what I’m doing anyway… :-)

Large Hadron Rap

I saw this mentioned on Planet Python today:

Arrived in Switzerland

My three month stay at IBM has started, and so far it is going well. I arrived Sunday and had my first day Monday. The first day was spend doing administration: I got my picture taken for a chip card which is used for opening doors and for paying in the canteen. I also got a bunch of stuff about how I should look after the environment, how I should look after my own health (sit correctly at my desk), and especially about how I should take care not to leave any “IBM confidential” material lying around at my desk when I’m not there. Protecting the IP (”Imaginary Property”, ehh, I mean “Intellectual Property”) is taken pretty serious around here, but since I don’t expect to be involved in anything confidential I should be safe…

I have no pictures to show yet since i don’t have a way to get them off my camera. Back home I always just used the card reader in my monitor, but here I don’t have such a reader. So I will have to find and buy a USB cable (I couldn’t find the original before I left).

So that’s all for now!

Oh, just one more thing… VIFF 0.6 has been released last week. It has greatly improved performance, updated documentation, and various bug fixes. So please try it out!

DAIMI Mail System Upgraded

I am storing all my mail since 1999 on DAIMI’s IMAP server — around 750 MiB in total for 26,726 mails. It is very nice of them to provide such a server, but unfortunately it is slow… so slow that I have finally taken the time to switch to something better.

It was actually quite simple to switch. DAIMI uses the UW-IMAP server configured to store mails in mbx files which work similar to mbox files except that they contain some extra meta data that should speed up access to them. In my experience it still took the IMAP server 10–30 seconds to look at my 35 mailboxes only to determine that all but one contained no new messages!

I have had great success with the Dovecot IMAP server in the past, so that is what I switched to. It is very fast, easy to configure, and robust. Dovecot can use the excellent Maildir mailbox format which I like because it is simple and fast. Each mail sits in its own file which makes it fast to delete a single message or more it from one folder to another since you don’t have to rewrite a 200 MiB mbx file…

Converting the mbx files to Maildir directories was a two-step process: first I used mailutil to convert from mbx to mbox files:

for f in *; do mailutil -verbose copy $f "#driver.unix/../mboxes/$f"; done

That was done in the root IMAP folder used by UW-IMAP. The result is a bunch of mbox files in a directory called mboxes besides the IMAP directory. Converting these into Maildir directories was done using the mb2md tool. I first converted /var/spool/mail/mg into ~/Maildir:

mb2md -m

I then converted the mbx files in mboxes:

mb2md -s mboxes

It might have been better to use the -f flag which seems to strip out the dummy messages inserted by UW-IMAP. But deleting them afterwards was easy:

rm ~/Maildir/.*/cur/*.000000.*

I first looked at the files with cat to verify that they were all dummy messages. A final problem was the INBOX file which ended up as ~/Maildir/.INBOX where it was shadowed by the normal ~/Maildir/{cur,new,tmp} directories (which represent INBOX in Dovecot). Moving the contents of ~/Maildir/.INBOX/{cur,new,tmp} to ~/Maildir/{cur,new,tmp} makes the old inbox messages visible again.

After the conversion I had a nice ~/Maildir directory which can be read directly by Gnus and other mail clients. But I also want to read it remotely from my home machine. This is where Dovecot comes into the picture — I wanted to use it to present the Maildir as an IMAP server to my Gnus at home. This was very easy:

(setq imap-shell-program "ssh mail.daimi.au.dk dovecot --exec-mail imap")
(setq gnus-secondary-select-methods '((nnimap "daimi" (nnimap-stream shell))))

This makes Gnus connect to DAIMI via SSH and launch dovecot as an IMAP server. Dovecot defaults to look at ~/Maildir so this works out of the box. And it is much faster than the old UW-IMAP server, scanning for new mail now takes about 1 second.

Getting new mail into the Maildir instead of the usual place is a simple matter of making a .forward file:

"|/mail/mg/opt/libexec/dovecot/deliver"

I compiled Dovecot using --prefix=$HOME/opt so its local delivery agent ends up at $HOME/opt/libexec/dovecot/deliver. I actually use maildrop to sort and SPAM check my mail (the DAIMI filter is way to lenient) before passing it on to deliver. But the principle is the same.

What I am missing out on here is the ability to use the webmail interface offered by DAIMI. I mostly used this to check that I really saw all message in Gnus since the old IMAP server would often fail to show me new messages in INBOX. That should no longer be necessary with Dovecot… I guess I could also install my own webmail if needed — it just has to be able to manipulate a Maildir.

Get the new VIFF 0.5

VIFF 0.5 has been released — this is the latest and greatest VIFF release so far with preliminary support for pre-processing, an actively secure multiplication protocol (it is only twice as slow as the passively secure protocol!) and much more… Download at http://viff.dk/.

For the next release we are working on improving the documentation by using the excellent Sphinx documentation generator. This is the same system used by Python itself and I really like the way it encourages us to write documentation for humans instead of computers. By that I mean that the documentation is more free-form and so you automatically stop writing stuff like

@param share_a: The first share.
@type share_a: Share

which adds nothing to the documentation, except filling out some blanks in the generated API doc.

If you have any ideas for VIFF, then please let us hear! We would love to see more people use it — please tell us what is good and what is bad about the framework.