Archive for the ‘Computing’ Category.

Where to put the mail?

DAIMI has decided to rename itself to CS, and as part of that process my current address of will change to Okay, I can live with that, except that they apparently want to disable the old address at some point next year! That is stupid if you ask me — there are many, many papers out there that reference a address, and I have been using it for all my software development in the last couple of years.

So instead of just going with as my new address I want a new address, probably, which is a not-so-well-known address I’ve had for a while. Please let me know what do you think of that address.

Three days ago I got a mail from the DAIMI, ehmm…, CS staff that informed me that they will be changing their mail system next Saturday. I’m affected by this since I’ve already hacked their system by upgrading it to Dovecot instead of whatever old and slow IMAP server they were using. So I have until Saturday to either migrate my mail back to their slow server or to move to somewhere else.

The domain is hosted at DreamHost, and they provide all the space I could ever want. Except that they only allow 2000 mails in the INBOX folder! I have 3207 mails at the moment for the year 2008, so this will be sort of annoying for me. Also, their SPAM filter seems a bit crude: it will only allow me to see the messages in the quarantine folder by logging into their webmail system. And I hate such systems.

Another option is to move everything to Gmail — and access it through IMAP. A number of people seem to use Gnus with Gmail, so I guess it will work okay. There is even a Gmail Lab thingy that will let you make Gmail behave more like a normal IMAP server. I don’t know how good Gmail’s SPAM filter is, but I assume it is good? Though, after hearing from Claudio that it has flagged two of my mails to viff-devel as SPAM for no apparent reason I am not really sure.

Right now CS does some graylisting for me and run the mails through SpamAssassin. I run them through SA a second time with my own settings. During the last week I caught about 200 SPAMs by this second filter. By moving to Gmail I would have a much harder time doing such custom filtering (at least if it should be done asynchronously and before the mail hits my folders). Moving to DreamHost would allow me to setup any wacky scheme I want for now, but they too want to consolidate their mail on special servers with no support for .forward files.

Hmm… It sounds like Gmail would be the simplest solution. There is of course the issue about privacy, but I guess Google knows what I’m doing anyway… :-)

DAIMI Mail System Upgraded

I am storing all my mail since 1999 on DAIMI’s IMAP server — around 750 MiB in total for 26,726 mails. It is very nice of them to provide such a server, but unfortunately it is slow… so slow that I have finally taken the time to switch to something better.

It was actually quite simple to switch. DAIMI uses the UW-IMAP server configured to store mails in mbx files which work similar to mbox files except that they contain some extra meta data that should speed up access to them. In my experience it still took the IMAP server 10–30 seconds to look at my 35 mailboxes only to determine that all but one contained no new messages!

I have had great success with the Dovecot IMAP server in the past, so that is what I switched to. It is very fast, easy to configure, and robust. Dovecot can use the excellent Maildir mailbox format which I like because it is simple and fast. Each mail sits in its own file which makes it fast to delete a single message or more it from one folder to another since you don’t have to rewrite a 200 MiB mbx file…

Converting the mbx files to Maildir directories was a two-step process: first I used mailutil to convert from mbx to mbox files:

for f in *; do mailutil -verbose copy $f "#driver.unix/../mboxes/$f"; done

That was done in the root IMAP folder used by UW-IMAP. The result is a bunch of mbox files in a directory called mboxes besides the IMAP directory. Converting these into Maildir directories was done using the mb2md tool. I first converted /var/spool/mail/mg into ~/Maildir:

mb2md -m

I then converted the mbx files in mboxes:

mb2md -s mboxes

It might have been better to use the -f flag which seems to strip out the dummy messages inserted by UW-IMAP. But deleting them afterwards was easy:

rm ~/Maildir/.*/cur/*.000000.*

I first looked at the files with cat to verify that they were all dummy messages. A final problem was the INBOX file which ended up as ~/Maildir/.INBOX where it was shadowed by the normal ~/Maildir/{cur,new,tmp} directories (which represent INBOX in Dovecot). Moving the contents of ~/Maildir/.INBOX/{cur,new,tmp} to ~/Maildir/{cur,new,tmp} makes the old inbox messages visible again.

After the conversion I had a nice ~/Maildir directory which can be read directly by Gnus and other mail clients. But I also want to read it remotely from my home machine. This is where Dovecot comes into the picture — I wanted to use it to present the Maildir as an IMAP server to my Gnus at home. This was very easy:

(setq imap-shell-program "ssh dovecot --exec-mail imap")
(setq gnus-secondary-select-methods '((nnimap "daimi" (nnimap-stream shell))))

This makes Gnus connect to DAIMI via SSH and launch dovecot as an IMAP server. Dovecot defaults to look at ~/Maildir so this works out of the box. And it is much faster than the old UW-IMAP server, scanning for new mail now takes about 1 second.

Getting new mail into the Maildir instead of the usual place is a simple matter of making a .forward file:


I compiled Dovecot using --prefix=$HOME/opt so its local delivery agent ends up at $HOME/opt/libexec/dovecot/deliver. I actually use maildrop to sort and SPAM check my mail (the DAIMI filter is way to lenient) before passing it on to deliver. But the principle is the same.

What I am missing out on here is the ability to use the webmail interface offered by DAIMI. I mostly used this to check that I really saw all message in Gnus since the old IMAP server would often fail to show me new messages in INBOX. That should no longer be necessary with Dovecot… I guess I could also install my own webmail if needed — it just has to be able to manipulate a Maildir.

Get the new VIFF 0.5

VIFF 0.5 has been released — this is the latest and greatest VIFF release so far with preliminary support for pre-processing, an actively secure multiplication protocol (it is only twice as slow as the passively secure protocol!) and much more… Download at

For the next release we are working on improving the documentation by using the excellent Sphinx documentation generator. This is the same system used by Python itself and I really like the way it encourages us to write documentation for humans instead of computers. By that I mean that the documentation is more free-form and so you automatically stop writing stuff like

@param share_a: The first share.
@type share_a: Share

which adds nothing to the documentation, except filling out some blanks in the generated API doc.

If you have any ideas for VIFF, then please let us hear! We would love to see more people use it — please tell us what is good and what is bad about the framework.

Lots of releases

A lot is happening in the Python world right now:

  • VIFF 0.4 was released two weeks ago. It has the usual number of bug fixes and introduces new asymmetric commands. This means that you can write MPC programs where only a subset of the players give or receive output.

  • Mercurial 1.0 has been released! I have been a happy Mercurial user for about a year now, and I must say that I’m very impressed with the speed and flexibility it offers. Subversion was simpler and cleaner than CVS, and Mercurial is even simpler, yet more powerfull. I strongly suggest that you consider using Mercurial for your next project, it is already used by large players like Mozilla and Solaris.

  • Twisted 8.0 is also freshly released. We are using Twisted in VIFF, and I have been very happy with it. The Twisted code has this nice property that when you read it you realize that there is no magic going on and that things are generally written the way you would expect them to be.

Happy hacking!

VIFF version 0.3

Tonight I released the third major version of VIFF: version 0.3. This version introduces secure communication using Transport Layer Security (TLS, the successor to SSL). This is quite cool, for that means that we can finally judge the performance in a realistic setting with all the network overhead. My initial benchmarking seems to indicate that TLS gives no or only a very little slowdown.

You can download VIFF from