Interview With KDE-PIM Hacker Till Adam

Till Adam only started hacking on the KDE mail client, KMail, because he wanted some features implemented from the command-line client Mutt. Now he is one of the main developers of the KDE-PIM project, which KMail is part of. KDE Dot News caught up with Till to talk about e-mail protocols, groupware and implementing them in KDE.

Coming from Laboe, Germany, he is due to visit the NL-PIM meeting scheduled for May 27-29 in Annahoeve. During this meeting, sponsored by NLnet, Trolltech, NLUUG and many more sponsors, he will work with his fellow developers on improving KDE's PIM applications and framework. During his stay in the Netherlands for NL-PIM he has found the time to do a presentation at the NLUUG Spring Conference on using IMAP servers as groupware storage and, together with Kopete developer Will Stephenson, he will give a lecture at HollandOpen.

There is also a Dutch version of this interview.

You've done a lot of work on the IMAP protocol in KMail. What are the main differences between this and POP3 which most people still use to get their email?

Till Adam

POP3 is a protocol for fetching mails from a server to a client. It's then normally stored on the client and removed from the server. Like a real-life mailbox, the server holds the mail until the user fetches it. Whereas with IMAP the mail is stored on the server and clients only retrieve the mails they need when opening them. The client only stores caches not the data itself, it queries the server for everything.

KMail also has support for disconnected IMAP. What's the difference in operation between IMAP and disconnected IMAP?

With disconnected IMAP you have a full copy of the mail stored on the client. You're able to carry it around with you and access it without a network connection to the server.

Could you say that disconnected IMAP is using IMAP as if it were a POP3 server?

No, it's very different. Disconnected IMAP synchronises what is on the server with the client, but the stuff stays on the server. So now both the client and server have identical copies of the data. Then you can change stuff locally without a connection to the server such as deleting mails or moving mails between folders. When you are connected to the server KMail can synchronise again and all your changes are uploaded to the server. All changes on the server that happened in the mean time, such as new mail arriving, are synchronised with the client. After the synchronisation, both client and server are identical again. This is unlike POP3, where you get your mail once, and then delete it from the server.

How did you get interested in the IMAP protocol?

I had been using it for a while since I'm running my own mail/web server at a colo facility with a friend. I wanted to be able to access my mail from several clients and machines - IMAP nicely provides that, and it can even do it in a secure manner with SSL/TLS support. Support for IMAP was a fairly recent addition to KMail at that point and it was not really mature yet. Carsten Burghardt (who still maintains kio_imap4 and works on KMail as well) had done a great job of getting it into shape already but there was still much to be done. KMail was not in a very good shape overall at the time. In particular, it had stability issues. I'm not claiming I cleaned that up all by myself, but I do think I brought some new energy into the team and KMail has nicely improved in the last few years. The attitude towards it, even inside KDE, changed quite a bit. It went from "our crashy mail app" to "KMail, it works well and reliably".

How important is IMAP for an enterprise mail application - and especially for a complete groupware solution?

IMAP is crucial in the enterprise context. IMAP is what is used on corporate desktops. POP is mostly only used for road warriors, if that. It makes sense for enterprises to have the data on the server. It can be centrally backed up, maintained, filtered, etc. So decent IMAP support and performance is rather important.

What difficulties did you have in implementing KMail's IMAP support?

IMAP is not terribly easy to implement because it's a very powerful protocol. In KDE we have this idea of encapsulating protocols in KIO slaves [this makes it possible to access different protocols in a generic way from many different applications - ed]. This works very well for protocols like HTTP where all transactions can be described as "request something by name, wait, get it, do something with it". It's all nicely serialised and maps nicely to file operations: things like open, close, read, write and delete. But the IMAP model is different. You basically just bombard the server with requests, in no particular order. And then you get a bunch of responses back, as quickly as the server can give them to you, again in no particular order. This can be very fast. But to use IMAP as a KIO Slave we had to serialise all operations. So it's: fire a request, wait for the response, do something with it, fire the next request, wait for the response, etc. That makes our IMAP implementation inherently limited, speed wise. Getting it to perform nicely was therefore somewhat of a challenge.

It seems to be very fast in KMail's most recent release, 1.8 included with KDE 3.4.

Well, we (George Staikos, myself and also Carsten) spent quite a bit of time profiling and optimising it leading up to 1.8. Disconnected IMAP is especially a lot faster, but online IMAP is improved as well. I think there's always room for more improvements.

Featurewise, KMail is doing quite alright these days. We have some nice features, such as server side searches, annotation support, ACL support, etc. Some of that came from the Kolab and Aegypten projects.

What do you think of the rise of so-called 'rich' web clients, Google's Gmail comes to mind? Do you think they can ever replace traditional mail clients? Do you see them as competition for KMail?

I think it's partly a different use case. For certain workflows, those interfaces are just fine, even superior to something like KMail in some ways. Much cleaner, less clutter, fewer options, and a nice interface. But Gmail has just a very small fraction of the feature set of KMail, Evolution or other full email clients. So I think there will always be users who just need or want that power and flexibility. Of course they are competition. I personally think we can learn a lot from the web interfaces. For non-powerusers many of our interfaces are way too complicated. We hope to change that, though, with the help of the OpenUsability people, for example at the NL-PIM meeting. Ellen and Jan from OpenUsability will be there to work on that with us. So I'd say Gmail is a challenge, in some ways, but we are not afraid of dying out just yet.

Do you think that a more slick user interface is important for enterprise usage as well? Would you consider having separate 'modes' in KMail for simple and advanced usage?

Yes, enterprise users have different needs, but we could do better interface wise there as well. As for having a 'guru mode', as I like to call it, I don't really know. It would feel like giving up to me to have two user interfaces. Some days I think it's the way to go, but I still have that lingering hope that we can actually make interfaces that are straight forward for casual users but still powerful and discoverable so that people can grow with it and explore things when they need more features. Let's see if the closer cooperation with some excellent usability experts helps us achieve that. It's great to have them involved now.

Has all the activity around Kontact, KDE's PIM solution which includes KMail, help to improve KMail?

Of course. The individual parts of Kontact benefit from each other, and it also brought us some new users: people who want an integrated solution rather than standalone applications for email, calendaring, address book, etc. There are many ways in which the individual applications interact nowadays. Things like KMail using KOrganizer functionality for rendering invitations inside emails.

It's incredible that all these applications work so well together - Kontact really looks like one application. Were there any difficulties in getting the applications that make up Kontact to get along?

Well, there were technical challenges, sure. But since the KDE-PIM developers have a very good team spirit, and get along great, it was just a matter of sitting down and agreeing on how to do things. I think most developers involved with KMail or KOrganizer and the other applications consider themselves KDE-PIM developers these days. The applications are just focus areas if you like.

Has KDE's famously rich development environment, with its existing libraries, infrastructure, documentation etc., helped you in the integration of KMail into Kontact?

Oh yes, majorly. Without our powerfull infrastructure it would not be possible to get the level of functionality and integration with the very limited resources we have, in terms of manpower. It's sometimes astonishingly easy to get things done with the tools we have available. Of course we've also hit limits of the current technologies, but that's good, that means we can improve them. This in turn benefits all of KDE. We are proud of the fact that we've managed to maintain the ability to run all applications standalone without losing any of the integration. You can have KAddressbook and KMail running side by side and they will still interact just as well.

The well-documented code might also help companies to make local customisations to Kontact.

We try to provide extension possibilities, so third parties can do their own Kontact parts, for example. If you want to plug your third party application into Kontact and have it interact with the other parts, you can do that. Or if you want to display calendar info from a custom database or an application exporting its stuff via SOAP, you can do that as well, by writing a calendar resource.

Do you feel that KMail or Kontact is best combined with Kolab as a sister project to KDE?

It's the server project that has worked with the KDE folks the closest and it's probably the one which works best with Kontact. This is because Kolab as a project has people with KDE ties involved. Of the three companies behind Kolab, Intevation, Erfrakon and KDAB, two are run by KDE people, and KDAB employs several KDE and KDE-PIM developers. So naturally Kolab is very nicely integrated.

I also feel that it's important that the newer versions of Kolab allow users of different platforms (Outlook on Windows) to save their data in the same format as KMail & friends, making a slow transition to KDE a lot easier.

Yes, that's an important aspect of the Kolab concept. If we hope to offer people ways out of the Outlook trap a server solution will have to be part of it, and one which allows people to transition step by step. We can of course never catch up with Outlook when it comes to Exchange support.

You mentioned that having some commercial support has helped KMail. Do you think it would be useful for companies to invest in more fulltime KDE hackers?

Of course having people like David Faure hacking fulltime on KDE-PIM for a while is a great thing. The libraries and applications have benefited greatly from that. I do feel that companies would do well to sponsor development of the things they need. My motivation to add enterprise features I don't personally need in my spare time is rather low, which is probably understandable. But if I can do it during paid time, I'll be more than happy to; it's a code base I like to work with, after all. And I do enjoy seeing KMail improve, of course. I've also spent quite a bit of time on KMail and KDE-PIM by virtue of working on Kolab support for KDAB. KDAB will of course gladly implement whatever people need (shameless plug: at reasonable rates, e-mail info@kdab).

Q: What do you feel are the advantages of using a KMail/Kontact/Kolab setup above a MS Outlook/Exchange setup?

Freedom, above all. There are technical benefits as well, of course, but they are maybe less important. While Exchange is more feature-rich in some areas, for very many setups and needs, Kolab is more than sufficient. A lot of development has gone into Exchange of course, and they do get some things right. But for the vast majority of users many of the features of Exchange are useless. Kolab will be a better fit there, I'm sure, and at a price that can't be beat. And that's not even talking about the vastly smaller hardware and infrastructure needs.

In what areas would you like to see improvements to the KDE family?

Exchange's web interface is pretty good, I think. Horde is a web interface that can be used with Kolab and is getting there, but it's not on the same level yet. Of course, Kolab is vastly more platform independent - Exchange's web interface only works fully in Internet Explorer. In general, I think we still have to improve when it comes to user interface polish. Outlook has many little convenient details.

Sounds like there's still something to do at the NL-PIM meeting. We wish Till and all the other people that will visit NL-PIM good luck and a good time in the Netherlands!

Dot Categories: 

Comments

by a.c. (not verified)

Kolab needs to be split into into 2 groups. The intention of Kolab is to develop a server that is easy to set-up and can serve 1000s. Cool. Except in some ways, these are competiting against each other.

It would be nice to see the parts be swapable. In particular, it would be nice to see a simplified Kolab run on top of Postfix using flatfiles (/var/spool/mail) rather than ldap. Likewise, it should use the standard imap/pop to work with this.
and you ask why?

For the home to the small business. Speed is not the big need there, but simplicity is.

It needs to be something that is easy to use. Of course, now, that Kolab has moved away from using imap to using ldap only, it would mean that imap would have to work against /var/spool/mail files, and I do not know if that is easy. IOW, it removes the very ease that version 1 had.

by am (not verified)

"In particular, it would be nice to see a simplified Kolab run on top of Postfix using flatfiles (/var/spool/mail) rather than ldap"

I believe Kolab uses Cyrus-IMAP (fastest IMAP server around) for mail storage and only stores user info in ldap.

I think that Kolab needs a much simpler setup then it has now. It just takes too much effort to install. It needs more distro support to really take off.

by Till Adam (not verified)

Have you tried Kolab2? It's rather easy to install, we think, there's a script that basically does all the work for you. But we'll work with distributors to make sure it can be nicely installed on top of popular distros as well.

by am (not verified)

The question is... is it easy to maintain/upgrade after installation. Installing from source usually isn't the best maintenance path for large groups of server.

What about adding MailScanner as an optional package for virus/spam filtering?

I love kolab by the way, I tried to get my organization to switch to it. Currently we use Sendmail/Cyrus/OpenLdap/Kerberos/MailScanner (hell this is pratically Kolab :) ) for all of our big setups. (all have rpms with good support). but without support from a big distro Kolab was always a no go. :(

Keep up the good work! I am looking forward to switching when the time comes around!

by Till Adam (not verified)

Since Kolab relies on OpenPkg, it's very easy to install updates and fixes. As for feature requests, please send them to https://kolab.org/mailman/listinfo/kolab-devel or https://kolab.org/mailman/listinfo/kolab-users, where they can be discussed more appropriately. The parts that come from the OS are supported by the OS, the rest is supported by OpenPkg, where Kolab uses the unmodified packages, or the Kolab project where they are changed or extended.

by Andy (not verified)

I blelieve Kolab to be rather easy installable. I put the /kolab tree onto a own disk and can use it, no matter i run Suse or Fedora, even from inside of Xen. I like this separation very much, especially when dist upgrading, since i can revert to the old version without touching the precious data stored in Kolab.
Integrating it into different distributions in different ways (or even the same ways) would open it to the same vulnerabilities as every other package, different patchsets by different distributors and so on.
The main advantage of Kolab is its own tree, everything else doesn't matter. With the Linux ELF-module it should without changes be useable on *BSD too I guess.

by mdouhan (not verified)

Kolab runs perfect on FreeBSd, and has been prooven to scale very well in our experience, from small installations of 20 users up to serious enterprise solutions serving several thousand users and still growing

by anonymous (not verified)

Please don't forget Slackware, thank you!

by Richard (not verified)

Please, check out this
http://wiki.kolab.org/index.php/Kolab2_linux_distribution_integration wiki
page. In some cases the kolab code contains hardcoded values, and this need
to be tackled, before distributions can integrate a unpatched kolab.
Some hard work is being done to replace the hardcoded values, see:
http://www.kolab.org/pipermail/kolab-devel/2005-May/003441.html
As usual any help with this is welcome!

And oh, the cyrus imap patches should be submitted upstream so the
distributions can disitribute a unpatched cyrus. At the end I think this
may be the biggest showstopper to integrate kolab into a distribution.
See http://wiki.kolab.org/index.php/Kolab-major-app-patches

by Kevin Krammer (not verified)

> Of the three companies behind Kolab, Intevation, Erfrakon and KDAB, two are run by KDE people, and KDAB employs several KDE and KDE-PIM developers

I am sorry if I misinterpreted this, but it sounded like KDAB was the company not run by KDE people, while I think it is _the_ company run by KDE people :)

by Till Adam (not verified)

Indeed. Klarälvdalens Datakonsult AB, Sweden (KDAB) is run by Kalle Dalheimer and full of KDE folk. My apologies if that was unclear. The company not run by KDE folk is Intevation, but these guys are Free Software to the bone as well, just to make that perfectly clear.

by Tormak (not verified)

You last name Leeuwen is very close to mine Lee Wen, except mine comes from Chinese (cantonese) :)

by avleeuwen (not verified)

Thanks :). It means 'Lions' in Dutch.

by Gerd (not verified)

Also a city name Louvaine/Leuwen

..and then optimized for this 'use case'.

When reading this in the interview this truck me as kind of odd that IMAP needed to be put down to the 'serial' limitation of the KIO Slaves framework and be optinmized for that instead the latter being extended to allow for non-serial/random communication as well. Was this design decision based on some pragmatical reasons unknown to me, or was it just necessity to keep binary compatibility or some such?

You are right, it would be better to extend KIO to better deal with this kind of protocol and we plan to keep these things in mind towards KDE 4. As usual in KDE the people involved where concerned about making it work first, then making it fast, and then making sure that the infrastructure is improved so that next time things can be done better. So the reasons are historic and pragmatic, yes.

by Matthias (not verified)

IMHO their is a growing need to ensure syncronisation with PDAs and Cell Phones and I do not see there real advances. Sync with Palms is working but the main PDA platform today is WindowsCE based PocketPC and IMHO we have to accept that no Open Platform will replace it at least not in the foreseeable future. I tried to get some information about the current status of kitchensync but it is at least my impression that this project is more or less on hold. So I would appreciate very much if the efforts in this area could be strengthened - although I have to admit that it is a tough work having the number of different devices in mind.

Best wishes
Matthias

by Till Adam (not verified)

> So I would appreciate very much if the efforts
> in this area could be strengthened

The best way to ensure that, is usually to get involved yourself. This is Free Software, after all. :)

Till

by Simon Kenyon (not verified)

i use kmail and now kontact for all my email and have done for many years now. i really like it. however (and this really is not a troll) i recently had to read my email under windows xp. so i fired up outlook express and configured it. the speed difference was striking. this was on my laptop (which can dual boot). so i know there was no difference in hardware.

this was remotely from my email server.
i tried disconnected IMAP again after i upgraded to 3.4 and while it had improved to the point of being usable, syncing remotely was too slow and i reverted back to ordinary IMAP.

just wondering what express is doing differently that makes it so much faster? has anyone analysed the server usage of various email clients to see if any improvements can be gleaned from that

by Carsten (not verified)

Outlook (Express) can use it's own imap library which is of course optimized for this protocol. KMail has to use the kio infrastructure which is - as Till explained - not really made for imap.

by Joergen Ramskov (not verified)

First, thanks for a very nice interview!

I'm currently using Mozilla Thunderbird as a mail client (it's what I used before switching from Windows to Linux and KDE recently) and I quite like it. Since I'm using KDE I try to use as many KDE apps as possible, but the one thing that has kept me using Thunderbird is the baysian spam filtering technology that's integrated into the mail client and works great. I haven't found anything similar in KMail, but if there is another easy way to combat spam combined with KMail, I wouldn't mind knowing :)

Another issue that is touched upon above is the problem with sync'ing your mobile phone/pda with your PC. I live without it, but it is one area I wish was working since I more or less uses my phones calendar function to keep myself organised :)

Of course, that means I should get involved, but I just have to learn C++ first, so that is not going to happen for a while yet :( I did the next best thing a while ago though: I made a donation :)

Keep up the good work KDE folks!

by Jonas (not verified)

KMail has a plugin system, it is not tied to one spam filter. Just install the filter you want to use and enable it in the KMail configuration.

by Joergen Ramskov (not verified)

Ahh..I didn't know that, thanks for the info.

What filter do you recommend? I'm running Kubuntu and seem to have Bogofilter and GMX installed currently.

by Lane (not verified)

> KMail has a plugin system, it is not tied to one spam filter. Just install the filter you want to use and enable it in the KMail configuration.

This is actually one of my biggest problems with KMail.

My suggestion is to include a basic, quick, spam filter with KMail. Allow users to configure it via the normal Configure KMail settings. Here's why:

Spam filtering is a basic and important function in a modern email program. While I understand the desire to allow users to customize their spam filter, I think it's poor design to require the user to install an external filter.

Another problem is speed. I've tried a couple of the filters (spamassassin and bogofilter, I believe). They took forever. Compare this with Apple Mail. Excellent filtering built in to the app, with no noticable speed hit.

The last problem is configuration. It makes some sense to have a wizard to help users configure spam filters. But it makes much more sense to contain spam settings in the standard Configure KMail settings. I shouldn't have to go through the wizard every time I want to change my spam settings.

A last note - for advanced configs, it would be nice to allow for external spam filtering. For example, my email server runs spamassasin and tags my mail. It would be nice if I could configure KMail to honor the spamassassin tags without having to re-run spamassassin.

Anyway... enough ranting. I do love Kontact and KMail. I really appreciate all of the work that the KDE devs have put into it!

I suppose I should probably take these comments and put them into the bugzilla...

Cheers,
Lane

by Eike Hein (not verified)

I second that.