The Road to KDE 4: KDE PIM Libraries and Related Technologies

KDE has a number of sub-projects that have blossomed into enormous projects of their own. A number of them, such as KOffice, or KDE-Edu get a lot of press in the open source world, while the KDE PIM project has been quietly gaining corporate acceptance as a suitable enterprise suite. Today's feature are the libraries that power the KDE PIM project, and specifically, what changes have taken place since KDE 3.5.x, wherein the KDE PIM project is one of the most successful and stable components of KDE. Read on for more details.

The KDE PIM (Personal Information Management) team has stability among their primary goals. As a result, the changes to KDE PIM are not as flashy as some of the other technologies going into the KDE 4 applications, and in fact, many of the KDE 4 PIM applications will initially appear to be direct ports of their KDE 3.x counterparts. However, much of the focus at this point has been future-proofing technologies, since the opportunity to introduce new APIs and break old ones does not come around very often. To that end, a number of new libraries have been developed, and old ones reworked.

Akonadi

Akonadi is KDE's new storage backend, named after the oracle goddess of justice in Ghana. It's not really a backend, rather a generic backend API, which can have any number of real storage solutions implemented, from flat files to complex groupware servers. It supports caching, so offline modes should be implicitly available regardless of how the data is actually stored and retrieved.

It is considered to be the replacement for the existing resources framework in KDE, but is also extensible to data types that were not available to KDE 3.x. For example, the Akonadi framework will obviously be useful for email, contacts, and calendars just as the old frameworks were, however it can be extended to a number of other useful types as well. Volker Krause, a prominent developer of Akonadi, suggests that Instant Messaging (IM) logs may be useful to support, as it could just as easily query your hard disk for stored logs as it could retrieve the logs from the browser-based GoogleTalk. Of course, being cached, it would still be able to query these logs without going directly through GoogleMail every time.

Additionally, Volker writes that "Akonadi uses a completely language/toolkit independent interface, based on D-Bus and an IMAP-like protocol, making it possible to use it from non-KDE software without linking to any KDE/Qt libs." Which means that you can write lightweight programs that take advantage of Akonadi without having to link to the whole set of KDE and Qt libraries. This D-Bus control should also make access to the data stored in Akonadi trivial for writing scripts for automation and similar tasks.

That said, there are parts of Akonadi that integrate very well with Qt and KDE. In particular, it has support for very smart copy/paste and drag/drop for applications that want to deal with data stored in Akonadi. If you drag the name of a contact, for example, into a text file, you'll get their name since that's the only format that a text file can understand. If you drag it into a calendar, the calendar can easily recognize that it is accepting a drop of a contact, and could conceivably ask you if you'd like to make an appointment with this contact, etc. The libraries support this level of integration, but it may take a while for the applications to take full advantage of it.

For storage, Akonadi is only limited to those plugins that have already been programmed. The default will be to store most data in flat files, much as it is done now, but handle things like the relationships between the data using an SQL database backend or similar. In the past, KMail, KNode, etc. each had to do their own implementations of the data relationships, often in an application-specific binary format. This data will now be able to be shared more easily. Additionally, Akonadi provides "persistent unique identifiers for every item as well as change notifications" which are required in order to implement effective metadata searching, such as that being provided by the NEPOMUK-KDE integration. The metadata indexing isn't done by Akonadi itself, but rather, it is written in such as way that it becomes trivial to hook an existing system into whatever storage backend is active.

Since Akonadi itself is "type neutral", additional libraries have to be implemented for each type supported. In some cases, these libraries have been rewritten from existing KDE 3.x libraries (contacts, for example), and in other cases, the existing libraries should work with a little coaxing (KCal, KMIME, etc.).

However, I will once again stress that while this library will be shipped with KDE 4.0, the KDE PIM applications will not necessarily have been fully adapted to take advantage of the new functionality. At this point it is simply laying the groundwork and ensuring that the API is complete so that it doesn't break between future KDE 4.x versions. I would consider the development of Akonadi to be the biggest change taking place in the KDE PIM circles, and probably the most important for the long term forward progress of KDE PIM.

Khalkhi

Khalkhi is designed to be the new contacts framework for KDE 4, which happens to be the Georgian word for "people" (it is pronounced in a way that I cannot properly describe using English, but Friedrich has attempted to do just that in his blog, which also happens to be a nice introduction to the technology). It features a number of new things that the existing KABC cannot do, for example:

  • It has a concept of groups, and relationships between people. People can belong to groups. Groups can belong to other groups. NEPOMUK may find the relationships between people to be useful as part of its searching and indexing.
  • Share properties between groups/persons. You can change the contact info for many people at once if you define them as sharing a mailing address, for example.
  • Plugin-based property types. You are no longer restricted only to the contact information that is hard-coded into KABC, such as email addresses. You can add contact information for new types simply by adding a plugin. For example, one could add fields corresponding to the user accounts at a number of online websites (such as Flickr photo sharing, or Last.fm music profiles) which would be automatically available to all programs using contacts.
  • Incremental Updates. In KABC, a program had to reload the entire contact list any time there was a change, which becomes inefficient especially for large contact lists. Khalkhi can do these changes incrementally, without reloading.

Khalkhi is a little behind the other libraries as far as integration goes, and will probably not be visible before KDE 4.1. If you are interested in this sort of technology, you can contact Friedrich Kossebau directly, or drop into the #kontact channel on IRC and lend a hand.

KitchenSync

No name change, though pretty much a new project. According to Cornelius Schumacher, "It's called KitchenSync. It doesn't share more than the name with the KDE 3 KitchenSync though."

KitchenSync used to be KDE's syncing library and GUI, which implemented ways to get data from various PDA's and similar devices. KitchenSync is now a framework that sits on top of the OpenSync infrastructure complete with ported UI from the old program. Plugins should be much more stable and maintained now that they live in the OpenSync project. This will benefit KDE and other projects in much the same way as the abstraction of the SANE libs did for scanners.

KDE PIM developer Tobias Koenig says that "now the plugins [are] coming from the OpenSync project, so we have a broader and better tested set of plugins. The only plugin which needs porting to Akonadi is the OpenSync kdepim-sync plugin. However that will be done [shortly]". OpenSync is GUI independent, and simply deals with the communications to and from the device, which means we still need dialogs to configure this connection. Fortunately, he writes: "To make the implementation easier, a current [Google Summer of Code] project is about creating abstract descriptions (XSD) for the plugin configuration settings, so that configuration dialogs can be created automatically."

KitchenSync is already functioning quite well for KDE 4, and with a little polish, should be ready for 4.0.

Mail Transport

This is a new library for KDE 4.0. It is essentially an integration library that shares email settings in a far superior way to KDE 3.x. In KDE 3.x, you had to set up your mail account in every application separately. Volker Krause describes it as: "A small library which takes care of configuring and sending mails. The really nice thing here is that it is used by KMail, KNode and Mailody and allows them to share the same settings (including live updates if you change them in one app, etc.). Something similar is planned for identities (might not be ready for KDE 4.0 though)."

This library should allow easier integration of email services into other applications, without each application having to be aware of how the email is being sent. It will show up in existing KDE applications for KDE 4.0, but the real strength here is for third party applications.

Syndication

This new library takes the hassle out of handling a number of syndicated news formats, and presents an API for applications to get and use news feeds. It currently has support for the commonly used formats, Atom, RSS and RDF. Adding additional formats in the future are not especially difficult, and programs using syndication would automatically be aware of them. Many of the PIM developers consider this to be one of the most exciting additions to kdepimlibs for KDE 4.0.

The Wrap Up

There is a lot of new work going into the PIM Libraries for KDE 4.x, some of which won't be ready in time for 4.0. Expect that many of the KDE PIM applications are not yet updated for the new libraries when KDE 4.0 is released, but will still be using ported versions of the 3.x libraries. As I mentioned before, one of the main goals for KDE PIM is stability, and by using the older libraries for the 4.0 release, you can be assured that you will at least have a working PIM environment that is functionally equivalent to the 3.x applications. However, expect new and exciting things to come from KDE PIM beyond KDE 4.0. Also, as the KDE 4.0 release draws nearer, expect an article focusing on the PIM applications.

Lastly, I will leave you folks with a bit of gossip to chew on. This will be the last Road to KDE 4 article fully hosted by the dot as I am moving it to a new source where it will give KDE wider exposure within the tech world. I cannot reveal all of the details yet, but I will confirm that these articles will still be linked from the Dot, which is good since it has a fully open, unmoderated comments forum, and many of these articles have generated a great deal of constructive feedback for the developers. The next article will be on either Krita or Kate (haven't decided which one will go first).

Until next time...

Comments

by Troy Unrau (not verified)

This article was waiting in the dot queue for a short while being edited. While that was happening, events were set in motion and I can now reveal that this series will be moving to Ars Technica for future versions. Like I said, you'll still be able to find a link on the dot, but KDE will get wider exposure as a result. I'll also be covering a number of non-KDE topics while writing for them, but my personal focus will remain KDE.

Cheers folks.

by Andreas (not verified)

good

by Aaron J. Seigo (not verified)

very cool news, troy =) congrats and all the best with your new post (excuse the pun ;). can't wait to see more kde articles on other sites ....

by Navindra Umanee (not verified)

Fantastic news. Ars Technica is one of my favorite news sites... unfortunately their Linux desktop coverage hardly ever touch on KDE.

by link? (not verified)

Can we get a link to this Ars Technica site? I've never heard of it :) Thanks yall :)

by Vincent (not verified)

Hey...
I was hoping that there will be some kind of program that will help getting the PIM-data synchronized between serveral PCs.

by Adrian Baugh (not verified)

Isn't there an OpenSync plugin to do just this? In which case you'll be able to access it via KitchenSync.

by Tom Deblauwe (not verified)

Thanks for the overview, but it's kinda hard to understand how it all will come together. Maybe a simple graphical overview would be nice to see where each part is situated?

Thanks a lot for the series, keep it up!
Best regards,
Tom

by Troy Unrau (not verified)

You know, I was thinking about that as well, but the problem is that these technologies have a lot of 'technobabble' assocated with them that would make a really complex, but useless flow chart for most users. Anyway, Akonadi is the kingpin of the new tech, as things like khalkhi, nepomuk, etc. will be latching into it, and KMail and so forth will likely be storing their emails and such via Akonadi.

I try to rotate between libs and apps in my articles, with some eye candy things thrown in between them, so this is my libs article :)

by superstoned (not verified)

On the Akonadi page at the pim.kde.org site (http://pim.kde.org/akonadi/) you have a graphic: http://pim.kde.org/akonadi/architecture.png

by Pharaoh (not verified)

Completely offtopic, but how was that graphic created?

I am trying to create a radial chart for an astronomy application, and was wondering if there's an easy way to programmatically create such an image.

Thanks!

by Matt (not verified)

I would guess that it was done in Kivio. :)

by Jakob Petsovits (not verified)

At the last aKademy, one of the kdepim-developers had the tool for this graphic running in his presentation and he modified the graphic on the fly. It seems that it's an unreleased mini-program, and it had some strange German first name as name of the app. The likes of Nepomuk, you know. But I don't remember the exact title, sorry.

by birdy (not verified)

Too bad kdepim lacks developers. This "sub-project" really deserves more active contributors.

see
https://bugs.kde.org/show_bug.cgi?id=110350
I badly need this structure, and many people I know do.

I just read the bug report, and the idea of arbitraty data fields is a reasonable thing, and part of the reason that Khalkhi exists. However, the forms you suggest might take a little extra work - they appear to be a key & string storage system, similar to a dictionary or similar in python, which is really easy to implement in a simple sql table. The real question then becomes how to make the user interface something reasonable while still maintaining much of the existing functionality.

You can try hanging out in #kontact to see if anyone is interested. Try making more involved mockups and so forth if you cannot code. Just remember, almost everyone in there is a volunteer :)

Cheers

by Kevin Colyer (not verified)

Well done Troy for all these articles. You write well and with great (deserved) enthusiasm for KDE - 4.x will be great! Thanks!

by alsuren (not verified)

I vote kate for next post :D

by Kevin Kofler (not verified)

The move to OpenSync actually also happened on the 3.5 branch and 3.5.7 uses OpenSync instead of its custom plugin framework.

by KDERockandroller (not verified)

I am passionate about KDE, but have to disagree that KDEPIM is a mature enterprise-ready set of applications. I have been testing them for years, currently on Suse 10.2 and they leave a lot to be desired.

Kmail is unusable for IMAP in its current incarnation. It has lots of nice features, some of them are even innovative, but it is unable to handle a 100-200MB IMAP account. It quickly becomes unbearably slow.

Integration and syncing with egrouwpare is also an area that needs lots of work. Egroupware is, in my mind, the easieset to install groupware server around and very feature complete. Kontact promises egroupware support but doesn't deliver: double entries for appointments and kontacts, and the syncing simply doesn't work.

You may say that all of these complaints belong in a bug report. I believe that bug reports are intended for things that are not immediately obvious upon fifteen minutes of using an application.

I have high hopes for KDEPIM (I am stubborn) and expect that one day it will join the large number of KDE outstanding apps (k3b, amarok, konqueror, konsole, tellico, kdissert, etc) but right now it is simply unsusable for me.

I understand that there might be a lack of manpower to really fix these issues that I mention, but the truth is that a properly working PIM suite is more important than Plasmoids to end users.

Thanks and sorry if this is a bit rantish. Looking forward to Akonadi and hoping that a wonderful KDEPIM finally materializes.

by Louis (not verified)

There's an old saying; "One man's trash is another man's treasure." (Not to say that you called it trash) At the company I work for, we use KMail to collect mail from more than twenty POP3 accounts, filter it with rules, and redirect it to more than 100 people in a dozen or more groups. The machine that I have KMail doing this on runs 24/7 and flawlessly handles thousands of messages daily. The redirect feature is very cool as it preserves the address of the original sender so that the recipient can use his/her reply button as expected. It's saving me countless hours of headache and, I'm sure, lots of money. Also, I use Kontact daily for my POP3, Contact, and Calendaring needs, and have yet to find a showstopping problem. I won't argue with the IMAP deficiencies that you pointed out, but KDE PIM is definitely viable for (non-IMAP) business use.

by Iñaki Baz (not verified)

> but KDE PIM is definitely viable for (non-IMAP) business use.

The point here is that lots of companies use IMAP for their mail, not POP.

by Paul Eggleton (not verified)

> I believe that bug reports are intended for things that are
> not immediately obvious upon fifteen minutes of using an application.

I disagree. Anything that's wrong with a piece of software is a good candidate for a bug report. If you have better real-world experience of an issue than the developers of a piece of software then you should feel free to speak out, but most importantly you should speak out in the way that is most helpful to the development of the software, and that is to file a descriptive bug report.

Filing a bug report means that the issue is tracked - developers are a bit less likely to forget about the issue; if the developer in charge of fixing the issue moves on, it is easy for someone else to pick up where they left off and see easily where work is required; plus for new/prospective users it's easy to get a broad overview of problems that might be encountered before running into them first-hand.

by jospoortvliet (not verified)

Also, a huge amount of time and resources has been spend on bugfixing IMAP and other stuff lately... KDE 3.5.7 should do much better on IMAP.

by Aaron J. Seigo (not verified)

> PIM suite is more important than Plasmoids to end users

not sure i see the connection between those two things, tbh. there are a lot of things that need to be worked on ...

i agree that there are problems with the imap support in kde3; still it's more than adequate for many, while improving imap is one of the reasons for akonadi and its design from what i recall.

as for egroupware, maybe the egroupware folk could get more involved with what is one of the most used free software groupware apps and get themselves a few more users.

and of course, "unusable for you" doesn't mean it's not very ready for others =)

> Kontact promises egroupware support but doesn't deliver

your experience differs from mine here.

by ferdinand (not verified)

I use KMail on Opensuse 10.2 AMD 64

The mails are stored on a SUSE OPENEXCHANGE Server 4 and the imap accounts contain some 20.000 mails with a local mailbox size of some 50MB.

Local network connections are fast, remote connections - using a 2 MBit line - the startup of kmail is a bit slower but far away from being a noticeable problem.

About 1500 contacts are stored in a Kolab server using a slower (512) upload line. No problems either.

The only thing which is really much to slow is the "find message" functionality (not to be confused with the "search" function which only searches in some fields)

by jms (not verified)

For me the more serious problem with Kmail is the impossibility to reply to html formatted mails without destroying the format (kmail badly convert the mail into text).
This enhancement has been ask for years now. The problem is not only a lack of man power for but also philosophical one. There are many opponents against html formatted mail.
However html mail *are* used everyday by enterprises.
*All* my co-workers (thousands of them) use Outlook as mail client.
For example, consider the common situation where mails are sent to a group, each member of the group uses a different color in their reply. Its helps identifying who edited each part of the original message.
If I reply to their mails, I mess up everything with my reply.

I'm sure I'm not the only one in this situation, but I may be one of the last to stick to kmail/kontact hoping that this will be solved some day. I simply hope this day will come before my boss force me to ditch kmail (and linux).

by Robert (not verified)

I totally agree with you. The problems you point out are not the only ones, though: There is a bug about multipart/related HTML mail display filed since ages and none of the contributors have found the time / seen the need to implement it. So, HTML mails are only displayed correctly if all images are referenced externally, and since those are viewed as "evil" because some of them are web bugs tracking your viewing of the mail, this perception of "evilness" has somehow leaked through to HTML display in general, which is why it recieves so little attention, IMHO.

by cossidhon (not verified)

Couldn't agree with you more! On top of that, there's also the very long request for rich-text sig's. My employer demands a corporate sig that can only be done using rich-text. I though it made it in 3.5.7, but I can't find it, so I guess it's not in. To bad.

by Iñaki Baz (not verified)

I'm 100% agree with you: KDE PIM is not enterprise-ready. I use Kontact at the work and at home for long time.

At the work I have configured a groupware server (just IMAP folders with contacts, calendar and so).
- Just impossible with IMAP, I must use DIMAP instead or it is inusable at all.
- Every day I start my KDE I get a warning of Kmail about the need of "creating the groupware folders", always!
- I don't have a public key, so I can't sign mails. But I should have teh possibility of encrypting mails with public key of the recipient, but I can't because a bug since 2004-11-17 that nobody fixes:
http://bugs.kde.org/show_bug.cgi?id=93436
- I tried to use a webdav calendar, but most time I added a task Kontact crashed. Could be related to this bug:
http://bugs.kde.org/show_bug.cgi?id=108424
- In Korganizer reading the entire subject of a task is impossible because this bug:
http://bugs.kde.org/show_bug.cgi?id=141712
- The mail filters are applied "sometimes".

And of course, kontact crashes and crashes and crashes, at least 3 times each day.

I sincerely think KDE PIM is not suitable for businnes enviroments, or you must avoid IMAP, groupware, coorporative calendar and so.

Anyway I use Kontact because it "could" be great.... some day ;)

by Janez (not verified)

Kmail is unusable for IMAP in its current incarnation. It has lots of nice features, some of them are even innovative, but it is unable to handle a 100-200MB IMAP account. It quickly becomes unbearably slow.

I must strongly disagree as within our company I access mail via IMAP (Courrier IMAP) and my mail account has well over 3.7 GB of mails and access via KMail 1.8.2 that came with KDE 3.4.2 in SUSE 10.0 is perfect. So I guess you must have some other difficulties in accessing your 100MB account.

by Roger F (not verified)

You are completely wrong about the IMAP performance limitation.
I am currently using Kontact / Kmail and have gigabytes of email. 3000 messages in a single folder? No problem - it displays instantly. Searching for

I suspect it is your backend IMAP server. Before we switched to an open source groupware solution that uses Cyrus, displaying folders with lots of email was slow.

If your eGroupware server uses the Courier IMAP server, try switching to Cyrus.

I do think Kmail is under par in some areas though. Editing of rich (HTML) mail leaves a lot to be desired, especially compared to Thunderbird which even has great table editing.

by Matt Meola (not verified)

I'm noticing a common theme with respect to the stability of KMail and IMAP -- it seems those with positive reports are running OpenSUSE.

I'm on Ubuntu (Feisty), and it doesn't seem to like a large IMAP folder. Just hangs, eating 100% of the CPU. This going against either Courier imapd or Dovecot imapd. I may try Cyrus and see if that's any different.

KOrganizer is, IMO, one of the best things about KDE. The way in which it handles todo items is better than any PIM suite I have used.

Kudos to the developers.

by Joshua Lund (not verified)

My Maildir is almost 1 gigabyte and I have no issues whatsoever accessing it through IMAP. I too am running Ubuntu Feisty and I'm really perplexed by these comments criticizing Kmail for poor IMAP performance with large folders.

When exactly does it hang? I would love to help troubleshoot this.

by Matt Meola (not verified)

It typically hangs when clicking on a message. After a while, it sometimes crashes.

by gonzo (not verified)

Same problem here with a disconnected cyrus and Kubuntu. Problems started about 3 weeks ago - I never had any problems before. The local folders are about 1.5 GB

by Daniel (not verified)

Pervasive (system-wide, centralized) contact & messaging management + presence and collaboration services... I believe we are entering the Shangri-La. Good times indeed!

by Aaron J. Seigo (not verified)

agreed; i think it'll take until 4.1 or even 4.2 to really get the most out of these features (and probably even longer before these creative people run out of interesting ideas on top of that =) but the foundation they are building seems very solid... go kdepim! =)

by Joe (not verified)

"the KDE PIM project has been quietly gaining corporate acceptance as a suitable enterprise suite."

Which corporations?

by Anonymous (not verified)

What about Munich?!?

Ok, Munich isn't a corporation but public administrations have similar needs.

by Aaron Seigo (not verified)

i know of at least one fast food company in the u.s. that decided to go with kde because of kontact's ability to lock down users to their home dir for saving attachments so easily. (now -that- is an enterprise feature; i love how nobody whinges about these kinds of things, though, given how important html email evidently is ;)

i also know of companies in calgary here that, while not "enterprise" in size, also use kontact.

so, they do exist.

by Joergen Ramskov (not verified)

For yet another nice article!

It is good to hear that KDE Pim is seeing some work done. I personally use Thunderbird for mail as it has served my needs better, but if Kontact/Kmail improves, I will certainly switch :)

It's also great that KDE is getting more exposure on Arstechnica - Gnome has gotten a lot more exposure there, maybe because of Ubuntu but still. I look forward to your next articles.

by Knutt (not verified)

Using Thunderbird as well. The reason is the html-Thing. I see no problem with using a layout language in a mail. I have a very lovely girlfriend, and she really likes it from time to time to recieve an "i love you" not just in plain text, but in red letters and with a heart-thingy next to it ... women ;-)

So i guess, the no-html-in-email-guys are
1. all male
2. without girlfriend at the time?

by Morty (not verified)

My guess would be more along the lines:
1. The feature are not that important, the developers prioritize things they find more important.
2. They prefer to communicate with their girlfriends by other means than email, see 1.

by Scott (not verified)

I'm sure looking foward to the new address book - KABC is a bit of a pain. One thing i'd like to know is if you will be able to make global changes to the address book, like wether the first or last name comes first, instead of having to change it in every entry.

Scott

by Nuscly (not verified)

Very good work, when all features will be integrated, it will be very powerfull !

I hope it will be easy also to write plugin for PIM application like Kmail to have a lot of plugins available, "a la firefox".

Plugin are easy to contribute and add extra feature.

by Darkelve (not verified)

I was wondering, which new features are planned for Kontact?

by Darkelve (not verified)

Oh, and thanks for the article again, Troy :)

by Darkelve (not verified)

"I was wondering, which new features are planned for Kontact?"

Oops... should have mentioned; I mean the Calendaring part...

by Anon (not verified)

Hello KDE Development Team,

I am interested if KitchenSync will support WM5 Devices.

Known Problems so far:

- SynCE has only WM2003/2002 support and bad WM5 support (with a lot of Python hackage).
- OpenSync has absolutely no support at all, even their Plugins for Evolution etc. doesn't work correctly (knowing that they head forward to 0.30).

I really like to see my HTC Prophet (WM5) simply get connected and synchronized with KDE's PIM.