FEB
22
2007

The Road to KDE 4: CMake, a New Build System for KDE

When a project with the size and scope of KDE gets to be as big as it is, sometimes changing a decision established almost a decade earlier is very difficult. KDE has relied on autotools to build most of the project since its inception, but for the last year, KDE 4 has been building with CMake, a newer build system that is rapidly becoming a heavyweight contender in the world of buildsystems. Read on for more...

This article focuses on CMake, which is not actually part of KDE, but is developed independently by Kitware and a team of open source developers. It is released under a BSD-style license. I can't really show much in the way of screenshots for a build system, but I will do my best to explain why CMake has been a welcome change to the KDE development process.

But before we talk about CMake, a little history lesson about KDE and autotools. KDE has used Qt since the beginning, and one of the nice features of Qt is the meta-object compiler (moc). Autotools had to be extended to support using moc as a pre-processor for most of KDE's header files. That was only the beginning though, as KDE developers wrote the ground-breaking DCOP communication protocol which also had the side effect of adding more types of files that had to be autogenerated during the build process. The developers added document compilers, tools that automatically handled translations, and tools that compiled configuration file classes from XML: Qt's User Interface compiler (for .ui files) was born and needed to be supported by the build system. They also needed to add a whole suite of new configure checks, options, etc. to support all of the features that KDE was using. The KDE build system was evolving into a very complex animal, and using autotools did not scale well.

By the KDE 3 series, there were only a select handful of elite build gurus who could understand the whole of KDE's build system. KDE developers had trouble even moving things from one folder to another without having to spend hours tweaking build files to get it to compile again. And in some cases, starting an independent KDE project meant shipping up to 500 KiB of autotools support files for even a simple 'hello world' type application.

Something obviously had to be done for KDE 4 to improve this situation, and at an Akademy 2005 meeting, it was decided to explore other build system options. Initially, SCons was prototyped for building KDE, but it was slow going, and after several months of work it still did not handle kdelibs well enough to dump autotools. One of the bigger problems with SCons was its lack of modularity.

So, as is often the case with KDE, he who codes, decides: Alexander Neundorf stepped in and did the initial port to CMake which went very smoothly, and he did so with the support of the CMake developers. It took only a few weeks to get much of KDE building smoothly using CMake, and autotools was finally dumped!

The CMake developers have been very supportive of KDE's transition. They even joined KDE's buildsystem mailing list to help out. This relationship has been beneficial to both projects. When KDE started to really stress the capabilities of the still-maturing CMake system, KDE developers started a dialogue with them suggesting improvements - and they were happy for the feedback. As a result, CMake has seen improvements that are beneficial to all projects using the buildsystem.

Our working relationship aside, CMake has greatly improved the process of building KDE. Projects using CMake take less time to get started, since there is less time spent fighting with the build system. One KDE developer says, "CMake doesn't make you want to shoot yourself with a nailgun when building your project anymore."

The way CMake works is by processing an easy to read file called 'CMakeLists.txt' which the developer adds to their source folder. When you run the 'cmake' command, it looks for this file, and based on its contents, it generates Makefiles (on UNIX), or using a command-line switch it can generate XCode project files so you can build Mac applications using OS X's XCode developer tools, or using yet another command, it can even generate MSVC projects from your sources. One of the best KDE-related features of CMake is that it can automatically generate KDevelop project files based on the same 'CMakeLists.txt' used to generate the Makefiles.

KDE's code (with a few exceptions) was already pretty portable, however it was nearly impossible to build KDE on systems like Windows, where autotools was not as well suited. Now much of KDE's improved platform support is thanks to the ease of building on these systems (And of course, Qt being available as GPL on these systems).

In KDE 3.x, building KDE using the recommended methods looked something like this:

% ./configure --prefix=/foo --enable-debug
% make
# make install

As you can tell, this is pretty standard autotools-style building, except that the scripts which controlled the build process are pretty hard to understand.

With CMake, there are some syntax changes (which may actually be less obvious than the old, familiar configure switches), but the commands are mostly the same.

% cmake -DCMAKE_INSTALL_PREFIX=/foo \
-DCMAKE_BUILD_TYPE=debugfull .
% make
# make install

This syntax does not look like a huge improvement, but looks can be deceiving.

CMake searches for dependencies several times faster than the './configure' process did. CMake builds KDE 4's kdelibs about 40% faster than autotools build KDE 3.5.6's kdelibs, mostly due to a lack of libtool in the toolchain for CMake. The CMake toolchain (for UNIX) looks something like: cmake + make, whereas the autotools chain for KDE 3.5.6 looked something like: automake + autoconf + libtool + make + sh + perl + m4.

I'm going to rely on some of my own personal experiences here to help explain how much easier CMake is to use: Aaron Seigo asked for some help to port some of kdesktop's components into krunner so that kdesktop could finally be killed off in the KDE 4 tree. This is the sort of task that I would not have attempted to tackle in the KDE 3.x tree, not because the code was any harder, but because the build system was so difficult to deal with. In all likelihood, I would not have volunteered for such a task for KDE 3, but with KDE 4 and CMake, after moving the code around and doing some class renaming and such, I smoothly inserted the ported code into the build system, modifying only two or three lines of krunner's CMake build files. It built after a few minutes, and linked and installed. I was so impressed that I've kept contributing porting help. In KDE 3, I probably would have been stumped by the build system (not the actual KDE code) and given up, not bothering to commit my code to KDE SVN.

This only goes to show that the build gurus of KDE can worry a lot less these days, as anyone can get their projects up and running. Your milage may vary, but many developers have expressed similar feelings when dealing with the 'CMakeLists.txt' syntax vs. 'autotools'. However, almost all of the KDE developers would now be considered CMake newbies. The CMake developers have been personally involved in ensuring that KDE's transition is as smooth as possible.

The switch to CMake isn't the first time KDE has changed one of the technologies central to its development. When KDE was younger, we used CVS to control access to the sources. Maintenance of the CVS server was not scaling well as KDE was growing, amassing enormous amounts of code dating back to its original commits. Subversion (SVN) offerred a promising new revision control system that was a better fit for what KDE needed to accomplish, and was easier on our servers. At the time, however, no project the size of KDE had migrated to SVN, and it was a real test for the Subversion software. But KDE and SVN have been a pretty good fit, and since KDE migrated, many other projects have since followed in its footsteps.

KDE's use of CMake has helped to raise the public profile of the buildsystem, much like how the switch to Subversion helped their profile. Other projects have been switching to CMake too, including (but not limited to): Scribus, Rosegarden (switched from SCons), PlPlot, ChickenScheme, and more. There is also work being done to support building KDE 3.x programs using CMake (KPilot for KDE 3 can build using CMake, for example). Projects looking for an easier way to build whilst adding support for more platforms should give CMake a try. Adding a 'CMakeLists.txt' file to your sources does not interfere with your project's existing build system and can give you a good overview of what CMake can do for you. And, like KDE, if there's something it cannot handle, the CMake team is very open to improvements.

Some links that may be useful for those interested in more information:

I promise to have some eyecandy next week folks, until then...

Comments

Another stellar article. Thanks Troy!


By Paul Eggleton at Wed, 2007/02/21 - 6:00am

I was so preoccupied when I read this. I really enjoyed. Thanks for the nice article Troy!


By Totya at Wed, 2007/02/21 - 6:00am

Some screenshots of the new build system would have been really nice. :-P


By KDE User at Wed, 2007/02/21 - 6:00am

A screenshot of a Konsole building KDE, with colouring, progress reporting and colorgcc?


By Thiago Macieira at Wed, 2007/02/21 - 6:00am

For goodness' sake, can we make it a video so I can see the window properly wobbling while building?


By Wade Olson at Wed, 2007/02/21 - 6:00am

I'd love that, It would be great fun to pretend I can't understand why no one else finds that screenshot exciteing.


By ben at Wed, 2007/02/21 - 6:00am

...for example of the clean colored and readable output compared to the unreadable mess from autotools


By Josel at Wed, 2007/02/21 - 6:00am

it's fairly old, but: http://chani.ccdevnet.org/sshots/cmake-is-shiny.png

it seems I can only post in plain text, so I can't make a link. how odd.


By Chani at Mon, 2007/02/26 - 6:00am

Great article Troy. Thanks again for your excellent articles.


By Rafael Fernánde... at Wed, 2007/02/21 - 6:00am

Great Article as useuall.

And I was just wondering if there is any plan for a road to kde4 about Kmail?


By ben at Wed, 2007/02/21 - 6:00am

I've seen several blogs about a new mail program called Mailody. Where does this fit in with KMail?


By matt at Wed, 2007/02/21 - 6:00am

As far as I know, the kmail +/- mailody situation for KDE 4 has not been decided. However, it will be decided in a friendly manner as all of the developers involved are on extremely good terms :) When this decision is made, I'll likely write about it (especially if the decision is to move to mailody).

Cheers


By Troy Unrau at Wed, 2007/02/21 - 6:00am

currently Mailody only supports imap (unless things have changed recently) which means at this stage at least it is not a complete replacement for kmail.


By Danni Coy at Thu, 2007/02/22 - 6:00am

True for now. However so far the code for Mailody appears much cleaner, which gives us hope that we can add all the features kmail has, AND in the process fix a lot of bugs. Most likely we can also add a few feature requests that people want.


By Henry Miller at Thu, 2007/02/22 - 6:00am

And get a better name, just needs a K in it.


By ben at Thu, 2007/02/22 - 6:00am

Please no random K's!

Although Mailody isn't that great a name either.


By Tim at Thu, 2007/02/22 - 6:00am

And what's wrong with Mailody? its a nice querky pun. KMailody or Mailody-K would be a great name for the official client.


By ben at Thu, 2007/02/22 - 6:00am

> And what's wrong with Mailody? its a nice querky pun.

you nearly answered you're own question ;) quirky puns are neat, but also suck for apps. personally i like "cool names" for infrastructure (different audience, different use case) but really like straight forward names for apps (generally easier for users).

and rather unfortunately in this case, the name when spoken sounds rather close to this english word:

http://www.m-w.com/cgi-bin/dictionary?va=malady

the feature set and quality is slightly more important, though


By Aaron J. Seigo at Fri, 2007/02/23 - 6:00am

You really underestimate users, and the menu. In KDE3.5 it will have something like "Konversation IRC clinet". Any user can figure out that E-mail will either be in internet or perhaps office, and by checking both work out that _mail_ody Mail Clinet is probobly an E-mail program.

User's who can't will never find anyhting without a tutuor so don't bother trying to make it understandable to them, as the saying goes "If you think your users are idiots, only idiots will use it."


By ben at Fri, 2007/02/23 - 6:00am

honestly, with the name and the icon, for Mailody is some kind of programs to send music throught internet :/


By Cyrille Berger at Fri, 2007/02/23 - 6:00am

With attachments, it can be :)


By ben at Fri, 2007/02/23 - 6:00am

Excellent article, sorry to be pedantic but isn't it CMakeLists.txt?


By Richard Dale at Wed, 2007/02/21 - 6:00am

Yes... did I make a typo that needs to be fixed?


By Troy Unrau at Wed, 2007/02/21 - 6:00am

I personally think the:

% cmake -DCMAKE_INSTALL_PREFIX=/foo \
-DCMAKE_BUILD_TYPE=debugfull .
% make
# make install

Syntax is clumsy. It's just not how other UNIX tools work. Double dash for words and single dash for letters works for stacks of other GNU utilities, so why not cmake?

Please, at least the final release of KDE4, consider wrapping cmake with something that supports the legacy syntax. "prefix" is much easier to remember than "DCMAKE_INSTALL_PREFIX".

Just my 2p.


By Alistair John S... at Wed, 2007/02/21 - 6:00am

A couple of points:
* Remember that it isn't just a unix tool. It's a cross platform tool.
* that cmake line is passing a single letter (-D), not a word (-DCMAKE_INSTALL_PREFIX). CMAKE_INSTALL_PREFIX is the name of the variable.
* It might be more useful to contribute a proposed change than to criticise the tool and its existing usage.


By Brad Hards at Wed, 2007/02/21 - 6:00am

Even if it's not only a *nix tool, the syntax is still clumsy. And it sure does not look like the usual Windows command line tools either, so that point is rather moot.

I'd say it's plain stupid, and it's really bad interfacedesign. It looks like they change from a well known and familiar interface/syntax just to be different. Adding the fact that the other way is easier to read/remember and shorter to type, usability wise that's make really no sense at all.


By Morty at Wed, 2007/02/21 - 6:00am

Yeah, but in all honesty no build system is perfect. The syntax is a little clumsier than ./configure was, but the benefits far outweight this one problem.

Actually, on my system I took that command and created a bash alias, so I now just type 'cmakekde /path/to/source' and it automatically fills in the variables. Of course, if I wanted to redefine the variables (to turn off debug, for example), I'd now have to edit my .bashrc file.


By Troy Unrau at Wed, 2007/02/21 - 6:00am

Have you actually used it?

Also, see previous comment on providing potential fixes instead of complaining.

There is another alternative too - use ccmake.
I often use:
ccmake /path/to/source
and then you can work through the various options, selecting to build unit tests, checking output and so on. Then just generate.


By Brad Hards at Thu, 2007/02/22 - 6:00am

like said --prefix= --libdir= is smarter and not just easier but makes more semce. using ccmake is ok I guess for people, that love to just do make make install . not packaging it up etc . But the fact is to ./configure -help and you can see all the . Where is the help for this in cmake to know what replaces --prefix= . took me a lot of reading to find out how stupiod it is to do all this extra work . take wath made more sence and easy to remmeber etc . the list goes on. CFLAGS= CXXFLAGS= CC and how about if I'm running my x86-64 kernel and I could add --build=arch-blah-linux(-gnu for some) so it uses the right arch when compiling . way to many reason not to like cmake . it horrible
6hrs of googling to find out how to replace those and I only have figured out 3 prefix and maybe libdir and include but not arch at all not --sysconfdir=/etc
-localstatedir=/var
maybe these 3 memeory or not those are not easier at all and take way to much to get used to build script or not . 1200+ apps I compiel and very few use cmake . so glad for the many that don't .
been asking in kde and no one knwo of any where that shows what to replace any of them with , other than the one for prefix. Even when you find out look at it you can't be that blind to how it made it wprse not better.
-DCMAKE_INSTALL_PREFIX=/usr \
-DLIB_INSTALL_DIR=/usr/lib \

-DCMAKE_ADD_DEFINITIONS="$SLKCFLAGS" \
trying to compile 1 stinking lib for an app that uses this errr.
scons jam all suck . Control and what they use to replace are horrible to find out and they use the same crap.


By oneforall at Mon, 2007/09/03 - 5:00am

The syntax is similar to the gcc syntax. I would expect any KDE developer would be used to commands like "gcc -dFOO -dBAR -i../include " and so on for 5 lines. Most of the code I've compiled ends up with command lines like that. Of course make takes care of it, so you don't have to pay attention too often.


By Hank Miller at Thu, 2007/02/22 - 6:00am

To the dude complaining about the flags...

First get a grip. THIS IS A CROSS-PLATFORM TOOL. What that means is that 'tools' that use other platforms will be using it. No matter *how* the switches were set up it would either be not windows-like or not unix-like so tough, get over it.

But the more important fact is that if you are seriously doing a large cross-platform project with lots of folks from different platforms, the differences do indeed wind up 'wrapped' on each platform so CMake (or whatever you are using; we use both that and Bakefile on a daily basis) feels like a batch file on Windows (bleh) and it acts like make on unix. This is the way things sort out in the real world.

The other thing to really get a grip on though is if you are doing cross-platform development of any complexity, you are will have *far* more critical things to worry about than the syntactical style of the make system.

This too is a freebie from the Real World (tm).

Having used CMake for years (well, two) and Bakefile for nearly as long they both have their strengths and weaknesses. On unix I can set up a new CMake project faster than I can in Bakefile but bakefile gives me some nice configuration options that CMake does not, not the least of which is a more standard project file format (XML vs whatever CMake is).

But both are excellent projects and KDE would do well by either one. I know it has saved us TONs of time and work...

Cheers..


By JeffCobb at Fri, 2007/02/23 - 6:00am

Nobody said anything about CMAKE being bad. This isn't a parade against CMAKE per se.

I simply believe that it uses bizarre flags for no really good technical reason. I even suggested that CMAKE need not be changed, just wrap it.

The point is, it's fine saying to people you've switched from autotools -- I don't think _anybody_ really knows how they work -- but at least try to reduce the learning curve involved with using CMAKE, or fix CMAKE.

Other "cross platform tools", like Trolltech's offerings, don't suck like this. It's absolutely no excuse.


By Alistair John S... at Sat, 2007/02/24 - 6:00am

well considering I crosscompiled with make I'd say it is allready and cross platform too. Plus I cmpile 32bit on my 64it os that I yes crosscompiled. cmake is not easier at all its just made compiling worse . --prefix= is a heck of a lot smarter and simpler and makes more sence that -DCAMKE_....forgot the rest and no wonder. ./configure -help . I don't think your blind to what thats a 1000 times betrer for cmake umm O.o no idea spent hrs allready trying to find out whats it uses to replace --prefix= --libdir= . got those and man they are sick . still haven't founf out about CFLAGS CXXFLAGS CC and when i need to do LDFLAGS PKG_CONFIG_PATH arch-blah-lunux(-gnu for some distros) to make sure it uses um eg; i486 on my 64bit OS . Thats only a few of the horribleness of cmake and to make packages . I patch and edited configure to remove hard coded stuff like /usr/lib to /usr/lib64 etc. But cmake makes them look more peacefull an much better


By oneforall at Mon, 2007/09/03 - 5:00am

For KDE4, you can add:

function cmakekde { cmake -DCMAKE_INSTALL_PREFIX=$KDEDIR \
-DCMAKE_BUILD_TYPE=debugfull $@ && make VERBOSE=1 && make install; }
## Uncomment if dbus doesn't work
#alias dbusstart="eval `PATH=$DBUSDIR/bin \
#$DBUSDIR/bin/dbus-launch --auto-syntax`"

to your .bashrc, then compiling KDE is even easier. Then a simple `cmakekde ../kdegames` (for example) from the kdegames-build directory. CMake greatly reduces the barriers to entry for developing KDE, and lets me focus on coding rather than on an incomprehensible (imo) auto* build system.

See http://developernew.kde.org/Getting_Started/Build/Unstable_Version for details.

KDE is *far* easier to compile than it used to be. Long live CMake!

Regards,

Mark


By Mark Taff at Thu, 2007/02/22 - 6:00am

Locally I have a patch for cmake which (almost) does this, it let's you define "shortcuts" for the original cmake variable switches (e.g. you can put something like "CMAKE_INSTALL_PREFIX = prefix" in some config file and then cmake will translate "--prefix" to "-DCMAKE_INSTALL_PREFIX=".
But it's not done yet, it does translate the things you want to have translated but breaks all other options :-P

But I hope to be able to make it work before KDE 4.0

Alex


By Alex at Thu, 2007/02/22 - 6:00am

So if CMAKE becomes popular I could continue installing with ./configure make make-install and never have to worry about what build system is used?


By ben at Thu, 2007/02/22 - 6:00am

It'd be:
cmake .
make
make install


By Pau Garcia i Quiles at Thu, 2007/02/22 - 6:00am

Alex, this is really great. Not that the barrier to entry to KDE is very high to begin with, but this would lower it even more...

Great idea!


By Troy Unrau at Thu, 2007/02/22 - 6:00am

Agreed. I have similar problems when compiling a Qt or Python app. I'm suddenly a compile-newbie again. How about providing a ./configure wrapper script? It could translate the --prefix stuff to cmake commands, or at least inform the users about the correct syntax. There are more projects that provide a configure wrapper for user-friendliness reasons, KDE doesn't have to be an exception here.


By Diederik van de... at Thu, 2007/02/22 - 6:00am


By Thomas Zander at Thu, 2007/02/22 - 6:00am

Hi all,

I'm quite new to KDE and I wanted to know if there are auto-updates. I mean, if I use KDE 4.01, and then appears KDE 4.02, will it be notified and my system updated? If such feature exists, does it work for all KDE apps?

Sorry for my ignorance, but this is an important topic for me.


By David at Wed, 2007/02/21 - 6:00am

Your question would more appropriately directed to users of a distro of your choice. While KDE will not tell you of an updated release, they're not really in a position to. That would more properly be a function of a distros package management system.


By Sean Brown at Wed, 2007/02/21 - 6:00am

So that would be a "no" to the original question.

KDE does not have an auto-update feature like the one found in recent Mozilla Firefoxes.

KDE follows the tradition of letting the Linux distribution (SuSE, Debian, Gentoo, Fedora etc.) decide how to handle updates.


By Anders at Wed, 2007/02/21 - 6:00am

Offtopic but interesting idea out of this thread:

* mozilla autoupdates are useful on windows
* kde4 could be available for windows
* kde usually handles updates via linux distributions

-> there needs to be some sort of free software distribution mechanism on windows, like apt.

oh - somebody already thought of that: http://www.pcxperience.org/ideas/AptWindows.html


By Nicolas Dietrich at Thu, 2007/02/22 - 6:00am

actually there is some work going on on a Windows KDE installer/manager, which might indeed do this in some way.


By superstoned at Thu, 2007/02/22 - 6:00am

As others pointed already out there is no need for this on Linux/Unix as we already have such systems which do exactly what you want for every application you have installed.

On Windows for example I am always deeply annoyed when 5 programs tell me after system startup "Hello user there is a new hot cool update for me, I am so cute please update me." One central application where different vendors can queue their software repositories and which would allow you to update the whole system with one mouse click would greatly enhance security and usability on Windows. But honestly I am really lucky that MS did spend billions of Dollars but still did forget even the most basic things and that they still love their unstable and slow browser based update system. So Windows including the newest version actually looks like stoneage to me. :p

But beside that there even is a KDE program called Get-hot-new-stuff (see: http://www.kstuff.org/ ). It allows you to share and search for new application data for your system (for example new document templates, new wallpapers, new game levels...).


By Arnomane at Thu, 2007/02/22 - 6:00am

> On Windows for example I am always deeply annoyed when 5 programs tell me after system startup "Hello user there is a new hot cool update for me, I am so cute please update me." One central application where different vendors can queue their software repositories and which would allow you to update the whole system with one mouse click would greatly enhance security and usability on Windows. But honestly I am really lucky that MS did spend billions of Dollars but still did forget even the most basic things and that they still love their unstable and slow browser based update system. So Windows including the newest version actually looks like stoneage to me. :p

Agreed. After leaving my Windows off for a month, it is true agony when I start it up again.


By riddle at Sat, 2007/09/29 - 5:00am

A distro like kubuntu has a way to do this for all of your system with a nice system-tray item stating there are things to update. No need for KDE to reinvent that wheel.


By Thomas Zander at Thu, 2007/02/22 - 6:00am

Updating packages (including the various pieces that make up KDE) is more the responsibility of the operating system on which KDE is running (at least on *nix operating systems).

Having said that, there are various KDE applications which act as front-ends to the various OS-specific package systems. These provide eg. icons in your tray that pop up when updates are available, offering some way to perform the upgrade. Here are some examples to get you started:

(OS - Tool - URL)
Kubuntu (and Debian?) - Adept - http://kde-apps.org/content/show.php?content=44909
Gentoo - Kuroo - http://kuroo.org/
FreeBSD - Barry - http://www.student.uni-oldenburg.de/frerich.raabe/barry/
FreeBSD - KPorts - http://www.kde-apps.org/content/show.php?content=24579

It's not a bad situation because these tools update everything on the system (and sometimes the system itself, too), not just KDE. The few I have personal experience with have been easy to use. For example I've got my mum running Kubuntu (which uses Adept) - she's essentially computer illiterate but has no problem keeping the system up-to-date.


By macro at Fri, 2007/02/23 - 6:00am

This is indeed something that your distribution should do for you. Personally i have Gentoo and just run "emerge --sync&&emerge -uDNf world&&emerge -uDNtav world" once in a while. (I do not even have to type it every time, I just press Ctrl+R and type "sync", then the command comes back.) But if you want notifications directly from KDE, kde-announce is what you are looking for (see [http://www.kde.org/mailinglists/]).


By Erik at Sat, 2007/03/03 - 6:00am

Pages