The Road to KDE 4: CMake, a New Build System for KDE

When a project with the size and scope of KDE gets to be as big as it is, sometimes changing a decision established almost a decade earlier is very difficult. KDE has relied on autotools to build most of the project since its inception, but for the last year, KDE 4 has been building with CMake, a newer build system that is rapidly becoming a heavyweight contender in the world of buildsystems. Read on for more...

This article focuses on CMake, which is not actually part of KDE, but is developed independently by Kitware and a team of open source developers. It is released under a BSD-style license. I can't really show much in the way of screenshots for a build system, but I will do my best to explain why CMake has been a welcome change to the KDE development process.

But before we talk about CMake, a little history lesson about KDE and autotools. KDE has used Qt since the beginning, and one of the nice features of Qt is the meta-object compiler (moc). Autotools had to be extended to support using moc as a pre-processor for most of KDE's header files. That was only the beginning though, as KDE developers wrote the ground-breaking DCOP communication protocol which also had the side effect of adding more types of files that had to be autogenerated during the build process. The developers added document compilers, tools that automatically handled translations, and tools that compiled configuration file classes from XML: Qt's User Interface compiler (for .ui files) was born and needed to be supported by the build system. They also needed to add a whole suite of new configure checks, options, etc. to support all of the features that KDE was using. The KDE build system was evolving into a very complex animal, and using autotools did not scale well.

By the KDE 3 series, there were only a select handful of elite build gurus who could understand the whole of KDE's build system. KDE developers had trouble even moving things from one folder to another without having to spend hours tweaking build files to get it to compile again. And in some cases, starting an independent KDE project meant shipping up to 500 KiB of autotools support files for even a simple 'hello world' type application.

Something obviously had to be done for KDE 4 to improve this situation, and at an Akademy 2005 meeting, it was decided to explore other build system options. Initially, SCons was prototyped for building KDE, but it was slow going, and after several months of work it still did not handle kdelibs well enough to dump autotools. One of the bigger problems with SCons was its lack of modularity.

So, as is often the case with KDE, he who codes, decides: Alexander Neundorf stepped in and did the initial port to CMake which went very smoothly, and he did so with the support of the CMake developers. It took only a few weeks to get much of KDE building smoothly using CMake, and autotools was finally dumped!

The CMake developers have been very supportive of KDE's transition. They even joined KDE's buildsystem mailing list to help out. This relationship has been beneficial to both projects. When KDE started to really stress the capabilities of the still-maturing CMake system, KDE developers started a dialogue with them suggesting improvements - and they were happy for the feedback. As a result, CMake has seen improvements that are beneficial to all projects using the buildsystem.

Our working relationship aside, CMake has greatly improved the process of building KDE. Projects using CMake take less time to get started, since there is less time spent fighting with the build system. One KDE developer says, "CMake doesn't make you want to shoot yourself with a nailgun when building your project anymore."

The way CMake works is by processing an easy to read file called 'CMakeLists.txt' which the developer adds to their source folder. When you run the 'cmake' command, it looks for this file, and based on its contents, it generates Makefiles (on UNIX), or using a command-line switch it can generate XCode project files so you can build Mac applications using OS X's XCode developer tools, or using yet another command, it can even generate MSVC projects from your sources. One of the best KDE-related features of CMake is that it can automatically generate KDevelop project files based on the same 'CMakeLists.txt' used to generate the Makefiles.

KDE's code (with a few exceptions) was already pretty portable, however it was nearly impossible to build KDE on systems like Windows, where autotools was not as well suited. Now much of KDE's improved platform support is thanks to the ease of building on these systems (And of course, Qt being available as GPL on these systems).

In KDE 3.x, building KDE using the recommended methods looked something like this:

% ./configure --prefix=/foo --enable-debug
% make
# make install

As you can tell, this is pretty standard autotools-style building, except that the scripts which controlled the build process are pretty hard to understand.

With CMake, there are some syntax changes (which may actually be less obvious than the old, familiar configure switches), but the commands are mostly the same.

% cmake -DCMAKE_INSTALL_PREFIX=/foo \
-DCMAKE_BUILD_TYPE=debugfull .
% make
# make install

This syntax does not look like a huge improvement, but looks can be deceiving.

CMake searches for dependencies several times faster than the './configure' process did. CMake builds KDE 4's kdelibs about 40% faster than autotools build KDE 3.5.6's kdelibs, mostly due to a lack of libtool in the toolchain for CMake. The CMake toolchain (for UNIX) looks something like: cmake + make, whereas the autotools chain for KDE 3.5.6 looked something like: automake + autoconf + libtool + make + sh + perl + m4.

I'm going to rely on some of my own personal experiences here to help explain how much easier CMake is to use: Aaron Seigo asked for some help to port some of kdesktop's components into krunner so that kdesktop could finally be killed off in the KDE 4 tree. This is the sort of task that I would not have attempted to tackle in the KDE 3.x tree, not because the code was any harder, but because the build system was so difficult to deal with. In all likelihood, I would not have volunteered for such a task for KDE 3, but with KDE 4 and CMake, after moving the code around and doing some class renaming and such, I smoothly inserted the ported code into the build system, modifying only two or three lines of krunner's CMake build files. It built after a few minutes, and linked and installed. I was so impressed that I've kept contributing porting help. In KDE 3, I probably would have been stumped by the build system (not the actual KDE code) and given up, not bothering to commit my code to KDE SVN.

This only goes to show that the build gurus of KDE can worry a lot less these days, as anyone can get their projects up and running. Your milage may vary, but many developers have expressed similar feelings when dealing with the 'CMakeLists.txt' syntax vs. 'autotools'. However, almost all of the KDE developers would now be considered CMake newbies. The CMake developers have been personally involved in ensuring that KDE's transition is as smooth as possible.

The switch to CMake isn't the first time KDE has changed one of the technologies central to its development. When KDE was younger, we used CVS to control access to the sources. Maintenance of the CVS server was not scaling well as KDE was growing, amassing enormous amounts of code dating back to its original commits. Subversion (SVN) offerred a promising new revision control system that was a better fit for what KDE needed to accomplish, and was easier on our servers. At the time, however, no project the size of KDE had migrated to SVN, and it was a real test for the Subversion software. But KDE and SVN have been a pretty good fit, and since KDE migrated, many other projects have since followed in its footsteps.

KDE's use of CMake has helped to raise the public profile of the buildsystem, much like how the switch to Subversion helped their profile. Other projects have been switching to CMake too, including (but not limited to): Scribus, Rosegarden (switched from SCons), PlPlot, ChickenScheme, and more. There is also work being done to support building KDE 3.x programs using CMake (KPilot for KDE 3 can build using CMake, for example). Projects looking for an easier way to build whilst adding support for more platforms should give CMake a try. Adding a 'CMakeLists.txt' file to your sources does not interfere with your project's existing build system and can give you a good overview of what CMake can do for you. And, like KDE, if there's something it cannot handle, the CMake team is very open to improvements.

Some links that may be useful for those interested in more information:

I promise to have some eyecandy next week folks, until then...

Comments

by Troy Unrau (not verified)

Just thought I'd add the following link, which points to a blog post by Richard Dale about some of his experiences using CMake to build some other non-KDE projects. It's a good read, if you don't mind the technical details :)

http://www.kdedevelopers.org/node/2685

by Rodrigo F Toso (not verified)

This is an amazing job! Thank you for the time... This article is just perfect!
Again, thank you.
Regards,

by anonymous-from-... (not verified)

i usualy do push-ups while my project compiles.

by Thiago Macieira (not verified)

That's good for staying in shape!

However, I'm sorry, but we have cut down the time you'll have for this activity. KDE 4 compiles much faster now with CMake :-)

(And there's still potential for being even faster)

by superstoned (not verified)

Well, that sucks, then. He should sue KDE, when he gains weight and gets the associated diseases...

:D

by Ben (not verified)

I've now used both for developing smallish projects and I can see that they are very similar in capabilities now. They both have easy syntax (relative to automake), a bunch of tools for doing different build tasks, and sane dependency specification. However, it seems like CMake is a bit more inflexible if you want to add a custom build task (like running unittests automatically) into the build process because scons is a dsl vs. cmake which is a custom language.

Likewise, scons has little/no support for configuration and reconfiguring the build process for the users machine which is a big negitive. waf tried to address this but I guess it was too little, too late.

So I guess it's a bit of a toss up. CMake is ok for kde4 need's but I really hope that the kde community reexamines either scons or rake (or another dsl build system) for kde5.

Cheers
Ben

by Ben (not verified)

Hey, another Ben :)

by oneforall (not verified)

How can you call either one of those easier for the accual compile. they both are horrible for that. Both are hard to find out what even they use to replace --prefix= (-DCMAKE_INSTALL_PREFIX) look at that the first is a better syntax by far . makes it a lot easier to remmeber and use and the rest to follow. --libdir= --mandir docdir etc sdo easy . ./configure --help for all the options . So far I have spent hrs of googling asking in kde kde-devel with no help at all for someting they have stuck use with. So where is the replacement for ./configure --help thats probably longgggggggg winded too. No control over where things go and if there is nothign helpfull like ./configure --help . how can I do the one for the arch I want to build ? --build=arch-blah-linux . So its not building x86_64 when I want 32bit etc . cmake even for x86_64 still puts arch-unknown-linux when I want it to put the right one. So manty reason I can't stand this cmake crap
glad out of the 1200+ apps I compile there is only 1 or 2 that use this horrible crap

by Michael (not verified)

Have you ever tried CCMAKE? Just run ccmake . to get a list of options.

by oneforall (not verified)

yes I have looked at that too and it doesn't list them in the way you can see to use them cmake . .
cmake is a poor horrid auto tool . Yes with physfs it s a poor done CMakeLists.txt iirc that does the searches and makes the cmake_install.cmake. I dunno maybe I have that backwards. but its a horrible tool for paths and hard coding and making it way harder to figure out how to patch or sed a line than it is with configure. The hard coded path for physfs I had to fix it after like a sed for libtool . which I never like it being after it should be fixed before. With libtool I figured out how ro fix it with sed in the configure. I didn't have to know how to add code in there like with the physfs one and others that are the same. nywasy I spent too long compared to using configure make etc and it still is not easy to use at all. Mostly because of it not as flexable it geared to things must be installed here and no where else attitude. all capps and long definition that just in that sense are not an improvement to do thing the long scenic route. --libdir= nice and simole easy to remmebr than -DCMAKE_INSTALL_LIB= or is that one -DLIB_INSTALL_DIR= so many that need to addvertize CMAKE and INSTALL no reason it couldn't still be simple -Dlibdir -Dprefix= . Anyways it never took me long to be comfortable with configure and figure out how to use it for most things. Glad that most apps don't use this crap and hope they don't . Even in kde-devl I posted a lot about poor with paths and the fact on a 64bit if you want to compile a -m32 you can't even use or find out how to tell it the frigging arch --build=arch-vendor-linux-(opt gnu) . CFLAGS="-m32 (-O2 -fPIC etc if you want )" this can be used with cmake too but the arch every one says the same to goto the cross compile page . well with configure I don't need to go back to cross compiling and cross compile tools. --build= host= but mostly just -build= works =i486-vendor-linux =i686-pc-linux etc It a frutrating auto compile tool at most . people that just do cmake and no option have no clue any more than some one that does that with configure .

by Ben (not verified)

This seems like it's just another attempt to make kde look more interesting for devs on Windows and OSX really. I personally don't get all the craze of kde being able to run on Windows or OSX. QT4 is basically only advertised as being a cross platform toolkit which brings kde to Windows.
How about concentrating about the main (and most important) platform? The "free software" platform. I don't hear much of that lately.

by Ben (not verified)

If you read the aticle you'll notice that this makes developing KDE on Linux/BSD much easier. It also makes it much possible to build it on Mac and Windwos but even if you don't give a damm about the non-free platforms its still a major improvement over KDE3

Personally I havn't herd much about mac/windows relitive to Linux.

by Troy Unrau (not verified)

Well, you will note some things in the article that specifically relate to the free platforms. Specifically, that CMake can autogenerate kdevelop project files is a big plus, as kdevelop is one of the best IDE's available for open platforms.

But KDE's support for other platforms is sort of a byproduct of the good code that exists for *nix. We aren't specifically designing KDE to suit Windows or Mac users so much as the underlying system makes the transition easier. 99% of all KDE developers are linux or bsd people, and even those that are working on the Mac or Windows ports are still free platform users.

/me wonders if you complain about firefox being available on windows as well, even though it has it's roots in *nix.

by BenAgain (not verified)

I'm not against porting applications to other platforms at all actually.
I just always thought that KDE was about empowering the free software platform.

With the emphasis on cross platform compatibility usually come compromises you have to make (especially if you have no control over the underlying system - i.e. on proprietary operating systems). That will/could directly affect the usability on the main platform (e.g. like when you realize why the default web shortcut for google in konqueror isn't just "g:". bad example maybe because it's so minor, but hey ;-).
It also means there will be need for even more maintenance and developer time to support other (less important) platforms ("[PATCH] to make *thing* work on Windows"). As far as I know KDE is already lacking manpower.

My reply wasn't specifically in the regard of cmake but more in general because I feel KDE4 is being advertised as being the big Cross Platform Desktop Opportunity instead of a Free Software Desktop.

Well, maybe it's just me. ;-)

by Aaron J. Seigo (not verified)

the free software platform is and always will be our prime concern. that's why we provide an entire workspace shell for it, which we are putting even more effort into for kde4.

the win/mac efforts are left up to people interested in win/mac, which is 95% not people who were already working and who continue to work on free platforms.

by BenAgain (not verified)

I just read http://aseigo.blogspot.com/2004/12/how-to-kill-open-source-on-desktop.html.
Are you still feeling that way now since cross platform KDE (applications) is becoming real?

by Anonymous (not verified)

I actually thought the point of KDE (and open-source software in general) was the freedom. So, then, why are so many people so adamant about forcing people to use Linux? Isn't that basically blasphemy, or self-contradiction?

by Ian (not verified)

Is there a www.kde4.org (someone has already got that) type site where all the information about KDE4 is disseminated?
There is a momentum building throughout for information about it but at the moment you need to be "in the know" of what the constituent parts are going to be in order to find out information. The ideas are fantastic when read about in their isolated silos but it would be good to see them all together showin in a framework of a single site/link from www.kde.org. People love roadmaps and technology maps.
A lot of the names like Plasma, Okular, Decibel don't really tell you what they are about - (sometimes you can guess), how they fit into the total structure, if they depend one each other etc. They all sort of sound like something a marketing driod has thought up, at least with the "K" prefix, you had an idea it was KDE related. (i expect this has all been hashed out already).
I could go on but I've a train to catch....

Thanks to all the developers for a great DE and the apps to match.

by Troy Unrau (not verified)

There is no such site yet. Right now, the best way to find out is to build from sources, or try the technology preview releases (there should be one happening this week sometime) which occasionally get packaged by some distros...

When KDE 4 is eventually released, the official www.kde.org website will be updated to KDE 4 content.

But you're right, it'd be nice if there was a website that linked to all of the up-and-coming KDE 4 technology websites, as most of these projects already have a functioning website. The problem with such a suggestion (like all free software) is that there needs to be someone interested in doing such a project, and then be willing to maintain the content to ensure it's all up date all the time. That's difficult when KDE 4 is such a moving target...

by taj (not verified)

This whole series of articles has been fantastic to read. Thanks Troy!

by Gleb (not verified)

Pity, "scons" was abandoned in favour of an antiquated and non-intuitive "cmake" system. Very pitty. Just laxly?

by Rainer (not verified)

In our project http://pvbrowser.org we use qmake for building our (Qt-)applications.

Why didn't you use it to build KDE ?
You could have extendet it if you needed more custom build steps.
In the past i have added custom build steps for corba idl and oracles embedded SQL for example.

The advantage would be,
that you use the original build system trolltech is using anyway.
And qmake is very simple and clear.
And it is portable also.

PS: We always felt uncomfortable with autotools also. And have trown it away a long time ago.

by Tony O'Bryan (not verified)

I had the same thought the moment I first heard about KDE dumping autotools (thank you, thank you, a million times thank you!). Is there a critical weakness in qmake that cmake addresses better?

I've returned to using qmake for my multipart projects (I detoured into KDevelop for a couple years before giving up on it) for a few months now, and I find it to be a capable build system. It doesn't generate an install target for some strange reason, but maybe I'm just missing a setting.

Hi,

Don't bother about the eye candy I prefer thought candy and this article is full of it. It deserves a much wider audience than the Linux community. Why? Because it describes a process that leads many commercial companies to go bust or get in big trouble. It describes how power can be exercised without (it appears) power broking. No doubt there are implications for the democratic process itself but I won't try to expand on that.

huh?

I thought it described an easier way to build software, where dose power brokeing and democracy come in?

by Troy Unrau (not verified)

I think he's referring to the whole 'he who codes decides' bit. Sometimes KDE is referred to as a meritocracy, rather than a democracy. If your work is good, people accept it.

Ah, I didn't think of it since that was just a casual mention in the article.

P.S. Meritocracy's are the worlds best *ocracy

Ours did. I took them from MSDEV/make to CMake and then to Bakefiles. Now everyone gets something that feels pretty native to them and having a simple to understand cross platform make system has been our saviour (doing cross-platform multimedia frameworks)...usually all it takes is someone who has learned the make system well enough to demo to each set of proponents for each platform how it will save them time. Usually the argument that works is it keeps windows programmers from being frightened at the command line (as a properly-configured cmake or bakefile arrangement is a mouse-click to a Win32 dude/dudette) and it keeps unix folks from having to deal with dev studio at ALL. With this system, I can create an entire project under linux and (assuming I have my act together at the code level, something beyond the scope of the article), mail it to a windows dude who then (only the first time) runs 'bakefile -f msdevprj myproject.bkl and the result is a dev studio 6 project. I do 'bakefile -f gnu myproject.bkl and I have a nice unixy make file. Works for Mac OS/X too. IOW, to paraphrase an old rock song, the coders on each platform don't always get what they want, but they always get what they need....

by Darkelve (not verified)

I'm writing a tutorial for new Linux users and I have a question about KDE 3.5.x: if the Kwin window manager handles the drawing of window decorations, window contents, etc. , then does Kicker handle the drawing of all the components and parts of kicker? And for the desktop (wallpaper etc.) that would be Kdesktop? What about KDE4?

Then I have a request for the upcoming "Oxygen" article:

can you explain a bit how Oxygen will try to overcome the problems which exist with the current icon/iconset approach:

- When switching between iconsets, it is possible that you 'loose' some icons because alternatives do not exist in the iconset you are switching too. Mostly you'll get a really bland/ugly icon instead

- In KDE's "icon overview" screen, will it be possible to group the icons per theme? Now they are all mixed up and this may give problems when looking for an extra icon for an application, and you want to know if the icon is part of the current iconset you are using

- Are there some kind of icon 'preferences', so that e.g. you can make another icon the default instead of the current default icon. For example, I set the folders in /home/ to use a grey folder icon. But every time I or a program creates a new icon, as well in several Konqueror views, I get a blue icon. Will it be easier to set preferences of this kind in KDE4?

by Troy Unrau (not verified)

These are a lot of seemingly random questions :P

Kicker looks after most of it's own drawing, as does kdesktop, but they still use kwin to control their position on screen, whether they live in the background (like kdesktop), and that windows can't overlap kicker, for example.

With regards to the Oxygen stuff, you'll have to wait for that article... it's probably two weeks away.

Cheers

by Darkelve (not verified)

"These are a lot of seemingly random questions :P"

That is probably because I am talking about two different things here, and because I do not yet understand all of this very well myself ;)

For the tutorial, I'm trying to explain to people new to Linux what makes their Linux/KDE system tick; GUI elements that are drawn on their screen being one of them.

"Kicker looks after most of it's own drawing, as does kdesktop, but they still use kwin to control their position on screen, whether they live in the background (like kdesktop), and that windows can't overlap kicker, for example."

Much appreciated :) Can I use that quote above for the tutorial, and do you want me to credit you for it?

"With regards to the Oxygen stuff, you'll have to wait for that article... it's probably two weeks away."

Well, I don't mind waiting. I just wanted to know if any improvements are being made to the iconsets and especially the iconset tools (KDE Control Center/Personal Settings), seemed to me to be about Oxygen. Or is Oxygen only about the visual appeal and completeness of the icons?

Thanks for the friendly reply.

by Troy Unrau (not verified)

Feel free to quote or paraphrase...

Oxygen is more about the visual style and making sure it's consistent across all of KDE. It will not be missing any icons when KDE 4 ships. What will likely happen is that when you choose a non-Oxygen icon theme, and this theme is missing an icon, it'll just default to the Oxygen icon as necessary.

About fine tuning icon control, someone would have to write a kcontrol module to do that.

by James (not verified)

milage should be mileage.

Good article! I haven't used KDE in a while, but it is nice to get an overview of the build tools change. Thanks!