SEP
11
2005

KDE to Migrate to bksys/SCons Build System

At the build system BoF at aKademy it was decided to start moving the KDE 4 build system from unsermake to the SCons/Python based system bksys. To find out more about this important future technology, KDE Dot News talked to its lead developer Thomas Nagy about the reasons behind the change and what it will mean for KDE developers.

Thomas is ita on IRC

Please introduce yourself and your role in KDE.

My name is Thomas Nagy (I am also known as 'ita' on
freenode). I have spent some time writing some KDE
utilities (solvers, small games) and adding
some features to KDE applications such as KOffice and Kalzium
when i was still a student. Since i graduated i have
spent more time trying to create complete applications
like kdissert, a mind-mapping tool for the creation of
documents and its build system bksys.

What is SCons and what is bksys?

SCons is a software construction tool which is used to
build and install programs. It is written in Python
and its design is modular, making it easy to extend
for use with unsupported compilers or complicated
build rules.

bksys (as in Build Kde SYStem) extends SCons and aims
at replacing the autotools. It has been written in the
main place for compiling and installing kdissert and
became more modular over the time. The most important
feature is the object-oriented style API that can be
accessed from either Python code or throughout XML
description files.

Some more documentation describing the framework can
be found in the the slides from my aKademy 2005 presentation.

Why is bksys separate from SCons?

bksys aims at replacing completely autotools for
common projects, so this goes way beyond SCons' main
goals.

In which way are they better than autotools?

There are many improvements over the autotools, but I
will try to describe the most striking ones:

  • Reliability: targets are rebuilt if compilation
    flags are changed or if md5sums of the files changed
  • Size: 100KB (miniSCons included) vs 600KB
    (autotools)
  • Simplicity: one language (Python) versus at least 3
    (m4, posix sh, make) for autotools (GNU/make,
    automake, autoconf, libtool); no intermediate file to
    generate for the compilation process (Makefile.in,
    Makefile)
  • Configuration: the configuration process is fast,
    easy to control and to modify
  • Flexibility: SCons can be extended easily thoughout
    modules. Besides, helpers can be written easily to
    help programmers (take all source files in a
    directory, or check automatically the consistency of
    the builds for example). SCons also has excellent
    support for win32 and OSX platforms, it handles many
    languages like Java or Fortran out of the box.
  • Evolution: the API and the code written for
    compiling the targets are re-usable so projects using
    it will not be locked.

How do they compare to unsermake?

Unsermake is a replacement for make and automake only.
Bksys replaces the whole autotool chain.

Can you summarise the build system BoF at aKademy?

Several candidates were evaluated. QMake is way
too limited and cannot be used at all. Unsermake is
not to be kept, as it is only a partial solution and
emulating the behaviour of automake and make consumes
a lot of resources. Two solutions remained in
competition for the future KDE build system: CMake and
SCons.

CMake is a Makefile generator like QMake; it uses a
macro language which is known to present several risks
as non-standard targets arise frequently in KDE. On
the other hand SCons is much more flexible as it is
based on Python, and was proven to work on fairly
complex applications already (KDE Games, Rosegarden,
..). SCons was then chosen and the conversion of
kdelibs 4 has started.

So what work needs to be done now?

So far, the main goal of bksys has been to replace
autotools for usual KDE projects. Now for kdelibs the
situation is a bit different as the tool will be the
base of all KDE applications, so a new framework has
to be designed, in particular:

  • We have to agree on a new set of API (stardard
    targets, naming, features...)
  • New bksys modules have to be written for detecting
    the system configuration (full autoconf replacement)
  • Some work has to be done for writing config.h files
  • More helpers will have to be written to make
    programmers lifes easier (scan for sources
    automatically..)

At the very moment, the bksys scripts have been added
to KDE SVN, and 3 main directories in kdelibs 4 are
already compiling (dcop, libldlt, kdefx). There is
some documentation in trunk/KDE/kdelibs/bksys/design.
The main aspects of the future framework are
presented, and some questions are left opened to
discuss.

In the past, the kdegames module was converted in
about 2 days, but for kdelibs it will take a lot more
time as even after the framework is complete some
reverse-engineering will have to be done on the
Makefile.am to achieve the conversion. I am striving
at the moment to get a first version of the framework
ready (most configuration modules complete and a few
folders compiling) to show how the new system will
look like and the main advantages of using it.

Who is going to do the work?

.. or who is working on it already? Three people have
committed code on bksys in svn so far, Simon
Haussmann, Stephan Kulow and I. Interesting ideas have
been raised on the IRC, and I am awaiting
more comments, when the first version of the framework
is ready for testing (a request for comments will be
posted).

Are non-KDE projects taking up SCons/bksys?

Most projects using SCons do it directly and use their
own framework, like Blender or Rekall.
The projects using the bksys layer i know of are
almost all KDE-oriented: Rosegarden, kio-locate,
skim...

Comments

So, does that mean that KDE mandatorily requires Python as dependency?
Or does this only matter on source-based distros like Gentoo?


By Jakob Petsovits at Sun, 2005/09/11 - 5:00am

python will be a requirement to build from source. but python is found on pretty much every modern UNIX-y OS that has even a tangential desktop slant and is easily added to those OSes that don't ship with it.


By Aaron J. Seigo at Sun, 2005/09/11 - 5:00am

only as a build dependency, not a runtime dependency


By sb at Sun, 2005/09/11 - 5:00am

Yes, that's what I wanted to know, thanks.
(Although I'm one of those who would like
to see it as runtime dependency too :)


By Jakob Petsovits at Mon, 2005/09/12 - 5:00am

Thought BK sttod for bit keeper for a sec - wondered why KDE developers wanted to open that can of worms again :)


By john at Sun, 2005/09/11 - 5:00am

can of worm?? BK is/was one of the best available vcs, too bad zealots complained about the license ...


By chouimat at Tue, 2005/09/13 - 5:00am

Sounds good to me. The existing build and tools chain is an absolute pain to work with at any time. With this and the use of things like unit tests should make KDE a very nice desktop to develop for, and develop with than it is already. Done right this could certainly attract more developers and contributors to KDE come KDE 4.


By segedunum at Sun, 2005/09/11 - 5:00am

First Subversion, now a new build system. What's next? Ditching Bugzilla for Trac? ;-)

This is great work so keep it up!


By am at Sun, 2005/09/11 - 5:00am

We can only hope :-)


By Roberto Alsina at Mon, 2005/09/12 - 5:00am

Yes, please! trac is really nice..!


By Jonas at Mon, 2005/09/19 - 5:00am

Is it possible to build using distccd with scons like "unsermake -p -j4 CXX=distcc"


By Asif Ali Rizwaan at Sun, 2005/09/11 - 5:00am

scons supports parallel compilation natively (and even does a much better job than automake, about even with unsermake). scons also includes support for notating targets that produce files as a side effect, which can break parallel builds.

As far as overriding CXX that way, I'm not sure, it's easier just to put distcc's gcc symlinks first in the PATH anyways though.


By Michael Pyne at Sun, 2005/09/11 - 5:00am

Yeah, if bksys is anything like scons. (Im talking about scons here because I've been using it for a while now.)

In the toplevel SConstruct file, you can say

env = Environment(
CXX = 'distcc g++',
....
)
or:
env.Replace(CXX = 'distcc g++') # ccache distcc g++ works as well is you don't wnat to use scons caching.

when running scons say: scons -j N where N is the number of jobs you want to use.
It should work better then make -j N


By ben at Tue, 2006/01/10 - 6:00am

How are the issues (hard to parse existing Python build scripts since they are too powerful to catch all possibilities) raised in the past on the mailinglists about integration of the new building system into KDevelop to be solved? Strictly limiting API/syntact to the amount understood/supported by KDevelop? With everything not complying being thrown out from the SVN? Who is going to monitor this?


By ac at Sun, 2005/09/11 - 5:00am

From what I understand bksys will also be able to support a XML description of the target and its build dependencies, etc. The idea is to make it possible to do most of what you'd need to do using XML alone (which is machine-parsable, and can therefore be understood by KDevelop). Not sure how successful it will be, but there are KDevelop guys already working on it.


By Michael Pyne at Sun, 2005/09/11 - 5:00am

Well, one idea was to embed the python interpreter in KDevelop and then query scons directly via it's API. With the information returned from the API we could write out a SConstruct file that KDevelop could understand.

Unfortunately, this does not seem possible. SCons API is nearly impossible to understand and it doesn't seem that the relevant information (a list of sources, etc, etc) are maintained in any member variables we can get access to.

So, probably what mpyne said above ;)


By manyoso at Mon, 2005/09/12 - 5:00am

...for not choosing CMake!


By Carlo at Mon, 2005/09/12 - 5:00am

Agreed.. I like Cmake, but it's syntax is hideous.. I had bad experience with that while building VTK (Visualization Toolkit)


By smt at Mon, 2005/09/12 - 5:00am

Would it make any sense to move amaroK 1.4 to bksys? Could we do so and keep autoconf? Sounds like most of the TODO list for bksys is configuration related.

Of course, I can't imagine amaroK is that much more complicated then kdegames if at all. Its just that there are several features that have to be enabled/disabled.

(BTW, codeine is another KDE project using scons).


By Ian Monroe at Mon, 2005/09/12 - 5:00am

There is support for enabling/disabling features in bksys, we use it in Rosegarden. It can be improved, but it's perfectly useable as it is. As for using autoconf in parallel, yes, no problem, both are independant, but there's little reason to do so.


By Guillaume Laurent at Mon, 2005/09/12 - 5:00am

Just for people keeping score, Max Howell has begun writing the SConscript's for amaroK.


By Ian Monroe at Wed, 2005/09/14 - 5:00am

Hello,
what is about the Java based Apache ant? Why is this not suitable?

Florian


By Florian at Mon, 2005/09/12 - 5:00am

Because it is bullshit.
Because it requires a JVM.
Because it makes average cases easy, some simple cases very hard and really big cases almost impossible. ;-)

More seriously, ant misses generic dependency handling which is a big drawkack, IMHO.


By Jean at Mon, 2005/09/12 - 5:00am

and you really, really, *REALLY* don't want to manually edit build files written in xml.


By Guillaume Laurent at Mon, 2005/09/12 - 5:00am

Hopefully this will help with my goal to use KDevelop on HP-UX. The main problem is -no-undefined being passed to libtool, which on HP makes it build static libs (hard to have a plugin thats built as a static lib).

Will BKSys be eliminating libtool?


By Tom Matelich at Mon, 2005/09/12 - 5:00am

Yes it does.


By Anonymous at Tue, 2005/09/13 - 5:00am

You always need to edit the generated libtool to not use the wrong options for -no-undefined. I've used a simple script.

I tried making libtool people fix it, but they refused to understand what I told them.


By Allan Sandfeld at Tue, 2005/09/13 - 5:00am

Try again. The libtool people have grown very responsive to bug reports & patches in the last months. There was a case where people were complaining that libtool was eating up build time during the linkage of gcj in the gcc tree for example. Lots of talk about ditching libtool. Then the libtool people fixed libtool to spend next to nothing on that task, and everyone moved on.

I find that trying again every now and then is a pretty good way to get my patches & proposals into libtool.

cheers,
dalibor topic


By Dalibor Topic at Sat, 2005/09/17 - 5:00am

I've been using bksys for my projects for a while now. It is a wonderful build system it brings a smile to my face to use it where using automake/autoconf would always bring my hours of despair. Great choice to those who made it. Grats to the developer :-)

Also it means the source tarballs shrink dramatically. My tarballs for Filelight went from 600kB to 60kB. Bandwidth savings for all!


By Max Howell at Tue, 2005/09/13 - 5:00am

> My tarballs for Filelight went from 600kB to 60kB. Bandwidth savings for all!
So you have to finish drinking coffee in 0.03 rather than in 0.3 seconds to keep pushing? :)

lg


By Erik at Tue, 2005/09/13 - 5:00am

Lately we were doing some testing to the scons build too we've but found it very difficult to work with since it was not so java oriented. Therefore I was wondering how did you handle the Java build in your system?. Did you implement the required changes/addiotions yourself or got some adequate reference from the Scons dev team?


By Shiri Danan at Tue, 2005/09/13 - 5:00am

If you allow, I am not a big fan of stuff that requires Python. It was a big pain to have Python stuff together with GNOME and I hoped that KDE would stay entirely free of it, not just runtime, but also buildtime. I feel a bit sad to say the truth because I've been following the Scons discussion on IRC for quite a while now amd came to the personal conclusion that Scons has absolutely no advantage over Auto* not to mention that this requires a new learning curve to the developers to keep on with the stuff and then in what aspects does it make stuff easier ? I was seriously considering keeping KDE as my main desktop of choice once I set up a new source based setup of my Linux system and Python was one of the main reasons that I wanted to get rid off. 40 mb of Python trash for just some config files - what's next MONO ?


By ac at Tue, 2005/09/13 - 5:00am

* Size: 100KB (miniSCons included) vs 600KB (autotools)

Not to forget 40mb of Pyhton which makes it 40mb and 100kb. Autotools is still smaller :)


By ac at Tue, 2005/09/13 - 5:00am

But you'll quickly reduce that saving once you start generating or distributing configure scripts, and Python is installed as standard on most distros, anyway. Don't you think it's better to pay the cost up front rather than every time you download or create a new package?


By Anonymous Python at Tue, 2005/09/13 - 5:00am

> and Python is installed as standard on most distros, anyway.

This is placebo talk and a very bad excuse too. On most modern distros we have MONO and JAVA installed too, so why not make it a dependency for KDE as well ? Oh and even more distros and developers use auto* because it's a defacto standard in professional building of source based software - but no, let's make Scons a new unnecessary dependency because someone said so.


By ac at Tue, 2005/09/13 - 5:00am

On the other hand it removes the dependency on Perl .. and also M4, GNU/Make, libtool, automake, autoconf.

What KDE or autotool-based project are you maintaining once again ?


By anonymous_curious at Tue, 2005/09/13 - 5:00am

> On most modern distros we have MONO and JAVA installed too, so why not make it a dependency for KDE as well ?

Because there's no compelling reason to do so. Scons is a compelling reason to depend on Python. Qt is a compelling reason to depend on C++. Et cetera.

Your excuse for the use of autotools of, "everyone else does" is worse. If scons is significantly better, then it should be used, and the fact that most distributions ship Python simply shows that the Python dependency is a non-issue for the great majority of people. Why aren't you lambasting KDE for moving to Subversion? Most other project haven't, and CVS is like a de facto standard.


By Dolio at Wed, 2005/09/14 - 5:00am

a) We don't know if Scons is really that significantly better. The reasons for switching away from autotools was because a significant amount of KDE developers were not willing to read the manuals so they could set up their stuff correctly.

b) Python may be shopped with most distributions and I don't deny that but it's not required on most distributions. It's an option to be installed for people who desperately need it to hack up some small scripts.

c) Python will not be a build dependency anymore because as soon as Python is detected by the KDE scripts as soon it starts to compile optional python files too and installs them (which is not wanted).

d) If you read the autotool documents correctly then you would figure that autotools has to serve on a lot of POSIX compliant systems with dozens if not hundrets of different toolchains, compilers, headers and whatever. It's not limited to linux, it's not limited on windows, it's not even limited on operating systems such as MorphOS, AmigaOS (which are totally NON POSIX). That are the big advantages of autotools. You enter configure and then make afterwards. But you of course know perfectly the difference between scons and auto* since you never ever read a damn doc about it.

So now name up some advantages of scons over auto* and I will reply in a few hours to crush all of them down.


By ac at Wed, 2005/09/14 - 5:00am

> So now name up some advantages of scons over auto*

not only does scons not make me cringe every time i look at it, but i can even reasonably hack on it without become a scons wizard.

it also means that instead of every 3rd party app shipping a couple of megs of configuration structure (have you SEEN how big configure, aclocal.m4, acinclude.m4, etc is these days?!), they can ship a tiny little script.

when your app is only a few hundred KB (if that) of source code, the current situation is just silly and a bandwidth waster.

scons does the job, does it well, does it with less black magic and does it smaller.


By Aaron J. Seigo at Wed, 2005/09/14 - 5:00am

"The reasons for switching away from autotools was because a significant amount of KDE developers were not willing to read the manuals so they could set up their stuff correctly."

Heaven help anyone who has to read the autotools manuals! I've done a fair amount of autotools work and whilst I see the benefits in small scale projects, having to write shell script-like code with bizarre tricks that only seasoned bash coders find acceptable is not exactly how I envisaged spending the 21st century. Of course, most autotools-based projects blow most of the benefits away by using automake, thus generating a configure script with 20000 system tests just to compile a "hello world"-level program.

"Python may be shopped with most distributions and I don't deny that but it's not required on most distributions. It's an option to be installed for people who desperately need it to hack up some small scripts."

The increasing recognition of dynamically-typed programming languages is another trend that passed you by, then?

"But you of course know perfectly the difference between scons and auto* since you never ever read a damn doc about it."

I've read the "damn doc about it" and, as I said, it has its place. But given that SCons has a certain amount of heritage in a project to make better and more usable development tools, and given that is extensible using a modern programming language, I'd suggest that you seriously reconsider those advantages whether you seek to publicly (and ridiculously, in my view) "crush all of them down" or not.


By The Badger at Wed, 2005/09/14 - 5:00am

> The increasing recognition of dynamically-typed programming
> languages is another trend that passed you by, then?

You mean (and I am trolling here) the same trend that KDE keeps passing such as dbus, hal, libnotify, gstreamer, cairo, glitz, libburn, pkgconfig, poppler, startup-notification and all the other standards found on fdo ? But yet create their own set of stuff to avoid being compared with GNOME ?

If I recall back times when I read some archives of XDG ML then well, I find really curious reasons raised by KDE devels to NOT USE them. The same justified reasons I accept people to dislike something because its from the evil opponent the same way I would expect people to accept my dislike of Python as depedency. Maybe both reasons are braindead but they are at least reasons, the one has those the others has others.

I don't want to bash or attack anyone here because I believe everyone has a good right to have his or her own opinion but please don't make it sound like Python is the trendsetter in languages. So why not using MONO to develop core KDE apps ? Anyone missing the trend of .NET and C# ?


By ac at Wed, 2005/09/14 - 5:00am

> [...] and all the other standards found on fdo ?

http://www.freedesktop.org

"freedesktop.org is not a formal standards organization"
"Unlike a standards organization, freedesktop.org is a "collaboration zone" where ideas and code are tossed around"


By Anonymous at Wed, 2005/09/14 - 5:00am

Saying that people don't use auto* because they are not bothering with the manuals is stupid. It's like saying people prefer wearing shoes because they are not bothering to form calluses in their feet.


By Roberto Alsina at Thu, 2005/09/15 - 5:00am

Well, maybe it wouldn't be such a bad idea to require Mono, Java, Ruby and Python. >:-)

At least it would open up the KDE platform to developers who use those languages and environments, and it would take away the pressure to use C++ for everything just because the other bindings are "optional".

Let me ask you a question: if Scons was written in Perl, C or C++, would you still have the same reservations? I believe you would, and this makes your arguments about Python secondary to the ones you're trying to make.


By Anonymous Python at Wed, 2005/09/14 - 5:00am

Except you don't have to distribute python along with your software, while you do have to distribute the auto* generated files. Come on, a build-time dependency on python is hardly a problem in practice.


By Guillaume Laurent at Tue, 2005/09/13 - 5:00am

> Come on, a build-time dependency on python is hardly a problem in practice.

It's not just a build time dependency.

;--- reply from another comment
c) Python will not be a build dependency anymore because as soon as Python is detected by the KDE scripts as soon it starts to compile optional python files too and installs them (which is not wanted).


By ac at Wed, 2005/09/14 - 5:00am

*sigh*. Look, if you really want to save every cpu cycle and HD block you have, I suggest you switch back to twm. Better yet, stick to the console. Your complains are really not making any sense beyond "python baaaad, whaaaaa".


By Guillaume Laurent at Wed, 2005/09/14 - 5:00am

You are not making any sense either, you see that I dislike Scons because of Python - which you seem to have an even bigger issue with - and you start to discredit my comments by making silly replies.


By ac at Wed, 2005/09/14 - 5:00am

That would be only silly if that weren't possible to disable.


By ac at Thu, 2005/09/15 - 5:00am

Either you are misinformed, or your distribution has terrible python packages.

My own research on my local system shows:

clee ~ % aptitude show perl autoconf automake1.9 m4 libtool | grep 'Uncompressed Size'
Uncompressed Size: 11.5M
Uncompressed Size: 1507k
Uncompressed Size: 1602k
Uncompressed Size: 348k
Uncompressed Size: 2380k

That's about 16MB for perl, m4, autoconf, automake, and libtool on my system.

clee ~ % aptitude show python2.4 | grep 'Uncompressed Size'
Uncompressed Size: 9191k

So, it's a little over half the size of perl + m4 + autoconf + automake + libtool.

Plus, you claim that the autotools work perfectly on every system ever, except that it's patently untrue; the vast majority of the problems our porters have on MacOS X and Win32 systems are directly because of autotools.

Benjamin Reed, the main developer behind the MacOS X port, said in this message (http://lists.kde.org/?l=kde-core-devel&m=111866452210421&w=2) on the Build system for KDE4 thread (http://lists.kde.org/?t=111863688400002&r=1&w=2):

"Writing tests in scons/etc. is considerably easier than doing it in M4
and shell."

Jaroslaw Staniek, the main developer on the Win32 port, said (http://lists.kde.org/?l=kde-core-devel&m=111866757017220&w=2) in the same thread:

"SCons would be my favourite. Among others, there's one more advantage: with python scripts I wont run off with win32 PIDs on cygwin. For now oferflowing PIDs counter forces me to (sic!) reboot."

So the "BUT THE AUTOTOOLS ARE CROSS-PLATFORM!" argument falls on its face. They're unusable on non-UNIX systems for real projects, especially ones the size of KDE.

There are other serious issues with the autotools, as well, which is the main reason that we're interested in moving away from them. Such reasons include the fact that the autotools make it very difficult (if not impossible) to properly support parallelized builds, and the fact that they are difficult and unintuitive to use, especially "properly". So much so that you have to read "dozens of documents" on them in order to make use of them.... (half of these documents, incidentally, are for obsolete versions, and the behaviours have changed significantly in some aspects depending on the version that you're using.)

So, to sum up:

1) The autotools have serious issues with their "cross-platform" support
2) The autotools lack features we desire, such as easy extensibility and easily parallelizable builds
3) The autotools use a variety of languages, none of which is simple or well-known by the vast majority of our developers.
4) The autotools actually take up *more* space on disk, both in the installed packages required and in the amount of space required for our autotools infrastructure (the 600kb vs 100kb argument)

I look forward to you "crushing my arguments into dust" though. Try to remember to cite actual sources, instead of just using rhetoric.


By Chris Lee at Wed, 2005/09/14 - 5:00am

Pages