KDE to Migrate to bksys/SCons Build System

At the build system BoF at aKademy it was decided to start moving the KDE 4 build system from unsermake to the SCons/Python based system bksys. To find out more about this important future technology, KDE Dot News talked to its lead developer Thomas Nagy about the reasons behind the change and what it will mean for KDE developers.

Thomas is ita on IRC

Please introduce yourself and your role in KDE.

My name is Thomas Nagy (I am also known as 'ita' on
freenode). I have spent some time writing some KDE
utilities (solvers, small games) and adding
some features to KDE applications such as KOffice and Kalzium
when i was still a student. Since i graduated i have
spent more time trying to create complete applications
like kdissert, a mind-mapping tool for the creation of
documents and its build system bksys.

What is SCons and what is bksys?

SCons is a software construction tool which is used to
build and install programs. It is written in Python
and its design is modular, making it easy to extend
for use with unsupported compilers or complicated
build rules.

bksys (as in Build Kde SYStem) extends SCons and aims
at replacing the autotools. It has been written in the
main place for compiling and installing kdissert and
became more modular over the time. The most important
feature is the object-oriented style API that can be
accessed from either Python code or throughout XML
description files.

Some more documentation describing the framework can
be found in the the slides from my aKademy 2005 presentation.

Why is bksys separate from SCons?

bksys aims at replacing completely autotools for
common projects, so this goes way beyond SCons' main
goals.

In which way are they better than autotools?

There are many improvements over the autotools, but I
will try to describe the most striking ones:

  • Reliability: targets are rebuilt if compilation
    flags are changed or if md5sums of the files changed
  • Size: 100KB (miniSCons included) vs 600KB
    (autotools)
  • Simplicity: one language (Python) versus at least 3
    (m4, posix sh, make) for autotools (GNU/make,
    automake, autoconf, libtool); no intermediate file to
    generate for the compilation process (Makefile.in,
    Makefile)
  • Configuration: the configuration process is fast,
    easy to control and to modify
  • Flexibility: SCons can be extended easily thoughout
    modules. Besides, helpers can be written easily to
    help programmers (take all source files in a
    directory, or check automatically the consistency of
    the builds for example). SCons also has excellent
    support for win32 and OSX platforms, it handles many
    languages like Java or Fortran out of the box.
  • Evolution: the API and the code written for
    compiling the targets are re-usable so projects using
    it will not be locked.

How do they compare to unsermake?

Unsermake is a replacement for make and automake only.
Bksys replaces the whole autotool chain.

Can you summarise the build system BoF at aKademy?

Several candidates were evaluated. QMake is way
too limited and cannot be used at all. Unsermake is
not to be kept, as it is only a partial solution and
emulating the behaviour of automake and make consumes
a lot of resources. Two solutions remained in
competition for the future KDE build system: CMake and
SCons.

CMake is a Makefile generator like QMake; it uses a
macro language which is known to present several risks
as non-standard targets arise frequently in KDE. On
the other hand SCons is much more flexible as it is
based on Python, and was proven to work on fairly
complex applications already (KDE Games, Rosegarden,
..). SCons was then chosen and the conversion of
kdelibs 4 has started.

So what work needs to be done now?

So far, the main goal of bksys has been to replace
autotools for usual KDE projects. Now for kdelibs the
situation is a bit different as the tool will be the
base of all KDE applications, so a new framework has
to be designed, in particular:

  • We have to agree on a new set of API (stardard
    targets, naming, features...)
  • New bksys modules have to be written for detecting
    the system configuration (full autoconf replacement)
  • Some work has to be done for writing config.h files
  • More helpers will have to be written to make
    programmers lifes easier (scan for sources
    automatically..)

At the very moment, the bksys scripts have been added
to KDE SVN, and 3 main directories in kdelibs 4 are
already compiling (dcop, libldlt, kdefx). There is
some documentation in trunk/KDE/kdelibs/bksys/design.
The main aspects of the future framework are
presented, and some questions are left opened to
discuss.

In the past, the kdegames module was converted in
about 2 days, but for kdelibs it will take a lot more
time as even after the framework is complete some
reverse-engineering will have to be done on the
Makefile.am to achieve the conversion. I am striving
at the moment to get a first version of the framework
ready (most configuration modules complete and a few
folders compiling) to show how the new system will
look like and the main advantages of using it.

Who is going to do the work?

.. or who is working on it already? Three people have
committed code on bksys in svn so far, Simon
Haussmann, Stephan Kulow and I. Interesting ideas have
been raised on the IRC, and I am awaiting
more comments, when the first version of the framework
is ready for testing (a request for comments will be
posted).

Are non-KDE projects taking up SCons/bksys?

Most projects using SCons do it directly and use their
own framework, like Blender or Rekall.
The projects using the bksys layer i know of are
almost all KDE-oriented: Rosegarden, kio-locate,
skim...

Dot Categories: 

Comments

by ac (not verified)

But these tools m4, libtool, autoconf, automake are existing for how many years now ? You can ask basicly everyone to help you with it, you can even go to the next chinese fish store to ask some linux guru to help you while you probably won't find that many people helping you with Scons problems. But then again, Scons isn't really my problem. Python is (as if I haven't said this two dozen times now).

by The Badger (not verified)

"But these tools m4, libtool, autoconf, automake are existing for how many years now ?"

For so long that we're all born with inate knowledge of them?

"You can ask basicly everyone to help you with it, you can even go to the next chinese fish store to ask some linux guru to help you"

Dream on! Makefile and autotools gurus may be able to help you, but they're nowhere near as common as you make out. Even if the Internet is littered with autotools files, as you suggest elsewhere, that doesn't necessarily make troubleshooting any easier, as anyone with any experience of TeX authoring will tell you.

"while you probably won't find that many people helping you with Scons problems."

Sure, specific SCons experience is likely to be limited, but at least there's a chance that people will either get the language or get up to speed with it, rather than getting bogged down with the usual horseplay that shell-inspired toolsets entail.

"But then again, Scons isn't really my problem. Python is (as if I haven't said this two dozen times now)."

If only to contradict yourself in two dozen other places. But to distill this objection to Python down, it would appear that you see the installation of a tool you don't use as "wasteful" - in which case I imagine that you roll your own minimalistic Linux distribution in order to avoid all the other wasteful stuff that gets installed. Or don't tell me: you've said all this but actually run Fedora Core?!

by Morty (not verified)

>You can ask basicly everyone to help you with it,

I'll take your word for it. I encountered a small case a few days ago, can you please explain it to me. Why does it behave this way? I had a build error in the kdeaddons package when building noatun-plugins/blurscope. For some reason it decided to try linking to KDE files(libkdeprint,libkparts etc) in /usr rather than where KDEDIR is(/opt). I have some old KDE binaries in /usr as a backup solution when SVN builds to /opt fails, but no -devel RPMS so it does not find any .so files to link with there. Making one small change to the Makefile.am made everything build correctly. So please explain to me why these two lines works differently?
"noatunblurscope_la_LDFLAGS = $(all_libraries) -module -avoid-version -n o-undefined `$(SDL_CONFIG) --libs`"
"noatunblurscope_la_LDFLAGS = -module -avoid-version -n o-undefined `$(SDL_CONFIG) --libs` $(all_libraries)"

Why does one make it link against /usr and the other against /opt?

by ac (not verified)

If I understand your problem correctly which I assume I do, then this is clearly the old known precedence problem (where to search first).

But sadly it's not just libtool or gcc alone. They search in the default paths for libraries first. So if you have older libraries sitting in /usr and have the new ones compiled and installed in /opt/kde then you can be sure that libtool and gcc seeks for libraries in /usr first rather than in /opt/kde.

Something you can not even solve with LD_LIBRARY_PATH. The correct solution would be to remove the old libraries in /usr anyways to avoid the problems. I encountered these issues some years ago when maintaining my own well known buildscript for another desktop environment.

This is no Makefile issue, nor is it a configure or autotools issue. It's an issue of precedence of where the compiler and linker or libtools is searching first.

IIRC JHBuild on freedesktop.org provides a patch for libtool to search in ${prefixdir}/lib first. But then these bugs are solvable it's just a matter of time that someone writes a bugreport and sents it to the maintainers so they can add it. Nothing wrong about that.

by Roberto Alsina (not verified)

If the problem is old and known and there is a patch around, why is it necessary to write a bug report and send it to the maintainers?

by ac (not verified)

The patch comes bundled with the jhbuild package, I don't know why it wasn't forwarded and I never bothered to ask either. Could be that it was sent to them and rejected, who knows.

by Christian Loose (not verified)

> This is no Makefile issue, nor is it a configure or autotools issue. It's an issue of precedence of where the compiler and linker or libtools is searching first.

That doesn't answer the question, why the change fixed the problem. Why does putting $(all_libraries) at last posititon change the precedence?

by ac (not verified)

I can not answer this without knowing the whole problem case. This also requires to know what system he uses, what's installed on it, what he really did, etc. Giving an direct answer to this over dot.kde.org without knowing all facts would be far to unserious. But all I know so far is that this is not an autotools issue. It's of course clear that if I force the precedence to be differently that stuff may work as in this case.

by Morty (not verified)

>The correct solution would be to remove the old libraries in /usr anyways to avoid the problems.

No it's not the correct solution, for several reasons. First the ability to have several versions of KDE installed is a feature, and I don't think removing the stable version for unstable/svn version would be particularly popular by some of the users on this box(me included:-).

>This is no Makefile issue, nor is it a configure or autotools issue. It's an issue of precedence of where the compiler and linker or libtools is searching first.

How come simple changes to Makefile.am make it work or not, without any other changes anywhere. If it had been that simple the rest of the package would also have had failed building, and not only the one lone directory. As $(all_libraries) are positioned both places in the rest of the package. And the same can be said of the 10 other KDE source packages which builds correctly in the same environment, not having any precedence problems. Personally I think such unpredictable behavior in the build systems clearly confirms it has major issues.

by Reinhold (not verified)

To quote Mozilla's webpage on why they're not using another build system like scons or ant:
"Mainly, because no one has implemented these systems for Mozilla."

by tfry (not verified)

Though this thread is probably long since dead, I can't resist chiming in for a rant:

> There are good documents people should consider reading, if you are not
> willing to spent time reading the auto* documents then you are not suited to
> make good files that work perfectly with your software. The problem with KDE
> rejecting auto* is that everone I've met so far complained about auto* but
> in reality it's because no one really spent time reading the manuals, they
> all want some quick hacks so it works and they start to copy stuff from
> other modules and wonder why it doesn't really work. How comes millions of
> open source and free software except the stuff inside KDE works perfectly
> fine with auto* ?

And this is precisely, where the problem is. I want to develop cool software. I don't want to spend hours wrestling with makefiles. I want them to just work. And when they don't work, I want to be able to understand, why not, and fix them, without having to read a week's worth of docs.

Also, I do want to just "copy stuff from other modules", maybe not blindly, but the fact alone that this is non-trivial is just plain bad. I have literally spend days of my life pondering layer after layer of files, auto-generated from files, auto-generated from files, each generation process using a different language - all just to track down some small annoying bug preventing the build on some other people's systems (where it went fine for me, so much about portability).

And yes, I have read the docs. Not from start-to-end, I admit, but trying desparately to find a pointer to my problems. Generally with little luck. In the end, so far, I've always found a solution, so, yes, I guess that goes to say, auto* can do about anything you want, if you just use it right. And millions of projects have somehow managed to tame the beast, eventually. But at what cost?

Admittedly, I don't know scons much, and I'm sure it has its quirks, needs testing on exotic architectures, etc, maybe even one or two more years to be considered mature. But at least: It's consistent within itself: Using only one language instead of four. It's supposed to be modular - hooray! - finally we will actually be able to understand _and influence_ what's going on in the build process without years of studying auto-stuff.

Modularity and consistency - if you can't achieve those in your framework, I'm sure any programmer will tell you, then it's simply flawed. Better ditch it sooner rather than later.

by anonymous_curious (not verified)

> I was pretty pissed when I found out one had to have Perl installed to get Gtk and GIMP to build.

Did you know that building KDE programs from the repository in KDE3 required Perl and M4 ? Did you know that unsermake was (and still is) required to build KDE4, and that it was Python-based too ?

> Has Jam been considered as an alternative?

Yes, and Ant too, and many other tools.

by Me (not verified)

>Did you know that building KDE programs from the repository in KDE3 required Perl >and M4 ? Did you know that unsermake was (and still is) required to build KDE4, >and that it was Python-based too?

I don't build KDE from repository, but from the standard packages, like most people who build KDE do. Since I've never had to use 'unsermake', I didn't even notice it so I don't feel concerned really.

> Yes, and Ant too, and many other tools.

Did you check out Haiku's build system?

by non-developer (not verified)

Like many people I don’t develop KDE but I do compile it in order to use it.

Will I still be able to do:

./configure
make
make install

by anonymous (not verified)

Yes.

There are wrappers so it does not change anything for users, except for the "make -f Makefile.cvs" (before configure) which is removed.

by Andre Somers (not verified)

It would be really advantagous to me if it made the configure faster too. It keeps on checking stuff over and over again for each program I compile, and when you're compiling KDE, there are a LOT of programs...
I just hope KDevelop will make it easy to work with this buildsystem and convert programs from auto* to this new buildsystem.

by ac (not verified)

What you don't understand about autotools and why it requires so long for configuring is that you limit your view on Linux based systems only. Forgetting all the eventuality of other systems such as solaris, windows, cygwin, morphos, amigaos, etc. They all have different things that needs to take account for. It's a huge complex thing and all eventualities needs to be taken care of. So the slowness as you see it is no slowness of the scripts, it's the slowness because all eventualities has to be taken into account. That's why autotools is so strong and that's why it works perfectly on a huge number of systems, even non POSIX and non UNIX systems.

by Roberto Alsina (not verified)

Weird. When I tried building a bunch of auto* based software on Solaris 9, I ended having to install bash, GNU m4, perl, GNU awk, and GNU make or they wouldn't even run their configure scripts properly.

It's probably just me, I suppose.

by yaac (not verified)

the configure script which is supposed to be distributed with a package requires only a bourne-compatible shell plus some shell tools. it should _not_ require autoconf, automake, perl or m4.

by anonymous_curious (not verified)

I have worked on HP-UX and Tru64 and autotools rarely worked as expected. Bash or zsh or coreutils would not build properly on Tru64 for example.

by Ita (not verified)

> It would be really advantagous to me if it made the configure faster too. It keeps on checking stuff over and over again for each program I compile, and when you're compiling KDE, there are a LOT of programs...

Indeed. We are thinking at the moment on a re-using the Qt data somehow (qmakespec).

> I just hope KDevelop will make it easy to work with this buildsystem and convert programs from auto* to this new buildsystem.

A proof of concept for kde3 is available at:
http://freehackers.org/~tnagy/pmanager-0.6.4.tar.bz2
Kdevelop only needs to create an xml file that bksys can parse to build the targets.
I have exposed api for writing a Makefile.am parser but it looks like developers were not really interested in doing it.

PManager is too limited at the moment, and it will be rewritten once the KDE4 port is ready.

by Backwards (not verified)

Everyone should use sendmail because it's always been used and if you can't understand the config files then you obviously are too dumb to read the various documentation. Distros should never ship non-standard MTA's such as postfix even if they are easier to set up because everyone already knows the sendmail config files and you could ask anyone on irc and they'd be able to answer it for you.

by regeya (not verified)

LOL - well-put!

I'm not much of a Python fan, but if the KDE devs can decide on it, then, oh well! That's their decision. Now we get to look forward to years of complaining and bitching about a choice of base dependencies. Heck, maybe RMS will find some obscure reason that the Python licensing scheme violates the GPL so people can hate KDE all over again. *wink*