I was recently hired by SuSE to focus on optimizing KDE, and as part of my work, I have written an analysis of KDE memory usage based on Waldo Bastian's previous paper concerning GNU/Linux linker issues and KDE startup-performance. As it turns out, the linker issues examined by Waldo do indeed have an effect on the memory usage of C++ applications compiled and linked under GNU/Linux. The most important number from this paper: About 650KB of memory wasted per KDE application not launched via KDE Init. I have contacted the GCC/binutils developers with this information along with some suggestions, and there has already been one response from Jakub Jelinek (of Red Hat), developer of the prelinker tool. Another mail has been sent to kde-core-devel with a suggestion as to how we could probably deal with the issue until a full GCC/binutils-based solution is available.
Comments
How will this affect us non-GNU users? We who don't use GNU ld, binutils, glibc and so on. Will anything be done to increase speed/decrease memory usage for non-GNU platforms? Will objprelink be available for Sparc?
Lots of questions :) Hope you don't mind if a question is a little bit stupid, I don't have that much technical knowledge.
:: Will objprelink be available for Sparc?
Assuming you're running Solaris (not a perfect assumption, but you don't specify), *if* gcc and KDE are optimized for each other, then you would probably preferentially use the gcc compiler and linker. Unless you have a reason why not to (I haven't used Solaris heavily since 1995ish, and gcc wasn't really considered an option back then at the place I admined, so I don't know how well it performs on a Sparc).
Otherwise, this is (simplisticly) a C++ function pointer handling issue with the GNU linker - it may not even be an issue with the Sun supplied linker. Or it may be an even worse issue. Or there (most likely) may be an entire different set of issues when it comes to optimizing. Regardless, the code isn't being changed. It's the simple fact that the KDE team is working so hard on this project that they are identifying performance bottlenecks in system components. When that happens, you go to the person supplying the system component (in this case the gcc dev team), and present your case, preferably with a proposed solution. That's been done. If the Sun compiler/linker has performance bottlenecks as well, it will have to be profiled and presented to Sun.
--
Evan
>>It's the simple fact that the KDE team is working so hard on this project that they are identifying performance bottlenecks in system components. When that happens, you go to the person supplying the system component (in this case the gcc dev team), and present your case, preferably with a proposed solution. That's been done. If the Sun compiler/linker has performance bottlenecks as well, it will have to be profiled and presented to Sun.
Are the GCC/GNU tools a part of KDE?
Of course I understand that the KDE team want their software to perform as well as possible, but why just on GNU/GCC based platforms. Isn't non-GNU platforms important for the KDE team?
Maybe my questions should not have been adressed here, since they don't directly has anything to do with Lubos Lunak's work. That maybe has created a confusing situation. I apologize for that.
> Are the GCC/GNU tools a part of KDE?
No, they are separate projects.
> Of course I understand that the KDE team want their
> software to perform as well as possible, but why just
> on GNU/GCC based platforms. Isn't non-GNU platforms
> important for the KDE team?
Now that's a loaded question. That's like a parent who buys a shirt for a child and the other child says, don't you love me?
Read the article. SuSE hired him to make these improvements. SuSE != KDE. SuSE is a Linux distribution which uses Gcc/binutils.
Another obvious fact is that the vast majority of KDE users use gcc/binutils. Not only because most users use Linux, but even on other platforms many users use these tools. So obviously if you are going to improve a subsytem you would focus on the subsytem that helps the most people.
Now, are these other platforms important to you? If so, why don't *you* do something about it? Open Source is not about criticizing people who do work because they don't do the work you prefer.
> Maybe my questions should not have been adressed here, since
> they don't directly has anything to do with Lubos Lunak's work.
Exactly. Why don't you congratulate him and SuSE on their great work rather than complaining and accusing because they have not done more for free.
(BTW, The first question was rethorical...)
My questions was just questions, not demandings. Just plain questions.
>Now that's a loaded question.
I don't understand. My questions was not at all about jealousity, just curiosity!
I'm not accusing anyone. The only one who accuses someone are you accusing me for accusing SuSE and Lubos Lunak. I know fully why Lubos Lunak's work was made (to improve the SuSE distribution), so SuSE can sell more distributions, and make more money. Or is SuSE a non-profit organization? Or maybe a charity organisation? I don't think so.
Why is it so hard to understand that I'm not accusing anyone? Do you believe the worst of everyone? Man, I just was curious if the KDE team has any thoughts about improving KDE for non-GNU platforms. And just because I was curious, it doesn't mean at all that I demand improvements, as described earlier.
But: To Lubos Lunak and SuSE: Thanks for improving the KDE desktop.
Hope your improvements results in more SuSE distributions sold, so you can continue your contributions to KDE.
Best regards
Mats Eriksson, aka Loranga
The Solaris compiler most likely doesn't need objprelink. The only reason objprelink helped for the GNU tools was because the GNU tools were badly designed in the first place. One would hope Sun's Solaris tools are already performing prelinking or something equivelant. And if not, there's not much KDE can do about it. The KDE team can talk directly to the GCC team to fix the problems. I doubt Sun would listen to the KDE team.
Yes, non-GNU platforms are important for the KDE team, but they aren't as important as GNU, for the simple fact that more people run KDE on GNU, and more of the KDE developers run GNU themselves.
: Are the GCC/GNU tools a part of KDE?
No, nor are they required for KDE. That's like saying "I compiled KDE on Solaris, so 'Is the Sun C Compiler a part of KDE?'"
KDE is being developed in C++. If you're loking for some of mythical, ideal parity in support, and gcc worked fine, and the Sun C Compiler had a serious problem, would you prefer that the KDE team not address that?
It's the same thing with X - various X servers have different issues. KDE is written to be very X independant, while still supporting some nice features (like XFree's multi display xinerama feature). Would you prefer that KDE fail miserably on a multihead display just because the X server provided with AIX dosen't support xinerama extensions?
--
Evan
Well, the first question was rethorical... :)
This is all about standards compliancy, and how/why you should follow standards or
not. That's an issue which has been discussed a lot, not just in software
development. Why not take part of that on-going debate? I can recommend reading "The
Design of Inquiring Systems", 1971, chapter 9, by C.West Churchman.
A question is: How can loosely coupled teams like the KDE team or the XFree86 team
lead the development (and of course, the development of standards), compared when
large companies led the development, either alone or in consortiums together with
other companies?(1)
This is quite a new phenomenon in the free software world.
Example: XFree86 implementation of the X11R6.4 standard.
Taken from the x.org home page (http://www.x.org/current_honor.htm):
"The XFree86 Project, Inc is a non-profit corporation of the state of Texas, USA.
The primary charter is to design, implement, and distribute, as free software, an
implementation of the X Window System. The software implements the X Window System
specifications for many implementations of the UNIX ® operating system. All products
of The XFree86 Project, Inc, are freely available, and freely re-distributable.
Major supporters of XFree86 include Compaq Computer Corporation, Metro Link, Red Hat
Software, S.u.S.E. GmbH, and many others.
"
Voices are now heard among XFree86 users(2) that the X11 specification are old and outdated, and a X12 specification has to be made. What should XFree86 then do? Try to develop a X12 specification with the x.org consortium, or fork the standard and build an own
standard?
This discussion can be extended to the KDE project. If KDE wants to make Unix ready
for the desktop, then they have to be standards compliant, so it can be implemented on every Unix which also follow the
requiring standard(s)(3). If KDE just want to be a hobby project and/or a artefact only running on Linux and some other platforms,
often required to install (non-standard?) GNU tools, then it is just to do what they want to do.
The most important thing is to understand why this is not a purely
technical/technological issue, it is rather a philosophical issue!
(1) Some companies are not interested in following standards at all, somebody says
MS is one of them...
(2) Just rumours. Tell me if these rumours can be wrong, and why.
(3) I guess POSIX is one of these standards, but please, don't be naive and focus on a specific standard, think of
standards in a higher, more philosophical way.
Regards
Mats Eriksson
: If KDE wants to make Unix ready for the desktop,
: then they have to be standards compliant
KDE is written in C++. where is it not standards compliant? Qt is written in C++, and the version that KDE generally links against is the X version, where is it not standards compliant?
: If KDE just want to be a hobby project and/or a
: artefact only running on Linux and some other
: platforms, often required to install (non-standard?)
: GNU tools, then it is just to do what they want to
: do.
Where does KDE have any dependencies on GNU tools or Linux? I've watched the developer list for long stretches of time (I'll admit I don't follow it right now), and I've often seen conversations along the lines of (simplified): "That CVS commit that was just made breaks the build in AIX" "Okay... how about that?" "That fixed it".
The core KDE developers watch for, and maintain Unix and X (as far as it goes on top of Qt) compatability. Trolltech seems to do the same. I really really fail to see where *any* of what you said is relevant.
The whole objprelink and gcc issue was this: the KDE team found a bug[1] in the GNU linker. They profiled and reported it. End of story. Of course KDE users using the GNU tools would be interested in this news - it means that their builds of KDE (and any C++ app similar to KDE) will have some nice optimizations. There's absolutely no issue of "standards compliancy" in either a "technical" or "philosophical" viewpoint.
: please, don't be naive and focus on a specific standard,
: think of standards in a higher, more philosophical way.
What standards? What the heck are you talking about? I agree that this is an interesting question to raise; the concept of KDE's direction is always in the air everytime someone asks about porting KDE to Qt/Embedded or something along those lines. But I fail to see how the objprelinker is related at all.
[1] It was a bug report in the sense that it was doing very slow runtime linking at inital execution. Probably not an issue for most programs... it was just more noticable in an environment where you're running lots of programs by clicking on an icon and waiting. A bug in the "wishlist" category.
--
Evan
:KDE is written in C++. where is it not standards compliant?
I haven't said that KDE is not standards compliant, but it has to be if it wants to "make Unix ready for the desktop".
:What standards? What the heck are you talking about? I agree that this is an interesting question to raise; the concept of KDE's direction is always in the air everytime someone asks about porting KDE to Qt/Embedded or something along those lines. But I fail to see how the objprelinker is related at all.
It's not related to objprelinker at all. This discussion was about standards in a general, philosophical way, that is, how/why KDE (or other free software projects) should follow standards or not. I'm not talking about a specific issue here, neither a past one, a current one or a future one.
I just wanted to extend the discussion above the everyday discussion. But I'm only accused of bashing KDE (or some KDE developers) for not doing a good job, which I think I haven't done. If someone thinks I have done so, I apologize.
If you (or someone else) wants to discuss how standardization work can/should be done, you can contact me through my e-mail adress.
Regards
Mats Eriksson
For me, this thread is closed. Just wanted to clarify that. But you can always e-mail me if you have anything to say.
I'm not sure. Maybe you should ask <*BSD developers/Sun/whoever develops your system>.
BTW, there's a difference between objprelink and prelink. Prelink is the tool developed by Jakub Jelinek which I used in the tests. For objprelink details, see the other dot.kde.org article (about 2-3 months old, it's called 'Faster startups' or similar).
What kernel did you use to test test this? Linux 2.4.x is notorious for reporting wrong values for SHARED. Rik van Riel seems to be working on a patch which will solve this (and many, many other) problems but AFIAK it isn't merged yet.
Even more: 2.4 doesn't even calculate it, I think because it can't be calculated in a good working manner anymore with the new VM-architecture (not sure about this, though). I guess this has been tested by implementing some method in kde_init to get the right values, or maybe just using a debugger / memprofiler.
I quote : "All the tests were done on SuSE 7.2 (x86) with gcc-2.95.3. I repeated some of the tests also on Mandrake 8.0 with gcc-2.96, which was the newest gcc version I had available, with similar results."
mandrake 8.0 : kernel 2.4.3
suse 7.2 : 2.2.4 and 2.2.19 (2.2.4 by default)
Whooooops, shame on me :-/ I must have missed it, thanks.
2.4.7 . And I don't think the values reported were wrong. libkdeui contains about 100KB unshared data (as reported by objdump --headers) and when linking against it, the unshared memory usage increased by about 100KB. (I hope that by SHARED you don't mean the value reported as 'shrd' by top. I mean the column in top called SHARED.) I also checked (some of the values) by checking /proc//maps .
please check the memory results on solaris or HP-UX before going ahead with this mainly because I dont completely trust linux to report the right results.
(would you want more than one judge in a race ?)
there are free tools from both SUN and HP to do this and GCC is supported on both platforms
regards
john jones
Much of what's reported in this analysis is the same as what Waldo found when trying to find the bottleneck in KDE app startup time. Waldo determined that simply linking kdecore would create 162 dirty pages (an x86 page being 4kB, so 648kB). In his paper, he was primarily concerned with the relocation phase which took some considerable time in loading KDE apps. This new paper is more concerned with the other sources of dirty pages in KDE apps, such as conflicts (which could be solved by the glibc and gcc teams) and data stored within the image of the library/executable that is modified at runtime (and therefore copied-on-write on Linux).
Testing on HPUX or Solaris would gain them nothing, as having 100% correct numbers mean nothing, only the fact that there are considerable numbers of dirty pages is enough to prove that something needs to be done, and it doesn't really matter one bit if you trust Linux's reports or not.
So for us linux (and other gnu) users this seems like a problem with
gcc and the linker.
Are there any alternativ compiler linker and ld.so that can be used
on Linux ?
There is a non-free C++ compiler from Intel, and they claim to be not compatible with C++ output from gcc, so linking seems to be different. I doubt that the speed improvements would justify porting KDE (and all applications) to their compiler though - buying a faster CPU is probably a cheaper solution...
You can find the compiler here: http://developer.intel.com/software/products/compilers/c50/linux/
If this is better than gcc, perhaps the the people who make the RPMs could get it? We don't all need to own the compiler, most probably use DEBs or RPMs.
Your entire system would have to be recompiled. You couldn't just get one RPM that had been compiled with the other compiler, since it's incompatible with GCC.
Ummmm...no.
You wouldn't see any benefit unless your entire system was compiled with Intel's compiler. Earlier in the thread it was stated that the compiler is incompatible with GCC. If this is true it can't use GCC-compiled libraries, which means programs compiled with it can't run on GCC-compiled systems unless they are statically linked, which is a really dumb thing to do most of the time. The supposed increased speed would probably be offset by the much larger binary size of a statically linked program. Especially with KDE programs, since a lot of KDE technology is based on shared libraries.
So you could, I guess, in theory get one RPM compiled with Intel's compiler. But the program contained in it would be bloated and memory-hungry, and probably slower for this reason. There would be no benefit unless your entire system including all your libraries was compiled with Intel's compiler, in which case you could do dynamic linking. However, you wouldn't be able to use any other currently available RPMs on this new system, so that wouldn't be practical either.
From a users point of view, I dont see what whats the big deal with 650 KB wasted per application. Anyone with a system configued with only little memory woud not be using KDE in the first place, they'd be using a more Spartin window manager or none at all right. I rather the focus be on maximizing speed instead of maximizing memory usage. Perhaps the two are correlated,I don't know, just my thoughts as a KDE user, and a guy with 512 Megs of RAM
You are just wrong.
Someone with a small system will use a lightweight windows manager and not KDE or Gnome, but he might still want to use some KDE apps and this is exactly the problem as those 650 KB are wasted only when there is no kdeinit in the background to start the application. So those 650 KB per applications will be wasted on those small systems that don't run KDE as their windows manager.
Don't forget that filling unnecessary memory takes time, thus reducing speed. And I'd love to see KDE run faster on my 40MB / 133MHz Pentium I. I don't intend buying a new computer any time soon.
I don't know if you've read much about Windows XP, but one of the big IT analysts (I think it was Gartner Group) recently said that 70% of PC users would have to upgrade in order to meet the minimum requirements for running Windows XP.
Now, if memory serves me right, Microsoft's minimum requirements for XP are a 300MHz processor and 128MB of RAM.
So, you can deduce from this that 70% of PC users either have a processor less than 300MHz or less than 128MB RAM - a huge, huge market that SuSE could easily tap into now that Microsoft has basically given up on them.
Personally, I run KDE at home on a PII-233 system with 64MB of RAM, and actually it's pretty speedy - but there are areas that could be better.
Kudos to SuSE for sponsoring this vital work. Whilst I'm not sure they can tempt me away from the joys of Debian on my home machine, when I have finally convinced my work to switch to Linux and KDE (not too much longer to go now, I nearly have them convinced!), SuSE will be top of my list of recommended distros.
PS if both SuSE and Mandrake included apt as the main software update tool, you might even convince me to switch my home machine :)
Try urpmi for Mandrake and Suse systems.
urpmi is ok, but it is really rather limited compared to apt. For instance, how do I tell urpmi to fetch a source package, all the dependencies (in source form) then build and install all the packages?
Or, having fetched a package of something from, say, cooker, and all the dependencies, then, when I update, get it to fetch the latest version of that package and its dependencies from cooker again, but only update the rest of the system to the latest _stable_ version?
All this, and a bunch more, is possible with apt - and in fact it's quite easy.
If I could use apt to update from release to release as well as the minor security updates that occur during the lifetime of a release, then I would be more than happy to pay a subscription for the service. I like Debian, but I admit it's not the easiest of distros to use, even with apt - if a nice user-friendly distro like SuSE or Mandrake had apt, then I think I'd be prepared to pay.
There you go, there's a nice new additional business model for SuSE and Mandrake. With apt subscriptions they could pull off the same kind of business model that Microsoft wants to move to, but with far more style, panache and power.
Apt is in Mandrake Contribs. It works quite well. Mandrake also have urpmi.
Yes, I'm aware that Mandrake does have it as an option - which is why I said 'apt as their *MAIN* software update tool' (with added emphasis). Whilst it does work, as a contrib rpm it won't be supported or integrated, so your average user just isn't going to use it.
See my other response as to why I think urpmi is rather inferior.
Yes, I'm aware that Mandrake does have it as an option - which is why I said 'apt as their *MAIN* software update tool' (with added emphasis). Whilst it does work, as a contrib rpm it won't be supported or integrated, so your average user just isn't going to use it.
See my other response as to why I think urpmi is rather inferior.
Yeah, you're right. I really wish that Mandrake would support apt like Conectiva does. Urpmi was created when apt wasn't available for RPM. Now that it is, why maintain it? Apt is clearly better, IMHO.
> From a users point of view, I dont see what whats the big deal with 650 KB
> wasted per application. Anyone with a system configued with only little
> memory woud not be using KDE in the first place, they'd be using a more
> Spartin window manager or none at all right. I rather the focus be on
But that's the point: people with a small system do not use KDE *because* it has such a large memory footprint (The rendering speed is limited by the X server, so using a spartan window manager will not yield higher speed).
If the memory footprint of KDE can be reduced, even people with a Pentium 133 and 32MB ram might be able to use KDE to browse the web.
And don't forget that these improvements will benefit not only KDE, but all C++ applications compiled by gcc. So this is a huge improvement for the "Linux Platform". KDE is just the first complex C++ application framework that exposes the weaknesses of gcc and binutils.
> maximizing speed instead of maximizing memory usage. Perhaps the two are
> correlated,I don't know, just my thoughts as a KDE user, and a guy with 512
> Megs of RAM
They are most definitely corellated. Having non-shared code that could be shared will affect the effectiveness of the 2nd level cache, and that is a huge performance factor.
> The rendering speed is limited by the X server
I just want to point out this project : http://www.directfb.org (and particularly http://www.directfb.org/xdirectfb.xml)
I think memory waste isn´t a outrage - but it shouldn´t be ignored. Memory is very cheap these days, but most computers have a quite low limit of RAM they can use. My box, for example isn´t old (PIII/500, < 2 years old), and it can´t use more than the 256 Meg i´ve installed. I would upgrade to 512 or 1 gig, but i can´t. I thing it should be fixed, even on boxes with a huge amount of RAM the memory should be used for usefull things. This isn´t as important than other things, like stability for example - but the developers should keep an eye on it. Ignoring it would be M$ style.
Just try KDE2 on a older box and it's quite obivious, even on FreeBSD, it's very very slow simply because there is far too much disk swapping to do simple things especially on system with < 96megs ram.
KDE2 takes almost 2mins to load on my P166MMX with 32megs ram running FreeBSD 4.3-RELEASE and XFree86 4.1 compared to Windows 95 which only takes 25secs to load everything from the kernel to the gui (FreeBSD takes around 20~30secs from when FreeBSD is booted to when the login prompt appears). Konqueror also performs very badly, taking 1mins to load, compared to Internet Explorer that takes only seconds to load, and surfing is extermenly slow especially compared to Netscape Navigator 4.x (under Windows 95) which has a very old rendering engine which is very uneffective in rednering modern webpages which use lots of tables.
The bigest problem that causes all of the above problems is KDE2's memory usage. Loading 2 copies of Konqueror, one copy of Kate can see memory usage ballon up to over 64Megs total system memory (32Megs RAM + over 32Megs swap) causing the system to swap nonstop, and it's even worse when you load more copies of Konqueror and other KDE programs.
Since many people (and companies) are using systems with only 64megs ram (or even 32megs) it would be a good idea to opitize KDE by cutting down memory usage across all platforms (instead of focusing on Linux's problems which are THEIR problem in the first place) to encourage people to migrate from Windows over to UNIX[-like] platforms like FreeBSD, NetBSD, OpenBSD, Solaris, Linux and other systems.
THE PROBLEM IS THAT KDE WILL NOT WORK WELL ON MACHINES THAT MANY PEOPLE WHO WOULD LIKE AN ALTERNATIVE TO MSWINDOWS OWN! LINUX LOSES OUT IF NEW USERS GET FRUSTRATED AND ARE NOT INTERESTED IN A SYSTEM THAT HAS A STEEP LEARNING REQUIREMENT.
I'M A NEW USER THAT UPGRADED TO 850 CPU AND 512 MEGS RAM TO GET SO SO PERFORMANCE, BUT I'M DETERMINED TO GET ALONG WITHOUT A MS OS IF I CAN.
MY OLD SYSTEM WAS A 266 CPU AND 64 MEGS RAM AND KDE WAS TROUBLE WITH EVERY LOG-IN. I ONLY USE THE MACHINE FOR E-MAIL AND INTERNET ACCESS AS MOST WINDOWS USERS WOULD. WINDOWS 98 DID NOT EVEN RUN WELL ON THE OLD SETUP.
I USE MANDRAKE LINUX AND IT COMES CLOSE TO BEING THE ANSWER I AM LOOKING FOR, BUT WILL I BE ABLE TO MASTER THE NON GUI PART? I'M NOT LITERATE IN COMPUTER TECHNICAL DETAILS SO IT MAY BE A STRUGGLE I WON'T WANT TO TAKE ON!
AN EASY TO UNDERSTAND NEWCOMERS MANUAL WOULD BE NICE IF I KNEW WHERE I COLD GET ONE!!!
Your most serious problem seems to be your broken capslock key.
Nice to hear that SuSE is still hiring people to do innovative things.
Hopefully it will make KDE3 a little bit snappier ( there isnt a lot left
to improve judging from Win2ks speed )
You might also want to look at this page:
http://www.kosta.ch/~stefanh/speedup/
And I am looking for corrections or additions to these pages.
Bye,
Stefan
Anytime you want to translate that to English, you just go ahead! :)
Some of us like only know one language.
Pity that we know the wrong one this time, eh? ;)
http://babelfish.altavista.com/, crybaby.
Heh, next time you want a laugh, take a large, techincal article or paper, and run it through babelfish into another language, then back to your own.
better than
#ifdef DEBUG
cerr<<"I am still alive"<
seems I forgot the main point, of course I mean:
#define DEBUG 1
#define dcerr if (DEBUG==1) cerr<<"SomeClass::"
Alex
It doesn't require that much to discover that KDE works badly with 64MB of RAM ! Windows 2000 works better than Linux/KDE on a Pentium 200 Mhz with 64MB of RAM !
The main points that should be addressed (both for a whole Linux distro and KDE) are really memory usage and speed, unless people won't start using Linux/BSD instead of crappy Windows.