DEC
13
2005

KDE Quality Assurance Meeting Report

On the weekend of December 10th & 11th, a small group of nine KDE contributors met in Hamburg to work on quality assurance checks for KDE's code base. This not only covered C++ sourcecode but also other aspects such as checking the state of the API documentation, looking for common errors in KDE's manuals and evaluating the usability of KDE applications. Read on for the full report.


Group Photo from the KDE QA Meeting: Simon, Ellen, Harri, Harald, Adriaan, Frerich, Reggie, Dirk and Ben

Hosted by the kind folks of froglogic, saturday started with people arriving at the froglogic office and saying hello to each other. As people arrived, it became clear that there was some interest in looking to what degree froglogic Squish would be useful for testing the user interface of KDE applications, so a small Squish demonstration took place. Also, the original project (the infamous English Breakfast Network) which started the whole idea of having a QA meeting wasn't known to some people so a laptop was attached to the projector and a short "Browsing around the EBN" session was held. It was a nice opportunity to show off how the results of (different) automatic checks could be presented in a reasonably uniform manner.

After having some food, Ellen Reitmayr of OpenUsability.org fame was kind enough to educate the rest of us about usability. The fact that a whole bunch of factors which influence the usability of the user interface can easily be checked automatically came as a relieving surprise (so the idea wasn't all that crazy after all!). During her explanations, it became apparent that even though many things are terribly difficult to check automatically it would be a big achievement if we could have nightly checks for the low hanging fruits. Getting them out of the way (and there are a lot of them) would give the usability people more time to focus on the things which actually require human intervention.

The rest of the afternoon was spent on letting ideas for checks bounce off each other, brainstorming about the feasibility of automatic checking for such things (just imagine people occasionally moaning something like "You can't possibly do this and that automatically!" or "Such a tool would give so many false hits, it'd clutter the output and make the valid reports about issues harder to spot!"). A little bit of hacking was done as well (talking about and looking into the sources of different C++ parsers) - in particular, KDevelop developer Harald Fernengel hacked up an QTestLib-based test based on Dirk's binary compatibility checker (unfortunately nobody seems to know about 'gmake bcheck') which checks whether a given interface composed of a number of C++ classes is binary compatible to another interface, which would be useful for KDE's libraries (so that your KDE applications don't break after upgrading KDE's libraries to a new minor release because of binary incompatibility). In the evening, people had food together and then visited Hamburg's central Christmas fair, having some of the infamous German Glühwein together. :-)

While Saturday was much more about talking and exchanging ideas, a lot more work got done on Sunday. Reginald hacked up a little Squish script to show off how Squish can be used to automatically check the menus of different KDE applications for some of the usability points Ellen mentioned the day before (the picture of Squish opening and closing all menus available in an application - and the fact that one script was used to test different applications - caused some "oh!" and "ah!" and nodding among the viewers). Adriaan (who runs the English Breakfast Network) was given a free Squish license so that you should see the results of various usability tests showing up on the web pages. Work was also done on porting the QTestLib-based tests in KDE's libraries to the new QTestLib as stored in the qt-copy module of KDE's repository. In general, Sunday's soundscape was much influenced by the sound of people typing on their laptop keyboards. The day finished with a dinner together, after which the first persons left to venture back to the mundane world.

A few noteworthy things we got out of this weekend:

  • Two days were not enough. KDE has no real centralised testing framework in place, so a lot of time was spent on discussing very basic problems.
  • Usability is a lot less diffuse a term than most people think. A lot of things which make up the usability of a program (as defined by the KDE HIG) can indeed be checked automatically. Having a Squish license for running Squish on the EBN so that we can start generating the reports is a very nice achievement.
  • There doesn't seem to be a useful and free static C++ analyser, which is a pity. Maybe using Roberto Raggi's C++ parser to do something like that would be worthwhile, but it is still a lot of work.
  • How to present whatever results we get out of any checks and integrating that information into the workflow of volunteers working on KDE is a tricky issue - having five-digit numbers of issues on the front page easily scares potentially helping hands away. This is indeed a social problem, and we could have spent the entire weekend on just evaluating different approaches at how to present rules appropriately.
  • Visiting the Reeperbahn in Hamburg on a Sunday night is quite boring.

Comments

There is a pretty vivacious discussion on Slashdot about KDE's features and user interface, titled Torvalds Says 'Use KDE':

http://linux.slashdot.org/article.pl?sid=05/12/13/1340215

Don't forget to participate in the Slashdot discussion ;-)


By Vlad C. at Tue, 2005/12/13 - 6:00am

"Don't forget to participate in the Slashdot discussion"
-----------------------

I do not think it is of any benefit to participate in the Slashdot discussion for the sake of participating. And really bad would it be to participate in order to pour oil into the flames.

If anything, KDE supporters should steer towards getting a sober discussion going about what the Linux desktop (KDE for us) is still missing, what hinders its more widescale adoption, what can be done to overcome the hurdles, etc.

We should also focus on what common ground we have with Gnome, and what common competition there is to take market share from. After all, more than 800 million people use MS operating systems. And 5.000++ million people do not yet use any computer at all!

Let Gnome be different where they want to be different -- we claim the same right for KDE. But let's also cooperate and share where we both benefit from it, and when it helps to win a bigger share of the 6.000 million user market for any Linux desktop in the near future.


By Kurt Pfeifle at Tue, 2005/12/13 - 6:00am

You really missed the smiley...


By Peppelorum at Tue, 2005/12/13 - 6:00am

Well, you can't keep up with the postings anyway (895 now)

And they took the dot down out of frustration.
I think there is something profoundly wrong with the mental status of many of these gnomedotters.


By reihal at Tue, 2005/12/13 - 6:00am

We've been under massive spam attacks since November 12, so I don't think there's a direct correlation. I had to take the dot down momentarily because the db grew too big and needed packing. flatforty is down for the moment so that Google does not get quick links to index any of this new spam.

We're deleting every single one of those spams so I don't know what good it does the spammer except for being a malicious attack on us. Trying some spam prevention methods but spammer is adapting.


By Navindra Umanee at Tue, 2005/12/13 - 6:00am

Just wanted to suggest those "type letters in the pic" post verfication used on many blogs, but the "choose the fitting button" you already put in place here is more confortable. Good idea. =)


By ac at Tue, 2005/12/13 - 6:00am

Hmm, so something happened in the beginning of november that pissed them off.
Maybe this article on Slashdot:
http://linux.slashdot.org/article.pl?sid=05/11/06/1413237
(Shuttleworth has KDE on his own desktop machine)


By reihal at Tue, 2005/12/13 - 6:00am

> .. is still missing, what hinders its more widescale adoption, what can be done to overcome the hurdles, etc.

Missing? It has most bits already, but what's truly missing is *the* enormous bug-hunt (last mile). And that probably can't be done without major investment as no-one likes to debug the hardest 1% of the bugs unless is paid well to. KDE is _way_ better than GNOME on this front (probably thanks to C++/Qt) but still _far_ from being perfect.


By Janne Karhunen at Tue, 2005/12/13 - 6:00am

First I note that it is good to see that KDE is starting to work on QA.

I agree, that what is now needed most is bug hunting. KDE is a mature product with most of the features that users need. So, it is now more important to hunt down all of the bugs that we can find that to add new features. That is, the feature which is now most important to many users is stability.


By James Richard Tyrer at Tue, 2005/12/13 - 6:00am

I strongly agree. However, all the bug hunting in the world won't help KDE 4.0 if they fix the bugs in KDE 3.5. What KDE needs even more than bug hunting is a large suite of automated tests. Not just usability tests and static code analysis. Those can do only so much. What is really needed is for the entire product to be automatically tested with unit testing and integration testing. Where KDE can't be unit tested or integration tested, it should be redesigned so it can be.

Obviously, this isn't the end-all of QA, and people may argue that it's too hard to do, but many other projects manage to do this. See GNU Classpath, and the Mono Project, for example. GCC does it, and so does Perl. The list goes on.

KDE also has the problem that it's harder to test graphical apps than frameworks and compilers. The way to do that is to separate the UI code from everything else and to make the UI code as small as possible. Until KDE starts doing this, it will not be stable and the bug counts will continue to grow.


By Anonymous at Wed, 2005/12/14 - 6:00am

> Obviously, this isn't the end-all of QA, and people may argue that it's too hard to do, but many other projects manage to do this. See GNU Classpath, and
the Mono Project, for example. GCC does it, and so does Perl. The list goes on.

The non-GUI code in kdelibs already has quite some unit tests and the test suite will be greatly expanded with KDE4.

> The way to do that is to separate the UI code from everything else and to make the UI code as small as possible. Until KDE starts doing this, it will not be stable and the bug counts will continue to grow.

Thanks for enlighting us with your great wisdom. I'm sure no KDE developer thought about this before.


By cl at Wed, 2005/12/14 - 6:00am

I'm sure they have thought of this. The KDE libs idea shows this. I also know that there are unit tests already, but, to my knowledge, there is no standardized way of testing the entire framework, there are no nightly runs of the unit tests, etc. If I am wrong, I'd love for someone to point me to a status page of some sort where it lists the regressions. The problem isn't that it isn't being done. The problem (as I see it) is that it's not being done enough, and this seems to be a move to help fix that.

My article wasn't meant to be enlightening, or to tell the KDE developers how to do their jobs (well, not jobs, but you get the point). I was simply stating my agreement with the parent poster and giving my reasons. Nothing I said should be news to anyone with a software engineering background.


By Anonymous at Thu, 2005/12/15 - 6:00am

> KDE also has the problem that it's harder to test graphical apps than
> frameworks and compilers.

Isn't that what GUI test programs are for?

> The way to do that is to separate the UI code from everything else and to
> make the UI code as small as possible.

I'm not sure that this is really practical. Well actually, it depends on the application. If I was making a program to solve differential equations, yes this would work since the GUI would be just an interface to a rather complex program. But with the desktop, I don't see this as really practical. In many cases, if it isn't already separated (i.e. click this widget and that function gets called), it would probably just add a lot of additional library calls with no great improvement in reliability. The only way I could see this as being applicable would be to separate out those functions into libraries (isn't some of that already being done?) so that they could be tested without using the GUI and I wonder if this would do much good. Would it be easier to use a GUI test program or to write a test probram to test the library?


By James Richard Tyrer at Thu, 2005/12/15 - 6:00am

Yeah, that's what GUI test programs are for. It's just that they generally aren't as good as unit tests, since you are really doing integration testing at that point.

As for the practicality of separating things, well, you're right: sometimes the complex logic is in the UI. There is still a lot that can be done, especially with MVC architecture and other OO techniques. However, when taken to the extreme, you get frameworks that are too bulky and overengineered to be useful, so obviously there are tradeoffs. Like I said, it's not a silver bullet. I was just making the point that this news is a step in the right direction.

Anyway, in general I find testing libs a lot easier than scripting GUIs, but YMMV.


By Anonymous at Thu, 2005/12/15 - 6:00am

Same thing, but open-source, actively maintained, and works on more toolkits. Seems like a no-brainer to me.

http://people.redhat.com/zcerza/dogtail/


By Dominique Quatravaux at Wed, 2005/12/14 - 6:00am

As far as I know Dogtail does not work with any Qt application since it uses the ATK interface to manipulate GUI objects and thus is tied to Gtk.

Also, I don't think that AT-SPI (which is what Dogtail uses to talk ATK) provides the same degree of flexibility as Squish.


By Frerich Raabe at Fri, 2005/12/16 - 6:00am

Another tool that comes to mind ist CXtest by CodeWeavers, a visual regression testing application that can run interaction scripts: http://www.cxtest.org/

Maybe there's some potential there.


By Eike Hein at Thu, 2005/12/15 - 6:00am

You might try looking at synopsis -- the author is very bad at publicizing his work, and parts are still "in-progress", but it really is a quite impressive generic introspection tool. (IIRC it can parse boost, and provide nice inspection APIs in python, at different levels of granularity...)


By Nathaniel Smith at Sat, 2005/12/24 - 6:00am