JUL
4
2006

KDE Edition of GUI Testing Tool Squish

froglogic GmbH today announced the availability of Squish/KDE. Squish/KDE is a free of charge edition of the Qt GUI testing tool Squish to create and run tests on applications developed for the popular K Desktop Environment. Squish offers a versatile testing framework with a choice of popular
test scripting languages (Python, JavaScript and Tcl) extended by
test-specific functions, open interfaces, add-ons, integrations into
test management systems, a powerful IDE aiding the creation and
debugging of tests and a set of command line tools facilitating fully
automated test runs.

With the release of Squish/KDE, which is based on Squish/Qt 3.0,
froglogic has moved to support the quality assurance efforts of the Open Source project KDE. This release is a result of feedback from the KDE Quality Assurance Meeting hosted by froglogic last year.

Squish/KDE is a free of charge (closed source) version of Squish which can be used to create and run tests for Open Source KDE applications (KDE 3.x and 4.x). Furthermore, froglogic will also fund a few part-time testers to create tests for the core applications of KDE to get the GUI
testing efforts going. To quickly understand Squish see this Flash video demonstration.

"We use KDE internally at froglogic as one of our main
development and office platforms
", said Frerich Raabe, software
engineer at froglogic. "Many of froglogic's engineers are part of
the KDE project themselves. Therefore we decided to support KDE's QA
efforts to give something back to the KDE community.
"

Adriaan de Groot, leading member of the KDE Quality Team, says:
"KDE believes in a strong and consistent user interface; our
cooperation with OpenUsability.org underscores this. Our HCI
guidelines set out what we believe to be a good user interface. We
have seen demonstrations of the power of Squish in testing the
quantitive aspects of the HCI guidelines and are pleased to be able to
deploy it within our quality assurance framework.
"

You can have a look at first efforts to automatically test
the usability of KDE applications using Squish at http://www.englishbreakfastnetwork.org/usability/.

froglogic GmbH is a software company based in Hamburg, Germany whose flagship product is Squish.
They also offer Qt consultancy services and other Qt-based
development tools such as the Tcl/Tk/Qt migration framework Tq.

Comments

This is very worrying. Surely KDE shouldn't be building dependencies on non-free software into KDE, even if it's just for developers? Didn't we learn anything from the whole BitKeeper thing?

Maybe I'm misinterpreting the KDE Manifesto:

<<<
What is KDE
===========
KDE is an Open Source desktop environment built from components that meet the Open Source guidelines in full.

KDE is OpenSource (TM)

No Compromises
==============
KDE is open and free in the full sense of the word.
>>>


By Luke Plant at Tue, 2006/07/04 - 5:00am

KDE in no way is 'depending' on this software. This is pretty much the same as the source code analysis tool used to scan KDE's code base (which helped in fixing a massive amount of bugs). If 6 months from now Froglogic doesn't want KDE to use their product to scan KDE's code anymore then KDE won't be in a messy situation or anything, though if it helps to fix 1 issue KDE will be ahead of where it would of been without it.

Also using this program on the code base doesn't make KDE closed source, or change even a single license, so I don't see how this is a violation of the manifesto (the manifesto says nothing about boycotting all non-oss software or the like)


By Corbin at Tue, 2006/07/04 - 5:00am

Another point is that these guys are really dedicated to KDE. Like with Trolltech's toolkit, I don't think we should be afraid of using their tools...


By superstoned at Tue, 2006/07/04 - 5:00am

What dependency? If this software becomes unavailable, in the worst case we will end up having no automated GUI testing, which is exactly the case right now.

This is similar to the coverity scanning, except that developers get to run the software themselves. All I see are upsides - froglogic get testing and visibility for their tool, KDE gets a GUI testing solution that is not available at the moment.


By taj at Tue, 2006/07/04 - 5:00am

There is a dependency. It is nowhere near as strong as a library dependency, but it is still there. If you assume that you are going to have this tool, you will build infrastructure around it. If it suddenly gets taken away (like BitKeeper did) then you are suddenly *worse off* than before, because otherwise you would have invested your efforts in creating free tools and test suites that didn't have that dependency. Now you have to rework your test suite.

It could also easily become a barrier to entry -- you will soon only be able to work effectively on certain parts of KDE (i.e. in the quality team) if you have a license for a certain piece of software. This is different from the coverity scanning, which imposed no such constraint on developers wanting to help with KDE. Put it this way -- if this precedent were followed in every area of KDE, it would quickly become impossible to be a casual KDE developer.

The analogies with QT that other people made aren't helpful, because if Qt were not free today and KDE wasn't using already, I'm pretty sure (and would strongly hope) that KDE wouldn't adopt it. All the movement has been in the opposite direction i.e. away from non-free, non Open Source software.

I understand completely the need for pragmatism in the real world, and it's because of the practical problems this creates that I'm against it (though I would also like to see free software being promoted more).


By Luke Plant at Wed, 2006/07/05 - 5:00am

What you say remind me strongly of what was said when KDE started with Qt 1.4, which was not GPL compatible. Still KDE has survived with this.

The people at froglogic are long time KDE contributors. If I am not mistaken, Reggie started KOffice and Harri Porten has contributed to many significant parts of KDE. The idea that they would try to harm KDE or try to control it or disturb it is simply ridiculous.


By Philippe Fremy at Wed, 2006/07/05 - 5:00am

Sure it survived, but you would think people would have learned from that episode. The nice thing about open source is that it comes with a license that you can depend on so that you don't need to depend on the good-hearted nature of people. Compare it with lending money to family members, everyone has the best intentions yet it causes a great number of family feuds when financial situations change.

Making the right decision now is a lot easier than fixing problems down the road when people have invested a lot of time and effort in test cases.


By Waldo Bastian at Sat, 2006/07/08 - 5:00am

Well, I think that the most logical thing to do is simply that
froglogic and the KDE e.V team sign a long term agreement stating
that Squish/KDE will remain free to use "forever" to KDE developpers
in general.

A bit like the agreement that exists between KDE e.V and Trolltech,
this way we will not end up in a mess like BitKeeper or similar.

I think that in the next few months after concluding efforts,
this kind of agreement should be made.

It might be premature to create one right now, but before KDE4 is released,
such an agreement should be signed, if the tool is useful and used significantly.


By fp26 at Thu, 2006/07/06 - 5:00am

First it does not build any dependencies, so that part is pure nonsens.

It's just a tool to make the finished product(KDE) better, if you have problems with using non-free software you don't have to use it. You can still use and develop KDE without touching it.

There is no way KDE can become less Open Source by using it, so there are no compromize on that part. Besides this is compareable to the way KDE has benefited in the past, by input from non-free code analyzing tools. The only difference this time is that the tool is free for every kde developer to use, if they want.


By Morty at Tue, 2006/07/04 - 5:00am

I agree that it doesn't make KDE itself any less Open Source, and maybe my quotation from the manifesto wasn't relevant.

But if these tests become part of official KDE test suites (as implied by the quotation from Adriaan), then to be an *effective* developer, or even bug reporter in some cases, you would indeed be forced use this software. Once a system like this is in place, it becomes far easier to get a patch accepted if you have a test to back it up, and many projects depend heavily on a test suite -- under the XP methodology, which many projects are tending towards, your test suite often becomes the primary method of checking that the software works.

I am an active member and (I am assured) valued contributor to the Open Source project 'Django' (and have been involved in others in the past). If the core developers decided to include some tests in the test suite that required non-free software to work, this would be a serious turn-off for me, and I imagine it would be pretty scandalous for lots of others -- I strongly doubt that the authors would consider such a move, and they are a long way from being FSF zealots. And the tests are not just an optional help -- there was some large refactoring work I was involved in recently that depended massively on the test suite and have been just about impossible without it.

So, my point is -- if you want to be a serious KDE developer, and these test suites become part of KDE, you (virtually) force developers to use non-free software, and thus alienate the Free Software community, which is a massive and very important part of the Open Source community.


By Luke Plant at Wed, 2006/07/05 - 5:00am

Last paragraph should read:

-- if you want to be a serious KDE developer, and these test suites become part of KDE, you are (virtually) forced to use non-free software, and thus the Free Software community, which is a massive and very important part of the Open Source community, is alienated.


By Luke Plant at Wed, 2006/07/05 - 5:00am

Well that's difference between Open Source philosophy and Free Software philosophy. Open Source isn't about being puritanical shunners.

Regardless this testing suite isn't something like BitKeeper where not only was it very important infrastructure (can't get more important then the version control system in an open source project), but it was being hosted by someone else. And required agreeing to not make competiting projects (which is where the problem was). I'm guessing Squish isn't too worried about someone using Squish to aide them in creating a test suite. ;)

Automated QA is a great idea, would be a nifty thing for Amarok.


By Ian Monroe at Tue, 2006/07/04 - 5:00am

> Automated QA is a great idea, would be a nifty thing for Amarok.

Wonder if it will detect inconsistent icon sets?

Sorry, couldn't resist. It's been on of those days...


By mart at Tue, 2006/07/04 - 5:00am

Sometimes ago I noticed that those who
complain about companies offering free binary only
package of their product for opensource project
or any other reason, are also those who downloaded windows
applications out of warez sites ...


By Mathieu Chouinard at Wed, 2006/07/05 - 5:00am

That was slightly ad hominem chewy :). Although the original poster missed the point by a few miles. Many people here pointed out that Squish is in no way a dependency. And i would like to point out that Qt was not Free right from the start either -- using the BitKeeper history and ignoring Qt one is sort of unfair... So chances are, when froglogic gets to the point that open-sourcing their product is a viable option for their business, they will. And KDE right now can probably help them getting to that point -- so why avoid it, it is a win-win situation.


By mornfall at Wed, 2006/07/05 - 5:00am

Interesting. Is it possible to record screen interaction like in the flash video under linux?

I observed one old bug with the KDE filedialogue, when I have large name files and scroll down with the keyboard often the cursor jumps. This is probably related to numbers or umlauts. The sorting algorithm of the files displayed seems to be different from the sorting list of the scroll routine.

How would you write a test for that?

How do you enter random content and look how it behaves.

I mean for a PDF viewer it might be possibly the best to use the Google api, fetch a random pdf file from the net and then try to open it until the routine find a file which breaks the program.


By Muk at Wed, 2006/07/05 - 5:00am

Hi!

Yes, you can record interactions with any KDE application and Squish will generate a script for you which you then can replay, edit, etc.

For testing the scenario you mention, you would just emulate in the script a key event (e.g. cusror-down) and then check that the item which should be selected is selected.

For verifications you can access Qt API from the test scripts.

Quick example:

def main():
startApplication("kwrite")
activateItem(":kwrite-mainwindow#1.KMenuBar1", "File")
activateItem(":kwrite-mainwindow#1.file", "Open...")

view = findObject(":kwrite-mainwindow#1.filedialog.KFileDialog::mainWidget.KFileDialog::ops.simple view")
clickItem(":kwrite-mainwindow#1.filedialog.KFileDialog::mainWidget.KFileDialog::ops.simple view.qt_viewport", "adddebug", 53, 9, 1, Qt.LeftButton)
test.compare(view.currentItem().text(), "adddebug")
test.verify(view.currentItem().isSelected())

type(":kwrite-mainwindow#1.filedialog.KFileDialog::mainWidget.KFileDialog::ops.simple view", "")
test.compare(view.currentItem().text(), "akregator")
test.verify(view.currentItem().isSelected())

type(":kwrite-mainwindow#1.filedialog.KFileDialog::mainWidget.KFileDialog::ops.simple view", "")
test.compare(view.currentItem().text(), "amarok")
test.verify(view.currentItem().isSelected())

sendEvent("QCloseEvent", ":kwrite-mainwindow#1.filedialog")
activateItem(":kwrite-mainwindow#1.KMenuBar1", "File")
activateItem(":kwrite-mainwindow#1.file", "Quit")

Now separate the data into a CSV file, put everything into a loop using the data-driven API Squish offers and you have a generic test case which tests what you want.


By Reginald Stadlbauer at Wed, 2006/07/05 - 5:00am

What are the differences (pros and cons) between Squish and KDexecutor ?

(http://www.klaralvdalens-datakonsult.se/?page=products&sub=kdexecutor)


By Charles de Miramon at Wed, 2006/07/05 - 5:00am

Obviously I have a biased view so I won't post a list of differences here. Generally KD Executor seems to be more of a record & replay tool for events while Squish is aimed at professional GUI software testers with features which go far beyond simple record & replay.

I'd recommend to just try it out yourself to get a picture of the differences.


By Reginald Stadlbauer at Wed, 2006/07/05 - 5:00am

Although it was published half a year ago, this article from Qt Quarterly 16 is hopefully still informative:

http://doc.trolltech.com/qq/qq16-testing.html

Feel free to post corrections.


By David Boddie at Mon, 2006/07/10 - 5:00am

A well-known problem is that fields are longer sized than expected e.g. because of translations and we are unable to read the messages displayed or the size of the window gets to long. How can qsuid help here as these problems usually occur at runtime.


By jum at Wed, 2006/07/05 - 5:00am

Unlike with a plain event recorder objects are references by their name, type, index etc. rather than their pixel position. Geometry changes do not cause any harm.

Regarding the translations itself: while the text of e.g. a button does normally serve as a good identifier it can be excluded from the identification process. Instead the QObject name or another unique property can be chosen to find this object.


By Harri Porten at Wed, 2006/07/05 - 5:00am

On a second read of your question I realized that I probably got it the wrong way around: verifying that widgets have indeed changed their geometry at runtime to accomodate changed content is not much of a problem, either: fetch the QFontMetrics of the widget, use its length() or boundingRect() to calculate the space required by the text and compare this size with the actual QWidget::size() or similar. As KDE apps normally use dynamic layouting I've not often seen cut-off text to be a problem, though.


By Harri Porten at Wed, 2006/07/05 - 5:00am

Wow, I was looking for something like that for a long time. In the Zope 3 project we are using unit and functional tests, much like those of Squish extensively. Of course we are doing HTML-browser testing (not only using Selenium), but it is pretty much the same.

And since the Squish people use Python I can see that they are using the unittest framework. Having said that, I wish there would be a way to use doctests. Programmed tests like those are only of limited use, because they do not document the actions well. Here is an example of a SchoolTool (Zope 3 based project) functional doctest:

http://source.schooltool.org/viewcvs/trunk/schooltool/src/schooltool/gra...

Note: The zope.testbrowser module is not at all Zope-specific and can be used to test *any* Web site using Python unit or doctests.


By Stephan Richter at Fri, 2006/07/07 - 5:00am