Simon - speech activated user interface for KDE

Every once in a while, the KDE community stumbles across a third party application that is well integrated into KDE, but has somehow managed to fly completely beneath the radar. One such application is called simon (small 's' intentional), a speech recognition program that integrates well with KDE and provides a means of interacting with KDE using voice recognition.

What follows below is an overview of simon and excepts from an interview that the dot conducted with Peter Grasch, lead coder.

What is simon?

Simon is a speech recognition subsystem that can be tied into X11 or Windows and, using a plugin-based architecture, can be used to control the user interface. The name simon is a derived from the children's game, 'Simon Says', but in this case, 'Simon Listens'. The application is intentionally lowercased while the financial support organization is called SIMON listens. It is also apparently a back acronym for Speech Interaction MONitor. Peter Grasch writes on the project's origin:

The school [I was attending in 2006] has a subject called "project development" where students group to develop smaller projects with external companies. During the search for a project we were approached by Franz Stieger, a teacher for special needs children who in his day-to-day work was confronted with children who suffer from spasticity. Because their limited motor control they are not able to write clearly with a pen nor are they comfortable writing texts on a keyboard. Franz wanted to know, if it would be possible to use speech recognition software to help them participate in the classroom. With this idea he came to us. Under my leadership, we (a team of four students) then researched available speech recognition software and quickly concluded that none of the commercial or free offerings on the market were
capable to adapt to the speech impairments of our test subjects. And because we had no idea what we were up against we decided - why not create our own solution?

By 2007, Grasch and his team had a working prototype. The first word that simon recognized was thunfisch (tunafish in German) which, when spoken, triggered a full screen image of a tuna. Within a short time, simon had two available commands: Executables and Places. Since then, more commands have been added which are perhaps best illustrated by a video of simon 0.2 (the most recent release) in action.

Caption: Simon 0.2 beta 3 controlling KDE.

At the moment, commands are implemented as configurable plugins. For example, while currently there is no plugin that issues dbus commands to running programs, once such a plugin is written, it would be possible to configure and train these new commands using the graphical interface.

Under the Hood

Simon is written in C++ using Qt and KDE for the user interface and thus fully integrates with KDE 4. Grasch explains their decision to integrate with KDE:

It just seemed a logical step. While simon was [originally] a Qt only application, we had a lot of supporting libraries that in the end were just cheap clones from similar KDE widgets/routines. With time it just got harder and harder to maintain all this extra code. Of course, the KDE implementations were also a lot better tested and were all in all just more mature and complete. What made the decision easy was the foreseeable maturing of KDE on Windows. Because Windows was a target platform we wanted to support it was of course important that such a huge dependency as KDE would work well on win32. Once that was ensured, the porting was a breeze.

Applications built on the KDE 4 platform, such as simon, will happily run under Gnome, XFCE, KDE 3 or any other X11 desktop environments when the appropriate libraries are installed.

I asked Grasch if there were any major advantages to the transition to KDE 4 as it benefits simon, and he replied: More features, less crashes. Or to quote a great tagline: "Code less, create more". [ed: this is part of the tagline for Nokia's Qt Development Framework]

Grasch also comments on the relative stability of running on Linux versus Windows: Simon on Windows is a tick more reliable because of the huge mess that is the sound stack on Linux and especially Ubuntu where their questionable pulseaudio setup sometimes breaks simon (portaudio, actually) in the most unbelievable ways.

To get around these problems, the team provides updated packages for Ubuntu that should allow simon to run reasonably well on that distro. Packages are also available for OpenSuse from the homepage, or you can compile it yourself using the usual KDE toolchain using instructions from the wiki.

Voice Recognition

Simon uses a number of technologies under the hood that allow voice recognition to happen. Grasch provides rough overview:

The whole recognition process is basically a statistic calculation. Sound input is recorded and tested against a speech model which consists of an acoustic model and a language model.

The language model contains information about the language to be used. This can of course be whatever you define; Standard English, French, Klingon, whatever. The language model most importantly defines what "words" there are and of which sounds do they consist. The sounds are represented with phonemes (which are to spoken speech what characters are for written speech). There are well defined phonetic alphabets like the IPA or the X-SAMPA but again you can of course define and use your own - it doesn't matter. The language model also defines what word combinations are valid ("grammar").

The acoustic model defines how those phonemes sound. This is done by feeding a lot of transcribed speech (speech samples where the system already knows what is being said) into a training algorithm. The output of this training procedure is a Hidden Markov model (HMM). So during recognition, the microphone records speech data which is then digitalized to an uncompressed wave form. This wave form is compared with the data stored in the acoustic model to find out what phonemes are most likely being said. These phonemes are then - with the help of the language model - transformed into words and "sentences".

In comparison to commercially speech recognition offerings simon does not ship with any predefined speech model. Instead, it makes it very easy for the end- user to create his own. A process that is normally extremely complicated (a very similar process to what simon does internally is outlined in the HTK book - the instructions span about 100 pages - and the HTK book is targeted towards linguistic professionals). This makes simon independent of any existing language or pronunciation and gives all the control to the end-user.

The process of creating of the speech model and the output models are state of the art - this has also been verified by the signal processing and speech communication laboratory of the technical university Graz. The acoustic model is trained with the HTK toolkit which has to be installed separately and Julius is used for the recognition (a modified version is shipped with simon).

Grasch continues to talk about dictation, and the problem that it presents to the way that simon has been designed: As detailed above, simon does purposefully not ship with a speech model, but our design decision to enable (require) every user to create his own model opens the door to users with speech impairments or languages that just don't have a large enough user base to be interesting to commercial alternatives. It also makes dictation unrealistic. To achieve a large and well trained enough speech model to enable dictation commercial offerings use thousands of transcribed speech of representative professional speakers to create a "standard" speech model which is later just slightly adapted to match the individual pronunciation of the end-user. As we don't ship with a default model it would mean that every user would have to invest a couple of thousand hours into training the model.

However, we are investigating methods to keep simon as flexible as it currently is but adding the feature to use base-models if needed. At the moment this is only in a planning stage and no code is written. We however, already are planning a few projects that would give us access to large amounts of transcribed speech. As soon as any form of adapting a speaker independent model is added to simon we will also try to make using VoxForge models as easy as possible. VoxForge is a project to build large speaker independent speech models.

Currently, the VoxForge project does not have large enough models to properly enable dictation, however this GPL-licensed project could benefit from simon driving users to help contribute. Grasch suggests using this release for simple control of public kiosks or home computers, and he even provides an example of one user who has adapted simon to do home automation. Whether it can be used to control a spaceship is yet to be seen.

Current Release and Future Plans

The current release, simon 0.2, was focused on stability. To that effect, this is a very successful release. The whole flexibility of the system is really the killer feature [of simon 0.2]. For testing purposes we once quickly created a speech model that would assign commands to: coughing, yawning, snoring, whistling, tongue flicking, etc. After five minutes training we were able to surf the web by coughing into the microphone!

At this point, the project has started to mature and is no longer purely an educational project. It currently operates with a team of three developers, Grasch, Franz Stieger and Matthias Stieger who all work nearly full-time on simon. Of course the long term goals are full dictation, but in the meantime, the team's goals for 0.3 include integration of KDE's 'Get Hot New Stuff' functionality, and improving the recognition process by taking into account the confidence scores of the recognizer. The ability to download new commands, training texts, and vocabularies with less hassle will contribute to a growing community around simon. So, if you want to control Amarok, you simply pick the Amarok control package from a list, read a few texts to train the acoustic model, and start controlling Amarok using voice commands.

Licensing Caveats

In my investigations into the GPL-licensed simon, I've discovered two licensing snags that will likely partially inhibit its uptake, at least for the near future. When I first started researching this software, I thought that it would be great if simon could be shipped with the KDE software main releases, however two of its dependencies have weird licenses and would prevent it from being shipped thus.

HTK, the toolkit responsible for the HMM evaluation is distributed under GPL-incompatible, restrictive license that prevents redistribution. In order to install simon, one must separately download HTK from their website which requires registration. The source is available, and they encourage you to modify and contribute to it, but it cannot be redistributed.

Additionally, Julius, used for the voice recognition has an attribution clause which causes problems with the GPL in a way that is reminiscent of the old-style BSD license (the one with the advertising clause). Any research conducted with simon would thereby require a reference to the Julius authors in the bibliography.

These problems will, for the immediate future, prevent many distributions from shipping simon. Perhaps we should read the text of the GPL to simon to see what it thinks. Grasch notes that there are relatively few installations of simon, so for those using it, these issues are not likely to be pressing.

Final Words

Simon provides a unique way of interacting with your computer using voice recognition (not dictation, yet) which integrates well with KDE. Installation requires manually obtaining and installing HTK separately from simon, due to a licensing conflict which will probably inhibit the uptake. However, if there are enough users, we as a community may be able to gently push Cambridge to dual license in a compatible way. In the meantime, for the adventurous at heart, or those with special computer interaction needs, simon exists to fill this niche. With the right plugins, one could potentially even order Tea, Earl Grey, hot from within KDE.

Comments

OK, so this is pretty damn impressive. Very well done on a speech recognition system that doesn't suffer the same problems as other, proprietary systems!

I'm quite impressed, not only by the achievements of the authors but also the degree of realism they're bringing to this difficult task.

Looks great. I don't personally know anyone who would need it but I can imagine it to be very useful to many people.

One thing that occurs to me. There (unless I missed it) does not seem to be a way to enter text other than predefined lines.

Perhaps an easy way to implement that would be a simple on screen keyboard, giving the user the ability to spell out text.

Further down the line perhaps even predictive text based on a dictionary similar to a mobile phone.

Easier to suggest ideas than to implement them but I imagine it to be quite helpful to those having trouble with motor control.

Keep up the great work!

by bedahr

Actually we are currently working on a virtual keyboard.

Watch the development blog (http://simon-listens.blogspot.com) for updates.

Greetings,
Peter (simon developer)

Great stuff! Even people without impairments could use this, I believe! I, for one, could surely use this!

A small note: make at least an English version (maybe even other language version) of the webpage. The more people can read about the stuff, the more it will buzz, and the more contributors you get. The program seems to work in other languages (at least in English), so there is no reason not to make a webpage that talks in other languages!

Hi!

Actually, the homepage has been (largly) translated to quite a few languages:
English, Italian, Turkish, French and Spanish.

Just select the language with the help of the flags in the upper right corner of the page.

Greetings,
Peter

It requires non-free software, so it may as well be proprietary.

Just like the whole of KDE and it's programs may as well be propriety on Windows, as that is non-free software you mean?
*sigh*

Yes, it causes problems and limitations, but it certainly does not take away all of the advantages of free software.

Free software running on windows is different and important because it is for those who do not yet have a commitment for freedom in their software and gives them a taste of it. It also makes it easier when they do decide to switch because they can use familiar software on KDE on GNU/Linux.

Non-free software as a dependency to free software doesn't make things better but worse because it strengthens the hold to proprietary software. It is important to those of us who wish to use 100% free software with distributions such as gNewSense, Trisquel or others on the now fast growing list of 100% free distributions here: http://www.gnu.org/distros/free-distros.html For me and many others to want to use this software, the non-free dependencies must be replaced.

Hi!

I'm Peter, the simon developer.

Free software as ideology is nice and all that but you also have to be practical. Don't get me wrong, I am a huge supporter of free software (I devote a _lot_ of time to a GPL project after all). But on the other hand I try to be practical.

The freedom that free software brings is important but what about the freedom to write? The freedom of access to information? The freedom of having a job and earning your own money that you can spend however you want? All those are freedoms that, in my opinion, are more important. All those are freedoms that many disabled in our target group don't have. In the long term simons goal is to provide freedom to those by working around their disabilities.

Today, the HTK is the only way to get there quickly.

Also, AFAIK, simon is the only project that tries to achieve that using GNU/Linux at the moment. Isn't that the exact same thing as KDE on windows? Draw people to GNU/Linux? Giving them the ability to say goodbye to Dragon Naturally Speaking on Microsoft Windows (don't get me wrong we are not there yet)?

Now let's look at the options to get to an all free speech recognition solution. Here are the pieces that would make up such a system:
* Speaker independant speech model
* Acoustic / language modeling toolkit
* Recognition engine
* User interface / integration in DE which brings all the above together

simon is this user interface. simon is developed under the GPL. Really. Nothing fishy about it. The model is generated by simon itself (thats enough for now) but for the future some kind of community driven model would probably work best. Something that e.g. simon could support through it's integrated trainings process.

The recognition engine is Julius licensed under a similar license to the original BSD license. Again, free software.

Then we have the modeling kit. simon uses the NON-FREE HTK for that. Only _one_ class in simon comes into contact with the HTK. The model compilation manager. This class: http://speech2text.svn.sourceforge.net/viewvc/speech2text/trunk/simonlib.... Those 1200 lines (including other, julius related stuff) are everything that links simon to the HTK. The class could very, very easily be replaced with one that uses something else.

No let's be realistic. In case simon gets used by the community, don't you think that it only heightens the chance of e.g. the open source library ghmm getting more attention? Don't you think Julius would get more developers?

Greetings,
Peter

Hi,

"For me and many others to want to use this software, the non-free dependencies must be replaced."

I don't have the feeling you want to use this software as you've not even thanked Peter for his efforts or commented _in a constructive way_ on the work.

I suggest you live by your principles and start writing/contributing to a free _full_ HTK replacement. Once that's done I'd be happy to hear your point.

Your argument is lacking a mature reasoning so much that right now you're just making yourself look like a 13 year old fan-boy.

Grow up.

Francois

Even though I don't need this program, nor know anyone else who does, I have a very strong urge to try it out.

Yes it looks appealing and I will try it too.

Raina
(spam has been removed)