It’s been a while since I’ve posted about Fiber, so here’s a brief update.
I’ve read quite a bit about what people really want to see in a browser and I’ve repeatedly heard about the disappointment of my tab sidebar decision.
At first I dismissed it because I hadn’t really heard of it before, nonetheless know that it’s a beloved feature for many. I also wasn’t keen on the idea because all the tab form-factors were a horizontal strip and I wanted to (lazily) deal with one orientation.
The original plan was to allow an extension to handle the more crazy form-factors, but as I was blueprinting the APIs on paper I quickly found the tab-bar becoming a nightmarish monster which would have made custom tab extensions painful. Ultimately as a shortcut until a nice API can be made (and many more critical APIs can be rolled out) I’ll be adding sidebar tabs as a native feature. I may look at some sort of button form-factor as well, such as the ones commonly seen in mobile browsers.
(We announced flippin’ Plasma Mobile, no KDE developer doesn’t have phone on the brain!)
With this development though, I’m going to put tab extensions onto the backburner. I’m not completely satisfied with hardcoding something potentially invasive, but in these early stages battles much be picked, and there are other features the tab-bar can gain from that decision.Performance and Rendering Engines
Before I get into some performance stuff I have on tap, there’s an elephant I’d like to address.
I’ve heard a lot of complaints that WebEngine/Blink/Webkit are “bloated” and resource intensive, and that this selection would inevitably make Fiber a resource-hog browser “like Chrome”. I’ve also heard that Fedora isn’t shipping WebEngine, leaving users out in the cold.
On Fedora not shipping WebEngine… I can’t say much for that. Programs will start using it, and it’s going to be an increasing problem for them the longer they hold out. I won’t switch to Gecko, nor will I be providing some sort of “-engine” build switch for Fedora or other holdouts; it’s not a practical validation for the time or complexity required to support that, and it would certainly introduce more issues into the browser itself than it would solve for any issues Fedora finds in the engine.
On resource usage it needs to be pointed out that Webkit/Engine/Blink itself is not particularly heavy, and some browsers (like Epiphany) use it and got top marks in simple resource usage tests. Even if the results weren’t 100% accurate, the data was still compelling. Ultimately the rendering engine is only one factor of resource usage, and other factors will be at play.
That being said I’m not concerned about bean-counting megabytes, as I’m not out to make a lightweight browser. I’ll try to be efficient of course, but Fiber has some much steeper hills it must climb to maintain speed and resource usage for one reason; extensions. Extensions are inherently less efficient than native code, and many times when complaints Firefox or Chrome performance arise it can be traced back to misbehaving add-ons. With Chrome the browser separates extensions, plugins, and rendering engines into processes which is heavy – but stable. If Fiber did the same it would quickly become much heavier than Chrome as by design it will have multitudes more extensions.
Put simply if people start complaining when they have 3 or 4 inefficient extensions gumming up the works, imagine the moment of panic when you consider a browser that will ship with over a dozen default extensions for basic browsing!
Which brings me to the juicy performance stuff;Simple by default; Fiber ‘Performance Tuning’
You often see guides suggesting how to “tweak” Firefox or Chrome to be faster. Often they’ll recommend isolating bad extensions and altering the basic behaviours of how the browser operates. Often though the defaults have been carefully considered… It’s not like Firefox is shipping the “second best” configuration they can think of.
With that in mind computers are extremely diverse and for all intents and purposes no two computers are alike; OS, RAM, CPU, connection, physical location, and even drivers affect how well a computer can perform. For Fiber and it’s extension-happy orientation performance can quickly become a nightmarish prospect; how can we guarantee that a browser with dozens of potentially unoptimised ECMA-driven plugins will perform well? And, if the user must intervene and “tweak” the system, how can we make this process simple and efficient?
The solution being implemented is to boil tweaking down to simple but powerful sliders the user can adjust: “Extension Processes” and “Loading & Fetching”.
These sliders are the first two added for performance tuning, and more may be added if they make sense. These sliders aren’t completely voodoo though, and simply act as a front-end to the more fine-grain options hidden in the Configuration utility, and will adjust those values in a way that is safe and sane. “Simple by default, Powerful when needed”.Extension Processes
Though not anywhere near fleshed out, this had to be added early to ensure the concept would be propagated throughout the extension architecture. Extensions in Fiber will have a “separateProcess” variable in the manifest file which can hint how the browser should manipulate its process system. Depending on how the user has set the “Extension Processes” tuner, the two values together will affect how extensions will be processed in Fiber.
When set to “Fewer Processes” only extensions which specifically request process separation will be split from the main process. When set to “Most Processes” all extensions will be placed into separate processes regardless of setting. The middle-road option will favour process separation, but not force it.
The idea behind this is to let simple extensions like bookmarks managers be in the main browser, while more complex extensions like download utilities can safely ensure they won’t poison the main process.The only catch-22 may be processes which embed more dynamic widgets onto the UI; I’ll be trying to make stock widgets accommodating, but if embedding widgets onto the interface it may force an extension doing so onto the main thread. I don’t know fully yet whether this will be the case.Loading & Fetching
Loading and Fetching will affect how eager the browser is to load unopened tabs, and how willing it will be to forget inactive tabs. “Fewest Resources” will lazy-load everything, while being more eager to dump data. “Eager Loading” will actively load unopened tabs, and keep everything in memory.Work Status
I took a break to nail down some VDG work, so progress had briefly slowed during that time to pencil-and-paper work. This was prompted by me merging some Fiber-related icons into Breeze, and then going on a binge where I frothed at the mouth and made 47 application icons. DON’T JUDGE ME. Now that I’m just waiting to make some adjustments to some straggling icons, Fiber development is back on full-tilt – at least the full-tilt that a full-time employee can give in his off-hours.
Much like Firefox, Fiber uses Sqlite for profile storage. Earlier today marked a milestone where Fiber is now creating empty profiles with config tables, loading them with data, and providing nice clean functions to reliably read that data. Profiles themselves have solidified quite a bit, and now the relationship between all the moving parts is much more clear.
Work is now going into making profiles able to validate and repair themselves, adding “isValid” checks before attempts are made to use a profile, and getting proper but meaningful alerts in for serious issues.
The profile manager has shaped up nicely, and expanded a little bit beyond my original scope. There’s still a ways to go before the profile manager is 100% complete, but even to a relatively green Qt developer the quality of the toolkit has made things easy.
The Fiber icon has also been tweaked, and now has multilple resolutions which are pixel-perfect.
The beginning of the day was reading some social media in the morning with breakfast catching up with the times. While going though my Google+ feed I saw a post that I seen before about the a bug with a krunner plugin. The plugin in question was this which Riddell, Dan and I debugged to find some more info about the bug such as that is effects Kubuntu, Arch and openSUSE so it is upstream related. Riddell provided some info on the bug page to maybe help resolve it later. All and all I learned some debugging stuff, compiling, grabbing source and some very tiny C++.VDG
Then I was off to VDG again to work more on the High Contrast Color scheme for Plamsa 5 with Andrew Lake. The VDG started doing some concepts and design for a new secure login for KDE aka SDDMLUNCH
For lunch we had the cafe in the University that we used yesterday again. For a starter I had Spanish Potato and Tuna Salad with a plate of Grilled Chicken and french fries. Ovidiu really really disliked his choice of squid.AppStream/Muon and KDE Neon
After lunch Matthias Klumpp talked about his work with Debian/Fedora for AppSream with metadata to be used in Muon Discover as well as some redesigns for it to work better and look better for everyone. Up next was the long awaited KDE Neon where people from Kubuntu, Red Hat, and othersAfter Party
Thanks to the GPUL we had an amazing party with actually good music, free food and beer/wine. There was dancing, swapping name badges and having fun in general.
This is a picture of us walking to the mall for the party:
For a short while, the Plasma Mobile forums were hosted outside of the official KDE Forums. In our quest to put everything under KDE governance, we have now moved the Plasma Mobile forums under KDE’s forums as well. Enjoy the new Plasma Mobile forums.
As a few users had already registered on the “old” forums, this means a smallish interruption as the threads could not be quickly moved to the new forums. We’re sorry for that inconvenience and would like to ask everyone to move to the new forums.
Thanks for your patience and sorry again for the hassle involved with that.
HJens Lunch Time
Jens describes Skittles and Doritos
Sebas shows off Plasma Mobile phone with a look that suggests he wants world domination by next year
The opening ceremony to remember absent friendsby
I've just pushed a patch to KWallet5 allowing you to have your wallet unlocked automagically during login. This patch was originally done by Alex Fiestas for KWallet4, so all credits and free beers go to him; I've merely just forward-ported it.
You'll also require kde:kwallet-pam repo and pass "-DKWALLET5=1" to cmake. This will generate pam_kwallet5.so which then can be coinstalled with the same module for KWallet4 (plus it also enables some ifdef'd code inside the module). If you're still using some KDE4/Qt4 software which is using KWallet4, you will require both modules present.
How to set up kwallet-pam can be found over at Luca's blog (though he said he'll update it for KWallet5 later, so you may want to wait a bit :).
When approaching this issue I had been thinking about the issue for a while. I had mainly 2 problems: I was rather frustrated with previous Linux-based systems so far and the one I liked didn’t really scale for us. One thing was clear: We had to stand on the shoulder of giants.
- The first one to think about was the N9 SDK (and by extension N900). It used to have a scratchbox-based system that emulated the one on the phone. It was useful for testing the applications locally on the device (although I actually never used that), I think that this one had both cross-compilation toolchain and base system, as well as the host and it used QEmu if a host executable was run. It felt weird because it shoved you in a weird system and you had to pull your code in weird ways to fetch it. Otherwise it worked great. Afterwards, madde came. It’s what I was looking for really and played quite well with cmake, actually when I published the steps on how to develop for the N9 it’s what I used back then, but it probably was too late for it to become a thing I guess. Also many people weren’t too fond of it, as I learned after some time.
- The Android NDK is the other I took into account. I don’t think it would be fair to compare anything we could do with the actual Android SDK, so I’ll limit myself to this one. This one ships the complete cross-compilation toolchain and runs native (similar to the BBX SDK, IIRC).
For the N9 approach, I would have had to concentrate on figuring out technologies that were long dead (and probably should remain). For the Android approach I found 2 big problems: We have to actually work on generating the actual binaries (which means, from any platform to a specific ubuntu-vivid-arm target) plus all the dependencies. This meant, creating a new distro, and we already had debian/ubuntu for that, let’s use debootstrap! Oh, wait…Old school
For a start, I took what sebas already started, in fact. Using debootstrap to create a chroot jail that could cross-compile the projects into our platform. This started to prove feasible really soon, as 2 or 3 days after working on it (and after fixing some issues that kept arising, mostly on KF5 itself and packaging), I already started to output binaries that could be deployed on the device.
- My IDE is outside of the jail, so there isn’t much we can do to integrate at all (we can’t access the build directory, and most of the data in it isn’t too meaningful for the outside anyway). A solution would be to ship and run the IDE from within the SDK though.
- You need to be root. Not only to generate the system, but also to run it.
An idea I wanted to approach was docker. Everyone on the web world is shit-crazy about it and it’s deeply based on traditional Linux, so there’s quite in common already. Doing something similar to the deboostrap there was a piece of cake, and I managed to re-use the set up code I already had for the previous version.
- Also everything is inaccessible from the IDE
- I’m less aware of the limitations I’ll find.
Still, this second approach feels lighter and it’s quite fun to investigate something new like that.Tooling
It seems that the jailed systems are the way to go, at least for now, so the tools I’ve created so far assume that they live in the jail as well. So, what do we have?
- createpkg: Makes it possible to create a deb package that can be sent to the device for testing. It’s much simpler than the simplest deb package ever, but it works. Much better than sending executables over, much better than learning how to package for Debian.
- deploypkg: sends the deb file, installs it and starts the application.
- click-get: Downloads a click file from the Ubuntu Store, it’s harder than you’d think.
- kdev-debdeploy: Does it all, inside your good ol’ IDE.
- Workflow: Figure out a way to deploy KDevelop within either jail, but then we’ll also be able to use kde-debdeploy, that does mostly the same as these tools, integrated on the IDE.
- Emulation: Testing: QEmu + docker anyone?
With loads of help from people on #kde-devel, we finally managed to complete the KDE Network Filesharing port to KF5. Wasn’t easy, given that this was my first time porting frameworks, but it was real fun. Apart from apol’s blogpost shared in my last post, here’s another post that was immensely helpful to me while porting: Porting a KControl Module to KF5.
kde-dev-scripts has a kf5 bit wherein there are perl scripts that would help in porting fast (did most of my work actually). When everything worked successfully, there was a linking error which wasn’t getting fixed even after hours of googling. Turned out to be a moc header include issue, as pointed out by bshah on #kde-devel and fixing that, everything was a success.Up next
I now need to replace kpackagekit that Samba currently uses with packagekit-qt, and this involves plugging in the code I wrote earlier into it. This would probably complete the Samba bit of my work.
This is turning out to be a great learning experience and KDE Code no longer seems alien to me. With almost a month to go, I am positive that I would be able to get good results.
The second day at Akademy started off with 10 or so hours of sleep!, which was much needed for basic functions (really happy I don’t have to drive here). The hotel (Rialta) had great breakfast with coffee, OJ, bread with meat and cheeses, yogurt, cereal all the basics that makeup a great day!To the talks!
First talk to start was with Lydia Pintscher with “Evolving KDE” which see went over what is planned in the next stage of Plasma 5 and what she wants to be planned as President of the e.V.
Next up was Dan Leinir Turthra Jensen talked about the work going on with Plasma Mobile from porting to devices, running Android applications on it and more.Kubuntu Team
Harold and Rohan gave a talk about their CI(Continuous Interrogation) work.Lunch and Group photo
Then we had the awesome and big group photo with everyone currently at Akademy both the people who are going to it as well as the volunteers who help make the whole thing run. Shortly after the group photo was lunch which again was provided by the Schools cafe and paid for by Blue Systems, which was great with pasta, yogurt, fruits, and pudding!
Right after Lunch everyone was right back to hacking some Open Source goodness!
After the Lunch was over Ovidiu-Florin and I did a interview with Matthias Klumpp for the Kubuntu Podcast which has a new episode coming out next month on August 5 with hopes of including this and future interviews from Akademy and if not the next episode.
As you are probably aware by now we announced the Plasma Phone project during Akademy this weekend. In this blog post I want to discuss the role of KWin in Plasma Phone.
Plasma Phone uses Wayland as the windowing system with KWin being the Wayland compositor. This is our first product which uses Wayland by default and also the first product which uses KWin as the Wayland compositor. The phone project pushed the Wayland efforts in Plasma a lot and is the only reason why we are able to make Wayland a technological preview with the upcoming Plasma 5.4 release.
The phone project gave KWin/Wayland into the hands of many developers who started to actively use it and to develop with it. This obviously helped to find lots of small and larger issues which then could be fixed. It triggered a stabilization process which reached a stage that I can use a Plasma Wayland session on my notebook with enough confidence that I won’t lose data due to crashes. So thanks to the whole team for pushing the system to the limits.
An area which saw lots of work thanks to the Phone is the interaction between Plasma as the desktop shell and KWin as the Wayland compositor. With Wayland we keep the architecture of having a dedicated shell process which is not inside the compositor. This architecture has served us well in the past and we don’t see a reason why we should change this. It means that KWin can serve as a compositor for other desktop projects, but it also means that Plasma can be used with other compositors. Now unlike X11, Wayland’s protocols are secure by default. This means that Plasma as the desktop shell does not know anything about windows from other processes. To solve this problem we designed a special window management protocol which exports the required information. The protocols are still under development, but we hope that they can be also useful for other projects with a similar architecture (LXQt being an obvious candidate). Of course such protocols should only be used by the “blessed” desktop shell process – this is something we still need to implement and one of the reasons why at the moment Plasma/Wayland is only a technological preview.
Window management is not the only area where the shell process needs to be “blessed”. Also for announcing information about its own windows, we need some more information. We need to know whether a window is a “panel” or the “desktop” view. So on the phone the shell background is just a desktop window, the panel on the bottom is just a normal dock. This allows us to share the general layering code with the X11 implementation of KWin. Even more having the panels marked as panels allows us to properly define the maximized area. And the windows on the phone open maximized by using the “Maximizing” placement strategy in KWin.
Working on the phone project also had some surprises. For example the virtual keyboard just didn’t want to show. It turned out that the reason for this was that Qt(Wayland) only requests the virtual keyboard if it has keyboard focus. Keyboard focus inside KWin is bound to the “active” window. So we had to implement activating Wayland clients. But the initial implementation didn’t really solve it, we still had areas where the keyboard didn’t come up. E.g. on the desktop in the KRunner search field we couldn’t get it to show. The reason was related to the described problem: we did not yet support activating Wayland clients when clicking on them. This got solved by implementing mouse action (and touch action) support inside KWin. So thanks to this change (done for the phone) we can properly switch between Wayland windows on the desktop with a mouse driven setup.
Another nice touch is that KWin requires a running Xwayland server. This gives us support for all “legacy” X11 applications such as Qt 4 or GTK2 on the phone. Just we hit a problem with them: they did not react on touch events. The reason for this is quite simple: touch support is not yet implemented in Xwayland. As a solution we implemented simulating pointer events in case a connected Wayland window does not bind the touch interface. Thus all Wayland and X11 applications can now be used with a touch screen – be it on the phone or on a notebook with a touch screen.
So far I have only spoken about progress made in KWin which is relevant for the desktop. So what about the phone specific adjustments? Well there are hardly any. In the core of KWin there is no phone specific code. The only phone specific code is the hwcomposer backend. This is a plugin used for creating an OpenGL context using libhybris and for reading input events (unfortunately libinput cannot read events on a libhybris enabled system). According to cloc this plugin is just about 800 lines of code and about 200 are just very straight forward mapping of Android key codes to “normal” Linux key codes. For comparison: KWin + kwayland/server currently have about 120,000 lines of code. And even this hwcomposer backend is not phone specific. It could also be used to run a normal KWin/Plasma session on any libhybris enabled device. There is one important difference of the plugin to the rest of KWin which is worth to mention: it is GPLv3+ licensed, while everything else is GPLv2+. The reason for this change is the fact that libhybris is Apache 2 licensed and this license requires a change to GPLv3.
KDE has been trying for years to get Plasma working on different form factors with mixed success, so when I first started on this I was pretty intimidated. But we looked around for how to build this and it turns out there is software for it just lying around on the internet ready to be put together. Incredible.
It got very stressful when we couldn’t get anything showing on the screen for a few weeks but the incredible Martin G got Wayland working with it in KWin, so now KDE has not just the first open mobile project but also one of the first systems running with Wayland.
And with Shashlik in the pipeline we are due to be able to run Android applications on it too giving us one of the largest application ecosystems out there.
The question is will there be traction from the community? You can join us in the normal Plasma ways, #plasma on Freenode and plasma-devel mailing list and #kubuntu-devel to chat about making images for other devices. I’m very excited to see what will happen in the next year.by
Akademy, the yearly KDE conference is alive and kicking. During the last days we were discussing again about potential KDE licensing issues (for instance code that is licensed under GPLv2, but not GPLv3). That’s why KDE is maintaining a relicense checker script, that every KDE contributor should enter herself/himself. I’ve blogged about it already in January 2015, but it cannot hurt to repeat it from time to time: From the KDE relicensing page:
A couple of KDE dependent projects or even libraries have moved or are going to move to GPLv3.
Unfortunately, GPL v3 is incompatible with GPL v2. This means that it is not possible to create a project linking GPL v2 and v3 code together. There is no problem for projects which are licensed GPLv2+ (version 2 or above).
A few parts of KDE are currently licensed as GPLv2 only. So far we have no reason to believe that this was something other than an oversight. However, we still need to validate with the individual copyright holders that a relicense to GPLv2+ or GPLv2+v3 is okay with them.
Therefore, in an effort we’re trying to identify the contributors that have contributed under the terms of GPLv2 and where the “+” part was not explicitly mentioned. If we know that all contributors agreed to a relicense, we can go ahead and flip the license of the individual source file.
Short story: If you have not done so yet, please add yourself to the kde relicensecheck.pl file!
If you cannot add yourself right now, please consider sending a mail to firstname.lastname@example.org publically stating that you’re ok with relicensing all your contributions hosted on the KDE services under either GPLv23, LGPLv23, GPLv2+, LGPLv2+. Additionally, you can give the membership of the KDE e.V. the credentials to relicense your contributions in future (this makes a lot of sense if you cannot contribute to KDE anymore and are no longer around). Many of the KDE contributors did this already. Thanks!
PS: The official way to give KDE the rights to change licenses would be by signing the KDE e.V.’s FLA. Please note, that the KDE e.V. will not abuse this by immediately changing licenses you chose. Instead, this mostly about emergency cases (think of someone who suddenly passes away). But the script mentioned above is already a good start, since we can use it to quickly scan our code and identify problematic parts.
A common problem for many applications contained in the KDE Applications releases are non-incremented version numbers. Often the maintainer forgets to update the version number of the application, like I did it for Kate since the first KF5 based release.
This means: On Bugzilla, I get bugreports that always tell me “Kate 5.0.0″, not very helpful.
KDE Frameworks solves this by automatic setting of the current framework release version in all framework CMakeLists.txt.
For KDE Applications, we have now optional the same concept. For details: visit https://community.kde.org/Applications/Versioning
In short: If you are to lazy to update your version number yourself or are just fine with using the same version number als the KDE Applications releases, you can add the following three lines to your toplevel CMakeLists.txt:
# KDE Application Version, managed by release script
set (KDE_APPLICATIONS_VERSION_MAJOR "15")
set (KDE_APPLICATIONS_VERSION_MINOR "04")
set (KDE_APPLICATIONS_VERSION_MICRO "0")
These variables will then be patched by the release scripts to the “right” version for the current release series.
You can than either just use the MICRO version to suffix your own applications version to differentiate the bugfix releases or like Kate construct your complete version number from the three vars and be done ;=)
I hope this helps to have more consistent and more important meaningful version numbers once again in the applications we bundle. Thanks to all people that made this “magic” happen ;=)
Now that cauldron is open, the kde team has updated KF5 to 5.12.0, Plasma to 5.3.2 and Kde Applications to 15.04.3.
we have added minimal kde-workspace and kdebase4-runtime that can now be installed with plasma.
This migration is a good way to remove old and unmaintained apps.
If you want to help don’t be shy
SystemSettings has no icons if .kde is missing.
No customisation for now.
The yearly KDE conference Akademy is currently being held with lots of interesting talks and workshops. One big thing that was announced yesterday is Plasma Mobile, a free mobile platform.
The presentation of Plasma Mobile was quite impressive: A video of using Plasma Mobile on a Nexus 5 showed a nice visual design, smooth transitions and quite some usable functionality already. This impression was confirmed later when I was playing around with Plasma Mobile on the Nexus 5 myself.
So good job, plasma team!
Plasma Mobile already raised quite some interest in lots of news sites, with lots of user comments (mostly positive), see for instance:
- Slashdot: KDE Community Announces Fully Open Source Plasma Mobile
- OSNews: Plasma Phone OS, a KDE project for mobile
- Hacker News: Plasma Mobile
- Reddit: Plasma Mobile on Nexus 5 (Kwin/Wayland in action)
- Golem: KDE Plasma 5 läuft auf Smartphones (german)
- Heise: Plasma Mobile bringt KDE 5 auf Smartphones (german)
What’s important to note is that the project is still in a very early stage of development, and its target is to be really usable around mid of 2016. As such, there are most certainly stability issues and lots of features missing.
But this also opens opportunities: If you are into mobile platforms, this is the right time to get in contact and contribute! The plasma developers are really nice people and welcome every single contribution, be it in terms of brainstorming ideas (e.g. graphical mockups), code, or organizing developer events. So get in touch now through the Plasma Mobile forums and through the Plasma Contributor Mailing List! Just say hi and ask how to get involved
Before I start this blog I would like to thank the wonderful Ubuntu community for sponsoring my trip to the wonder hub of KDE development at Akademy!
My trip to Akademy 2015 in La Coruna Spain started at 4:45 pm on Friday in Miami with a flight to Lisbon. I was serve decent dinner and later breakfast. I did not get much sleep on the first part of the trip, but the second one from Lisbon to La Coruna I got about 1 hour or additional sleep with me finally arriving at 11:30 or so AM local time. I also saw the entertainment system reboot and show me that it was running Linux! I finally had the awesome experience of meeting some of the people I have been working with for over 2+ years over IRC, Hangouts and phone calls. Today was filled many great talks from our own Riddell and Clark on the new Plasma phone and continued work on the CI end of Kubuntu respectably.On the first day we also had the announcement of Plasma Mobile being worked on by Blue Systems and the larger KDE community as well. I’ll have some more pictures of that in there own blog post and album on imgur later on. Blue Systems has been kind enough to sponsor lunch for this weekend and next weekend. So here I type this blog post with under 2 hours of sleep for 36+ hours of uptime lol.
A while ago, I published a post about ReComment, a speech-based recommender system. Well, today I want to talk about SpeechRec: ReComment’s bigger, better, faster, stronger brother.
With ReComment we showed, that a speech-based interface can enable users to specify their preferences quicker and more accurately. Specifically, ReComment exploited information from attributes such as “a little” or “a lot” which naturally occur in spoken input, to refine its user model.
With SpeechRec, I wanted to build on the idea, that spoken language carries more semantic information than traditional user interfaces typically allow to express, by integrating paralingual features in the recommendation strategy, to give proportionally more weight to requirements that are said in a more forceful manner. For example, this allows to assign more weight to the constraint “Cheaper!, than to the statement “Cheaper..?”. In other words: SpeechRec doesn’t just listen to what you are saying, and how you are phrasing it, but also how you are pronouncing it.
Moreover, I wanted to up the ante when it came to task complexity. The ReComment prototype recommended compact digital cameras, which turned out to be a problem domain where a user’s requirements are fairly predictable, and finding a fitting product is arguably easy for the majority of users. To provoke conflicting requirements, and therefore better highlight the strengths and weaknesses of the recommender system under test, SpeechRec was evaluated with the domain of laptops. And let me tell you: recommending laptops to what were primarily students at a technical university is quite a challenge :).
Another problem we observed in the evaluation of ReComment, was that some people did not seem to “trust” that the system would understand complex input, and instead defaulted to very simple, command-like sentences, which, in turn, carried less information about the user’s true, hidden preferences. SpeechRec was therefore engineered to provoke natural user interaction by engaging user’s in a human-like, mixed initiative, spoken sales dialog with an animated avatar.Implementation
The developed prototype was written in C++, using Qt4. The speech recognition system was realized with the open-source speech recognition solution Simon, using a custom, domain-specific speech model that was especially adapted to the pervasive Styrian dialect. Simon was modified to integrate OpenEAR, which was used to evaluate a statement’s “arousal” value, to realize the paralingual weighting discussed above (this modification can be found in Simon’s “emotion” branch).
A comprehensive product database of more than 600 notebook was collected from various sources. Each product was annotated with information on 40 attributes, ranging from simple ones like price to the used display panel technology. As early pilots showed that users, facing a “natural” sales dialog, did not hesitate to also talk about subjective attributes (e.g., “I want a device that looks good”), SpeechRec’s database additionally included sentiment towards 41 distinct aspects, sourced through automatic sentiment analysis from thousands of customer reviews. MongoDB was used as a DBMS. Optimality criteria further allowed SpeechRec to further decode statements such as “Show me one with a better graphics card” or “I want one with good battery life”.
SpeechRec’s recommendation strategy was based on the incremental critiquing approach as described by Reilly et al, with a fuzzy satisfaction function.Demonstration
Check out the video demonstration of a typical interaction session below.
Note, how SpeechRec takes initiative in the beginning, asking for the user’s primary use case. It does this, because it determined that it did not yet have sufficient information to make a sensible recommendation (mixed-initiative dialog strategy).
Also note, how the system corrects it’s mistake of favoring to satisfy the “price” attribute over the “screen size” attribute, when the user complains about it (towards the end). SpeechRec instead comes up with a different, more favorable compromise, which even includes slightly bending the user’s request for something that costs at most 1200 euros (“It’s just a little more, and it’s worth it for you.”).
The conducted empirical study showed, that the nuanced user input extracted from the natural language processing and paralingual analysis enabled SpeechRec to find better fitting products significantly quicker than when a traditional, knowledge-based recommender system. Further comparison showed that the full version of SpeechRec described above also substantially outperformed a restricted version of itself, which was configured to not act on the extracted lexical and paralingual nuances, confirming that the richer user model facilitated by spoken natural language interaction is a major contribution toward the increase in recommendation performance of SpeechRec.
More information about SpeechRec and related research an be found in the journal paper P. Grasch and A. Felfernig. On the Importance of Subtext in Recommender Systems, icom, 14(1):41-52, 2015, and in my Master’s thesis.
SpeechRec was published under the terms of the GPLv2, and is available on Github. All dependencies and external components are free software and available at their respective websites.
Earlier today I gave a talk at Akademy 2015 about WikiFM. Videos of the talk should shortly become available. Based on the feedback that I have received during and after the talk, I have written a short resume of the points which raised more interest. They are aimed at the general KDE developer community, who doesn’t seem completely aware of the project and its scope.
You can find my slides here (without some SVG images).What is WikiFM?
WikiFM is a KDE project which aims to bring free and open knowledge to the world, in the form of textbooks and course notes. It aims to train students, researchers and continuous learner, with manuals and content ranging from basic calculus to “Designing with QML”. We want to revolutionize the way higher education is created and delivered.What does it offer more than $randomproject?
The union of these three key elements: students, collaboration in the open and technology. This has proven to be invaluable to create massive and high quality content.
All other projects usually feature just two of these elements, or concentrate on other material (e.g. video).
Additionally, we have virtual machines instantiatable on the fly on which you can start to develop immediately: check out http://en.wikifm.org/Special:DockerAccess (for logged-in users). By opening that link we istantiate a machine in the blink of an eye, with all the software you need already pre-installed. We support compositing and OpenGL. Your home directory is persistent among reboots and you always get a friendly hacking environment. Try it out yourself to properly appreciate it.Is it already used somewhere? Do you have some success stories?
The project started in Italy for personal usage. In spite of this, in just a few months we got national visibility and thousands of extemely high quality pages written. Students from other universities started to use the website and in spite of a content not planned for dissemination we get around 200 unique users/day.
In addition to this, the High Energy Physics Software Foundation (a scientifical group created among key people in istitutions such as CERN, Fermilab, Princeton University, Stanford Linear Accelerator, …) has decided to use WikiFM for their official training.
Moreover, we have been invited at CERN, Fermilab, and in the universities of Santiago (Chile) for delivering seminars about the project.How can this help my KDE $existing_application if I am not a student?
This fits in the idea of the Evolving KDE project that started in this year’s Akademy.
Hosting developer documentation, together with a pre-built developer environment which can let library users and students test your techology and step up in the hacking within a few seconds is an invaluable feature. It is possible to demonstrate features or provide complicated tutorial showcases while giving the option of trying out the code immediately, without having to perform complicated procedures or waiting for big downloads to finish.
For existing developers it also provides a clean development environment, where testing of new application can happen without hassles.Want to know more?
This is meant to be a brief list just to give you a taste of what we are doing.
I am at Akademy 2015 until the 29th. A strong encouragement: please come and speak to me in person!
I will be happy to answer any questions and eventually re-show you a shortened version of my talk.
Or, if you prefer, we are having a BoF on Monday at 11:30, in room 2.0a.