Fer de Lance - Truly Intelligent Multimedia Browsing

Monday, 7 May 2001  |  Inorog

Wolfgang Müller is the maintainer of GIFT, GNU's very interesting Content Based Image Retrieval System (CBIRS). A CBIRS is a system to search for images based on their content. Wolfgang wrote to the dot recently inviting us to make public his latest efforts to integrate CBIR and related technologies in KDE. He has launched a new project, Fer de Lance, which is meant to properly integrate GIFT's technology in Free Software desktop environments and browsers. The goal is to have a completely open source infrastructure which can make open source desktops "content-aware". Carsten Pfeiffer found this new development very innovative, and started to write kmrml to provide KDE support for GIFT. Kmrml, which already has a preliminary Konqueror plugin as well as an IO slave, is available here (screenshot). Wolfgang and Carsten hope that you will find a minute to read the relevant information at the sites linked above, and then want to contribute to their projects' development.

Comments:

Re: Fer de lance - Truly Intelligent Multimedia Browsing - Jos van den Oever - 2001-05-07

The Dutch scientist Ben Schouten recently published on this subject: How does a computer recognize a cow? (http://www.cwi.nl/cwi/press-releases/2001/PhD-Schouten-0301.html). He wrote some software to it too. Don't know if it's GPL though.

Integrating retrieval into the desktop - Wolfgang Mueller - 2001-05-07

Hi, Thanks for the link. Work about content-based image retrieval has been around for quite a while. This field is very active, and I hope it continues to be. Fer-de-Lance is an effort to bring this work to the normal user, and to show that retrieval services like the ones the GIFT and similar systems can offer are useful for daily work. At the same time this can give an impulse for research on query engines that are *more* useful for the user. The task is huge, but there are many useful milestones, so things should be fun to do. Cheers, Wolfgang

Re: Fer de lance - Truly Intelligent Multimedia Browsing - Fabian Wolf - 2001-05-07

Can anyone translate that? I can't read dutch :-(

Re: Fer de lance - Truly Intelligent Multimedia Browsing - Wolfgang Mueller - 2001-05-07

If you are interested in the field as a whole, try looking for Content-Based Retrieval, CBR Content-Based Image Retrieval, CBIR Or look for systems: Viper (the query engine of the GIFT) Photobook (MIT, relatively and famous) QBIC (Query By Image Content, IBM) The pointer page of the group I am working in has much more links like that. (http://viper.unige.ch/other_systems/) Cheers, Wolfgang

Re: Fer de lance - Truly Intelligent Multimedia Browsing - Rob - 2001-05-07

Just a quick note: any search for CBR is likely to turn up pages for Case Based Reasoning, not the same thing!

Re: Fer de lance - Truly Intelligent Multimedia Browsing - alex - 2001-05-07

i only speak a tiny bit, but you can gather the main bits just from the english. the screenshot is not quite so enlightening as carsten's, but the description confirms that it is in the same vain of "question and answer" sessions to improve the computers ability to recognise. the main thing i got was in english though: use of the 'drost effect', in a fractal version of an image the image still takes the same representation even when the details become apparent. nice... Alex

Link to thesis - Martijn Anthonissen - 2001-05-07

Well, I do speak Dutch, but the link provided is a rather uninteresting press release about Ben's work. His PhD thesis is also available online though, and it is in English (don't let the first few Dutch pages fool you). You can download his thesis from ftp://ftp.cwi.nl/ Next, click /pub and then /bens. There are two PDF files. Somehow a direct link to the PDF files doesn't work. Enjoy, Martijn

Re: Link to thesis - David - 2001-05-07

Dank je wel

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Lenny - 2001-05-07

Damned, you guys are so bleeding edge. :) Cool stuff!

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Christian A Strømmen [Number1/NumeroUno] - 2001-05-07

This is simply cool :) Now, anybody care to convince google to use this ? ;)

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Wolfgang Mueller - 2001-05-07

Thanks for trying :-) Google etc., they are all aware of the field. They have all people working on it (I suppose), but there is the problem that they earn only a finite number of cents per click, and they cannot spend more floating point ops than they get paid. Furthermore, if you have really HUGE collections you run into problems that GIFT/Viper does not encounter, because it's dealing "just" with a couple (50) of thousands of images in this demo. Cheers, Wolfgang

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Solhell - 2001-05-08

Main scalability problems are: - Speed of the search with increasing number of images, and if you want to create an index instead of linear sarch, how to design the index. - Quality and relevance of the search results when the number gets huge. This is quite challenging for a general data. This is usually the most important point and probably requires advances in the low level image processing algorithms. This part is usually what CS people are ignoring and focusing on the first part of the problem.

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Wolfgang Mueller - 2001-05-08

Thanks for the clarification. I take it you're from the business :-) I rather think that the current focus on indexing and learning techniques is rather a backswing from the times where people were looking for the "holy grail" of image retrieval, the "perfect" distance measure. I guess we agree that we need both good learning from interaction, and good image modelling. What I would like to see happen, is that the Fer-de-Lance enabled desktop becomes a playground for cutting edge research, with immediate feedback from interested users. I also think that for the success of both the GIFT and of Fer de Lance, it is crucial to have an open framework into which everybody who's interested can integrate their plugin to add some functionality. This is research, it is moving fast, so we need to be open for everybody with ideas. The GIFT is a framework into which you can add your query engine, and the next step would be now to make it easier for outsiders to add their own feature set for their own file types (i.e. without adding a complete query engine). Wolfgang

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - gis - 2001-05-07

Related to this topic, there was a news-item on content-based audio-retrieval at http://www.heise.de/newsticker/data/thd-03.05.01-000/ just a few days ago (sorry it's in German). Currently, it only accepts MIDI files as input, but they say it will be able to query for whistled input in autumn. Very interesting.

Window border - KDE fan - 2001-05-07

BTW, where did that window border come from? Looks great!

Re: Window border - gis - 2001-05-07

As usual, the great artwork of qwertz :) It's designed as an IceWM decoration, which kwin can make use of, thanks to gallium. I'm not sure there's a download page for this deco yet, but you might ask qwertz <kraftw@gmx.de> for it.

Re: Window border - Matt - 2001-05-07

Wow, different docorations like that should really be a part of the main KDE distribution (that don't take a lot of space) to show how many different looks KDE can have.

Re: Window border - Christian A Strømmen [Number1/NumeroUno] - 2001-05-07

It does.. it's from the kde 2.2 cvs..

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - gis - 2001-05-07

BTW, for those who try out kmrml, you will see that it doesn't provide a history yet, i.e. you can't hit the "go back" button from one query and it will bring you back to the previous. This will be fixed in some of the next versions. You might as well send in a patch tho :) Oh, and if you the leave the mrml part (i.e. by hitting the back-button) and spuriously get a Konqueror crash, this might be due to some library unloading problem (static objects involved). Well, I didn't get any, but in case you did, it will get fixed :}

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Chris Young - 2001-05-07

WOW! This is really cool for KDE. What we need now is the ability of KDE application to use images from mrml:// -based sites. This way, I could use a KOffice app to search for an image to insert into a presentation or word processing document. That would be really cool. The only problem is that the File/Open dialog doesn't handle Non-Direcory-based URLs, so I guess this would take quite a bit of work. Definitely, this needs a configuration tool that allows you to keep up with servers/image directories that you find useful. This could really become the new way to get clipart for an application. Now we just some sort of simple interface that allows a KDE-based appliction to get the images. Anyway, this is way cool! Keep up the excellent work.

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - gis - 2001-05-08

Actually, the mrml-slave _could_ return some kind of directory listing, namely all the urls returned by the server. I'm not sure it would be really userfriendly tho. Starting a search from the filedialog? Hmm. Lukas Tinkl is/will be working on an image-gallery for KOffice and that's where kmrml could be integrated easily. You have a thumbnail view of your existing images and can ask for similar images -- simply drag the result into your KWord-document or presentation. A configuration dialog will definitely be added, soon. As stated in the README, kmrml is currently more proof-of-concept than a final solution. So contributions and also ideas how and where to integrate this are very welcome.

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Antialias - 2001-05-07

It is fast, it is beautiful. here is my contribution to screenshot collection.

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Kent Nguyen - 2001-05-07

Woah!! That's soo awesome. I never can believe KDE 2 can look that good as well as that functional. *drool*. --kent

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Jelmer Feenstra - 2001-05-07

Hey dude ! That looks nice :) When are you going to release those new icon sets ? They look really 'different'.

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - Chris Young - 2001-05-07

My contribution as well. Just noticed an extra option when right-clicking on a file. Check out the right-click menu item :)

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - not me - 2001-05-08

Eeeek! It's the result of a mad scientist's cross between Windows and MacOS! :-) Acually, that's pretty cool. Now, if only we could get menu options like this when dragging archives around...

Re: Fer de Lance - Truly Intelligent Multimedia Browsing - johnson - 2001-05-08

When will konqueror add links to mount points in its tree? So, for example, your tree in konqueror would look like: +Bookmarks +History +Home Directory +Network +Root +System Logs +Floppy Disk +CD-Rom +Windoze +BeOze

Re: Fer de Lance - Truly Intelligent Multimedia Br - jd - 2001-05-08

Probably not a good idea to let konqueror try to figure out mountpoints, it could get messy. A configurable tree, which would enable me to define my own quick-access links, would be cool though. I would add Printer(configuration and job list) and maybe Scanner to your suggestions (is there a scanner kpart yet?)

Re: Fer de Lance - Truly Intelligent Multimedia Br - Roberto Alsina - 2001-05-08

Well, figuring out mount points seems to be pretty easy. Just call mount, and see what it gives :-) That is *probably* not portable, but any platform could get it implemented in minutes.

Re: Fer de Lance - Truly Intelligent Multimedia Br - David Faure - 2001-05-09

kio/global.* has code to determine mount points in a portable way already (that is, it has ifdefs for the 4 different types of API !). All that's needed is someone to call that and create the desktop files - one question is when, so that it basically happens only once :). Well the version number maintained by konq_tree could be used for that. Any takers ? Should be quite easy to do. David, who gives priority to KWord for now :)

Re: Fer de Lance - Truly Intelligent Multimedia Br - Risto Treksler - 2001-05-11

It sure would be nice to have CD rom and floppy links in the side bar, by default. For the longest time I didn't even know that the SideBar was configurable. I hardly ever use the "Go" Menu item in Konqueror, but now that I found the "SideBar Configuration" option in there, I added links to some URLs and to CDROM devices via the right mouse button menu in the folder that comes up. I think it would be great if there was an alternative way to configure the SideBar For example, right click in the SideBar, Create New Link to Location CDROM Device ... Or, Drag a folder to the sidebar and choose something like "add URL" AFAIK, Choosing "Link" is not good enough to make it appear in the SideBar.

Re: Fer de Lance - Truly Intelligent Multimedia Br - Martin - 2001-05-11

It is a good idea ! JP

Re: Fer de Lance - Truly Intelligent Multimedia Br - Jason Katz-Brown - 2001-05-08

Drag a directory from konqy to the sidebar.... tada! Jason