Fer de Lance - Truly Intelligent Multimedia Browsing

Wolfgang Müller is the maintainer of GIFT, GNU's very interesting Content Based Image Retrieval System (CBIRS). A CBIRS is a system to search for images based on their content. Wolfgang wrote to the dot recently inviting us to make public his latest efforts to integrate CBIR and related technologies in KDE. He has launched a new project, Fer de Lance, which is meant to properly integrate GIFT's technology in Free Software desktop environments and browsers. The goal is to have a completely open source infrastructure which can make open source desktops "content-aware". Carsten Pfeiffer found this new development very innovative, and started to write kmrml to provide KDE support for GIFT. Kmrml, which already has a preliminary Konqueror plugin as well as an IO slave, is available here (screenshot). Wolfgang and Carsten hope that you will find a minute to read the relevant information at the sites linked above, and then want to contribute to their projects' development.

Dot Categories: 

Comments

by Jos van den Oever (not verified)

The Dutch scientist Ben Schouten recently published on this subject: How does a computer recognize a cow?
(http://www.cwi.nl/cwi/press-releases/2001/PhD-Schouten-0301.html). He wrote some software to it too. Don't know if it's GPL though.

by Wolfgang Mueller (not verified)

Hi,

Thanks for the link.

Work about content-based image retrieval has been around for quite a while. This field is very active, and I hope it continues to be.

Fer-de-Lance is an effort to bring this work to the normal user, and to show that retrieval services like the ones the GIFT and similar systems can offer are useful for daily work.

At the same time this can give an impulse for research on query engines that are *more* useful for the user.

The task is huge, but there are many useful milestones, so things should be fun to do.
Cheers,
Wolfgang

Can anyone translate that? I can't read dutch :-(

If you are interested in the field as a whole, try looking for

Content-Based Retrieval, CBR
Content-Based Image Retrieval, CBIR

Or look for systems:
Viper (the query engine of the GIFT)
Photobook (MIT, relatively and famous)
QBIC (Query By Image Content, IBM)

The pointer page of the group I am working in has much more links like that. (http://viper.unige.ch/other_systems/)

Cheers,
Wolfgang

Just a quick note: any search for CBR is likely to turn up pages for Case Based Reasoning, not the same thing!

i only speak a tiny bit, but
you can gather the main bits just from
the english. the screenshot is not quite
so enlightening as carsten's, but the
description confirms that it is in the
same vain of "question and answer" sessions
to improve the computers ability to recognise.

the main thing i got was in english though:
use of the 'drost effect', in a fractal version of an image the image still takes the same representation even when the details become apparent. nice...

Alex

by Martijn Anthonissen (not verified)

Well, I do speak Dutch, but the link provided is a rather uninteresting press release about Ben's work. His PhD thesis is also available online though, and it is in English (don't let the first few Dutch pages fool you). You can download his thesis from

ftp://ftp.cwi.nl/

Next, click /pub and then /bens. There are two PDF files. Somehow a direct link to the PDF files doesn't work.

Enjoy,

Martijn

by David (not verified)

Dank je wel

Damned, you guys are so bleeding edge. :) Cool stuff!

by Christian A Str... (not verified)

This is simply cool :)

Now, anybody care to convince google to use this ? ;)

Thanks for trying :-)

Google etc., they are all aware of the field. They have all people working on it (I suppose), but there is the problem that they earn only a finite number of cents per click, and they cannot spend more floating point ops than they get paid.

Furthermore, if you have really HUGE collections you run into problems that GIFT/Viper does not encounter, because it's dealing "just" with a couple (50) of thousands of images in this demo.

Cheers,
Wolfgang

Main scalability problems are:

- Speed of the search with increasing number of images, and if you want to create an index instead of linear sarch, how to design the index.

- Quality and relevance of the search results when the number gets huge. This is quite challenging for a general data. This is usually the most important point and probably requires advances in the low level image processing algorithms. This part is usually what CS people are ignoring and focusing on the first part of the problem.

Thanks for the clarification.

I take it you're from the business :-)

I rather think that the current focus on indexing and learning techniques is rather a backswing from the times where people were looking for the "holy grail" of image retrieval, the "perfect" distance measure.

I guess we agree that we need both good learning from interaction, and good image modelling. What I would like to see happen, is that the Fer-de-Lance enabled desktop becomes a playground for cutting edge research, with immediate feedback from interested users.

I also think that for the success of both the GIFT and of Fer de Lance, it is crucial to have an open framework into which everybody who's interested can integrate their plugin to add some functionality. This is research, it is moving fast, so we need to be open for everybody with ideas.

The GIFT is a framework into which you can add your query engine, and the next step would be now to make it easier for outsiders to add their own feature set for their own file types (i.e. without adding a complete query engine).

Wolfgang

Related to this topic, there was a news-item on content-based audio-retrieval at http://www.heise.de/newsticker/data/thd-03.05.01-000/
just a few days ago (sorry it's in German).

Currently, it only accepts MIDI files as input, but they say it will be able to query for whistled input in autumn. Very interesting.

by KDE fan (not verified)

BTW, where did that window border come from? Looks great!

by gis (not verified)

As usual, the great artwork of qwertz :) It's designed as an IceWM decoration, which kwin can make use of, thanks to gallium.

I'm not sure there's a download page for this deco yet, but you might ask qwertz for it.

by Matt (not verified)

Wow, different docorations like that should really be a part of the main KDE distribution (that don't take a lot of space) to show how many different looks KDE can have.

by Christian A Str... (not verified)

It does.. it's from the kde 2.2 cvs..

BTW, for those who try out kmrml, you will see that it doesn't provide a history yet, i.e. you can't hit the "go back" button from one query and it will bring you back to the previous. This will be fixed in some of the next versions. You might as well send in a patch tho :)

Oh, and if you the leave the mrml part (i.e. by hitting the back-button) and spuriously get a Konqueror crash, this might be due to some library unloading problem (static objects involved). Well, I didn't get any, but in case you did, it will get fixed :}

WOW! This is really cool for KDE. What we need now is the ability of KDE application to use images from mrml:// -based sites.

This way, I could use a KOffice app to search for an image to insert into a presentation or word processing document. That would be really cool. The only problem is that the File/Open dialog doesn't handle Non-Direcory-based URLs, so I guess this would take quite a bit of work.

Definitely, this needs a configuration tool that allows you to keep up with servers/image directories that you find useful.

This could really become the new way to get clipart for an application. Now we just some sort of simple interface that allows a KDE-based appliction to get the images.

Anyway, this is way cool! Keep up the excellent work.

Actually, the mrml-slave _could_ return some kind of directory listing, namely all the urls returned by the server. I'm not sure it would be really userfriendly tho. Starting a search from the filedialog? Hmm.

Lukas Tinkl is/will be working on an image-gallery for KOffice and that's where kmrml could be integrated easily. You have a thumbnail view of your existing images and can ask for similar images -- simply drag the result into your KWord-document or presentation.

A configuration dialog will definitely be added, soon. As stated in the README, kmrml is currently more proof-of-concept than a final solution. So contributions and also ideas how and where to integrate this are very welcome.

It is fast, it is beautiful.
here is my contribution to screenshot collection.

Woah!! That's soo awesome.

I never can believe KDE 2 can look that good as well as that functional. *drool*.

--kent

Hey dude ! That looks nice :)

When are you going to release those new icon sets ? They look really 'different'.

My contribution as well. Just noticed an extra option when right-clicking on a file. Check out the right-click menu item :)

Eeeek! It's the result of a mad scientist's cross between Windows and MacOS!
:-)

Acually, that's pretty cool. Now, if only we could get menu options like this when dragging archives around...

When will konqueror add links to mount points in its tree? So, for example, your tree in konqueror would look like:
+Bookmarks
+History
+Home Directory
+Network
+Root
+System Logs
+Floppy Disk
+CD-Rom
+Windoze
+BeOze

Probably not a good idea to let konqueror try to figure out mountpoints, it could get messy.

A configurable tree, which would enable me to define my own quick-access links, would be cool though.

I would add Printer(configuration and job list) and maybe Scanner to your suggestions (is there a scanner kpart yet?)

by Roberto Alsina (not verified)

Well, figuring out mount points seems to be pretty easy.

Just call mount, and see what it gives :-)

That is *probably* not portable, but any platform could get it implemented in minutes.

kio/global.* has code to determine mount points in a portable way already (that is, it has ifdefs for the 4 different types of API !).

All that's needed is someone to call that and create the desktop files - one question is when, so that it basically happens only once :). Well the version number maintained by konq_tree could be used for that.

Any takers ? Should be quite easy to do.

David, who gives priority to KWord for now :)

by Risto Treksler (not verified)

It sure would be nice to have CD rom and floppy links in the side bar, by default.

For the longest time I didn't even know that the SideBar was configurable.

I hardly ever use the "Go" Menu item in Konqueror, but now that I found the "SideBar Configuration" option in there, I added links to some URLs and to CDROM devices via the right mouse button menu in the folder that comes up.

I think it would be great if there was an alternative way to configure the SideBar

For example, right click in the SideBar,
Create New
Link to Location
CDROM Device
...

Or, Drag a folder to the sidebar and choose something like "add URL"
AFAIK, Choosing "Link" is not good enough to make it appear in the SideBar.

It is a good idea !
JP

by Jason Katz-Brown (not verified)

Drag a directory from konqy to the sidebar.... tada!

Jason