A New Document Management System

Wednesday, 8 January 2003 | Asomers

newdocms is a proposal for a new and radically different way of managing your documents in KDE. It is a move away from the now over 30-year-old hierarchical file system towards a meta-data-based document retrieval system. A 0.1 preview has now been released along with a description and screenshots (typical, newdocms, save, open, results -- the GUI isn't all that pretty at this point). Although not yet ready for production use, newdocms has the potential to really change the way users interact with their computer. Herein lies a challenge to the Open Source community in general, and KDE in particular, to take a step ahead of the competition in a truly innovative way. Should newdocms be a standard part of a future KDE release?

Comments:

hmm - anon - 2003-01-08

Nice, very nice. However, everybody knows that the Single Directory System is the wave of the future. :-) PS Don't forget to vote <a href="http://www.linuxformat.co.uk/modules.php?op=modload&name=Awards&file=index">here</a> [linuxformat.co.uk] too! Everybody at GNOMEDESKTOP already has.

Re: hmm - David Johnson - 2003-01-08

However, everybody knows that the Single Directory System is the wave of the future. All too true, unfortunately. I cringe every time I walk by a cubicle and see every document the user ever worked on sitting on their desktop. They don't even bother with wallpaper because you can't see the desktop through the icons.

Re: hmm - dapawn - 2003-01-09

Interesting, but I don't think I'm quite ready to buy into a Single Directory system quite yet. :-) I do think the ability to store meta-data for files is an excellent idea. I would love to be able to browse the filesystem and have a document's meta-data pop-up in a tool-tip, or be displayed as a column in konqueror's detailed view. I also love the idea of the special 'Search Folders' (persistent queries in BeOS, VFolders in Evolution?), coming in kmail (KDE 3.2), where the 'Search Folders' are dynamically updated (based on your original search criteria) each time new messages arrive or are deleted. It would be awesome if this could be extended to the filesystem using files' meta-data! Example: In my digital photo collection, if I store meta-data about who is pictured in each photo, then I could create a 'Search Folder' of pictures of my Aunt who lives across the country. And every time I added another picture of my Aunt, it would magically appear in her 'Search Folder'. I could have a 'Search Folder' for pictures featuring myself too, and a picture of my Aunt and I together would show up in both 'Search Folders'. It wouldn't matter if photos were .jpg or .png or whatever. When she came to visit, I would show her her Search Folder (probably not the best example, but it was the first to come to mind :-). I could continue to organize my photos into regular folders like "Texas_Vacation_2002" or "Work/Joes_Farewell_Lunch". You could this with music files; basing the 'Search Folder' on style, hard, soft, artist, etc., or with documents, perhaps you have a 'Search Folder' for KDE, it could contain graphics files, presentations, spreadsheets, plain text files, HTML, etc. all about KDE. I don't know how feasible such a system would be, but I would love to see it! Anyway, good luck on the project! Its nice to see new ideas.

Complicated - Scott MacDonald - 2003-01-08

Very interesting, it is indeed a fresh and intuitive take on things are organized. However, I can see serious problems with users who are used to a simple preview of the document, organized by directory. While entering a concept name and shortcut keyword are nice for searching later, many people will find this unnecessarily complicated. It is unlikely that most people would take full advantage of such a system and would end up preferring how things currently are.

Re: Complicated - Ian Monroe - 2003-01-08

I agree. The users I see occasionally will use directories to organize their work, but usually don't bother. And once their (and this goes for myself as well) finished with a document, they usually have no need to open it up again, but keep it in the off-chance they need it. So in other words the "Recent Documents" feature found in most products is pratically good enough. But I know my grandfather could use something like this. He has saved about more then 10 years worth of documents. If he had been organizing them in this way the whole time, those documents would probably be more useful.

Re: Complicated - Robert - 2003-01-08

If you read his description, you would have noticed that users can simply save their docs without attributes, yielding a perfect "Recent Documents" structure, i.e. the docs are organized "save time descending".

Do we want this? YES!!! - Peter - 2003-01-08

Of course we want this, it is overdue. Managing the bookmarks in a HFS is a pain - I use grok for this, which is only a workaround. But a question: Does the newdocms have any impact on the shell? Is there a commandline interface to retrieve and store the files? I see it uses sqlite which seems to be a database - does that mean I can do `select * from theTable' and see all files along with the attributes? Anyways, my vote: YES!

Re: Do we want this? YES!!! - Shulai - 2003-01-09

I guess a thing like vi `newdocscm About:whatever` then showing posible results should work.

Usefull - Giovanni Di Guardo - 2003-01-08

I find it usefull especially in the area of education. It is a sort of LCMS (Learning Content Management System) simplified but more practical because catalogation (tagging) happens a the beginning of document creation and not in a second step.

What the desktop is missing - Thomas Olsen - 2003-01-08

I think it is a very interesting project you have done. You are absolutely right that HFS is a dead end if you want to organize data. Having tried to work at a 200+ people company using one fileserver for storing all data about customers and projects it has really been proven to me. But simply keeping track of the files on your home computer is also a major job. Some small points about your implementation: - I couldn't find it in your writing but do you follow any kind of standard when cataloging metadata? Most of the attributes you mention fits perfectly into e.g. The Dublin Core Initiative ( http://www.dublincore.org/ ). DC has the most common fields like Title, Description, CreationDate, ModificationDate etc. but also fields more suitable for doing 'intuitive searches' like Subject, a multivalue field which corresponds fine with your 'About' attribute, and Relation which as the name says defines relations between other documents. I think using DC as the basic metadata would standardize it a great deal and help people from getting confused about other peoples selfinvented metadata. Naturally it should be possible to extend the metadata set using other DTDs or schemas e.g. for multimedia, sales, communications etc. - Personally I would miss not being able to catalog my files both by metadata and by filesystem hierarchy. maybe I have missed something in your description, but I couldn't see if it was possible. Great job! Looking forward to seeing it grow - this could be a really great addition to the the desktop. As you mention this concept is mostly used in major CMS systems but try to have a look at Zope/CMF (http://cmf.zope.org) which implements some of the concepts and is both Open Source and free as in free beer :-)

This is the dawning of the age of Aquarius ;-) - Hafer - 2003-01-08

*Yes*, I'd like it for KDE as an option for sure!! Hell, hierarchical databases are so very dead for a park full of reasons. Time to send them HFS for companion. However, there's not only KDE and Gnome. What about gv, emacs, acroread, and the infinite bunch of shell commands that take files as argument?

Re: This is the dawning of the age of Aquarius ;-) - Robert - 2003-01-08

Hm.. I'm prgramming in Java (and on Windowze...) so i won't be able to deliver this myself, but, hey, it should be fairly easy to program a small console application that interfaces with the document management system and delivers e.g. a list of native filenames when supplied with metadata attribute values. Then the user would put this small tool in shell backticks and off you go like this: for i in `newdmssh attrib1=value1 attrib2=value2` do; [do what you want to the native files] done Wouldn't that be cool? Another approach would be to program a fully functional bash or sh extension that has some native commands and/or globbing tricks to allow interfacing with the doc. mngmt. system. (Of course all the tools like tar, gz, cat, etc. etc. can not be expected to support this in the near future...) Robert

Re: This is the dawning of the age of Aquarius ;-) - Roland Schäfer - 2003-01-09

Maybe some kind of virtual filesystem: -- cd /docs cd attrib1=foo cd attrib2=bar dir ... List of all files, which match the query attrib1=foo, attrib2=bar -- Every (virtual) subdirectory is a query on the results from the parent directory. Comments?

Re: This is the dawning of the age of Aquarius ;-) - ibotty - 2003-01-09

that's it. a virtual filesystem is great. (and to be done in not that much time). but id prefer no special characters in the interface ( e.g: attrib=foo ) consider acroread (or any other commercial software made without a good toolkit) . i won't count on the ability to quote these characters. and using > /me/.attr/foo/.comment/bar/.type/pdf to find pdf files is great! and all those unix tools, acroread, realplayer, opera... all software on your system will be able to use it, if the users knows how to cheat. (gworkspace, konqueror, nautilus, etc will provide a better gui, but you got the idea) ~ibotty

Re: This is the dawning of the age of Aquarius ;-) - ibotty - 2003-01-09

towards other metadata filesytems. having ONE fs with ONE user interface (e.g.: /me/.attr/foo/.type/pdf) and ONE api is better than having reiserfs4, xfs, <you name it>. each having their own api, their own way of getting attributes, their own api. having one api to bind them all in the lands of unix will help unix in general. (i imagine a posix standard to access metadata...). ~ibotty

Nice - Joel - 2003-01-08

Nice concept. If made user friendly and intuitive it could be a real killer. I'd really love to see something like this that works, and works well. Something which can quickly and easily bring the data to you. I think this should become a standard part of KDE, but I think the current model should also be available for those who would prefer to use it. Also, the system would have to be compatible with the way the current file system works so that files created in kde apps can be accessable with other apps etc.

Offtopic: run vim on middle button click - Alex - 2003-01-08

Hi, sorry for an offtopic question, but does anyone here know a trick to make KDE konqueror shell run "gvim -R filename" whenever I click the middle mouse button? Or where could I ask? Thank you! Alex

Re: Offtopic: run vim on middle button click - rinse - 2003-01-08

kde-linux@mail.kde.org Rinse

Re: Offtopic: run vim on middle button click - Philippe Fremy - 2003-01-09

GVim is obsolete. Try KVim and the KVim KPart: http://www.freehackers.org/kvim/

Files stored on a server? - Flemming Frandsen - 2003-01-08

How well would this work if the files were stored on a central server? I realize that this is a very early implementation, but getting too wrapped up in making it simple for one user to use could end up making it impossible to use for more than one user and that would be a shame. IMHO this would be most usefull in companies where different people will need to share access to a large amount of documents. There are two points to this post: 1) Please make it possible to have a file server to serve the files and have that fileserver tell the clients what process to talk to to get metadata, as that will enable clients to mount document directories from many different servers in different places and keep the same global view. 2) Please keep the SQL/database interface separate, it would be a great loss to be tied to a singleuser, file based sql-liberary, in fact I think you should have a server process that takes care of all database access (and caching). It would be very nice if you could install a company-global filesystem simply by creating a database instance on the enterprise database box and a coda/nfs/smb/whatever filesystem and then start the metadata-server pointing it at the directory and the database instance. You would probably also need some sort of replication of the metadata or a disconnected (laptop or whatever) user would be SOL. I don't know if including the actual file serving in the metadata-server would be a good idea, it would make setting things up much simpler, but it could also mean replicating a lot of work... Anyway, this seems to be a very cool idea, good luck!

Re: Files stored on a server? - ibotty - 2003-01-09

if using a (virtual) filesystem approach (as suggested above) one could easily mount it via nfs, coda, or whatever you want (smb might be harder, it is not fully posix conform, is it?)

Re: Files stored on a server? - moses - 2005-12-06

I am in need of du\ocumentation on compiler technics or with construction of processing,utility programme,liberary file Thanks Mr.Moses

Great Inovation, but problematic GUI - Hubert Krause - 2003-01-08

I think, this is the sort of inovation that "Linux/UNIX on Desktops" is missing. Something new, not a copy from Windows or MacOS. But for succsess it has to get a nice and intuitive GUI, perhaps an kio plugin to "browse" your dokuments similar to the "well trained" classical HFS-way. Further on it should integrate with the classical open and save-dialog, cause you will need both ways to access your documents (for example: configuration-files will depend on HFS the next few years I think ;) Maybe it is time to learn C++.

Doesn't look easy to me... - Stof - 2003-01-08

Most users (read: Windows users and traditional Unix users) are all used to the HFS. In order to use this system, you must learn something new. We all know "average users" don't want to learn, they just want to get the thing to work. Won't this get lots of resistance from users? I took a look at the screenshot, and honestly I can't figure out how it works just by looking at it (maybe it's just because of the GUI?). To me, it doesn't look intuitive at all. I can already see it now: thousands and thousands of people flaming about how Linux is difficult to use and how it sucks and how Windows XP rules. *sigh* Maybe this new way is "better", but I doubt it will succeed until users change their attitude and are willing to learn.

Re: Doesn't look easy to me... - Peter - 2003-01-08

Hi Stof I doubt that those users understand the HFS, they just memorized what to click when they want to open or save something. I have ~100 Users, and most of them expect that a double-click on an icon on their desktop will show them the files on the server even when they are not connected to a network - the whole concept of (network)drives, home directories and shortcuts is not really understood by most of them. So, the new approach _does_ force them to learn something new, but it is not that they would have to learn to think different - they have the chance to learn something sensible instead of learning a click-sequence by heart. So, if this could be made option in KDE - go ahead. Cheers Peter

Interoperability - Jonas - 2003-01-08

I think the author is unaware of all the other work on metadata over the years. This is nothing new in itself, but it work be great if KDE could get some user friendly metadata support in the open and save dialogs. *But*! You don't have to remove file names just because you introduce metadata. Filenames are there for a reason. They make interoperability so much more important, because they are a least common denominator between file systems and because they serve as a unique global identification (which is human readable and pronouncable). I don't want to dis the author for being an enthusiast and actually producing code, but I get the impression that he hasn't learned from history yet. So read up on all the work concerning metadata over the years! There's lots and lots of it and many good thoughts has been thought on the subject. (I think ArsTechnica.com had a good overview perhaps a year ago?) Summing up: Use filenames (and treat is as a metadata field)! Stick to a standard metadata layout! Look at file systems with metadata! (Especially reiser4!) Interoperate!

Think about names - pos - 2003-01-08

A great resource for thinking about file names and paths is located at the W3: http://www.w3.org/DesignIssues/NameMyth.html It's quite easy to read and is not as heavy as W3 specs are. Also, for anyone implementing a metadata scheme, I highly recommend all of the documents listed under the Semantic Web section of http://www.w3.org/DesignIssues/Overview.html Someone above pointed out the similarities to the Dublin Core project. I would like to add that anyone developing this system or intested in Metadata representations also check out: http://www.semanticweb.org/ http://DAML.SemanticWeb.org http://www.sciam.com/2001/0501issue/0501berners-lee.html http://www.w3.org/RDF/

Uh? - Datschge - 2003-01-09

Am I reading correctly, W3 recommends people to omit the file extension? o.O >>The last element of the path, "html" is not strictly necessary with most servers, as at least some servers will, given a URL of "readthis" , serve up the data from a file which is called "readthis.html". Here the student is making it difficult for himself later to change the format or formats in which the file is available, without at least some confusion. Suppose, for example, that he later decides that the information is worth providing in audio format for blind readers. The CERN server can easily be configured so that clients specifically requesting audio formats in preference to HTML can be served as preferentially whereas more normal clients will get the HTML. So, here again is a part of the path which may be later regretted.<<

This can overcome a long term desideratum - Georg - 2003-01-08

Well designed and implemented, newdocms would contribute an important an innovative part to future versions of KDE. "Well designed" comprises the consideration of teh following needs: - convenient GUI that lacks nothing of todays HFS GUIs - interface to "classic" programs (via virtual / live file system) - support of different metadata-systems (e.g. IPTC, Dublin Core,...). Modularity recommended. - multi-user and multi-machine / network support - full-text-retrieval - a powerful API - solutions for backup and data migration - switching from and to HFS should be easy Nice additional features: - implementation of non textual search (e.g. gift -> fer-de-lance.org) - cooperation with OCR systsems and integrated image-capture systems (e.g. kooka).

great! - fred - 2003-01-08

a very intelligent and highly effective idea. I only have a couple of comments: - the project, as it has already been said, badly needs a different GUI, so intuitive as to require no extra information at all about how the system works. Now, this system makes it possible. For instance, next to a preview of the file to be saved, there would be several drop-down lists answering questions like: what type of document is it? about what is it? and so on. There would already be a list of possible answers: photo, music and son, this is of capital importance. It is absolutely necessary that the basic category tree is ALREADY present - believe me, if the user had to provide its own category tree, 90% of them would not come up even with document -> photo. Of course advanced users could extend and modify the given category tree, but it must be there and must be standard (the dublin one is an excellent proposal). The same of course, would hold for the open dialogue - did you notice how this technically and conceptually very simple proposal does much more to change the way we interact with compters than the whole slicker nonesense put together?

nice, but should be as an option - somekool - 2003-01-08

im sure this features will rock. but im not sure im ready to lose my traditional file system. maybe they could be use together, or we may need tool for reverting back thinks like they was...

think Google - soderic - 2003-01-08

I want to push the overall goal just a bit further. What if I didn't have to think about organizing my data at all? For example, "save" doesn't need any kind of dialogue box and "open" works like Google. Save just puts it somewhere and an indexer looks at it and collects whatever it needs to enable a simple query based open. When I want to "open" something, I get a single text field where I type whatever I want and I get back a list of results I can browse, or I can further refine the query. Perhaps this is simple and too naive. However, I have spent many hours trying to explain the concept of hierarchical folders to my wife and my father and something like this would eliminate this hurdle. Anyone can use Google.

Re: think Google - Anonymous Coward - 2003-01-15

"and "open" works like Google" Sure, nice, but you shouldn't forget that Googles usefulness hinges heavily ont the pagerank system. How do you intend to do that without links between the documents?

Yes, really innovating but... - trukulo - 2003-01-08

But doesn't seem too similar to the new windows file system? I'm not an expert, but the concept is the same: http://news.com.com/2009-1017-857509.html

It is as complicated - antialias - 2003-01-08

as hierarhical file system and it requires a lot of work to get things organized. It can be useful for libraries, universities etc. but not for the end user. Database concept is great when somebody else do the work for you and there where is a huge amount of entries. I don't have thousands of documents and will never have them. I periodicaly do some cleaning in my apartment and throw old documents into the thrash bin. Everyone does it. I also periodicaly clean up my computer for garbage and old documents. And I can easily find the documents I need. Not a big deal. When I want to search for some documents that interest me I do gg: in Konqueror. Even if you do your own database localy (which is a huge job) there will always be a bigger and better database stored on internet. Database approach is as much complicated as hierarhical file system if you have many documents with similar or same attributes: then they don't fit to an open window, you have to scroll down or click on the next page and next page etc. Many of you have tried to search for a book(s) at a local library database and got a thousands of entries. There is nothing wrong with hierarhical file system when you keep it clean. Database concept gets hierarhical as well when you get too many results. And here is the problem: when you have a few documents you can easily find them, but when you you have too many of them you need an amount of time to find them no matter if you use a hierarhical file system or a database. (see the screenshot of Save - in save dialog there is an option called 'Edit category tree' - so even here there is a 'tree' and you have hierarhical system inside) Some people say that adding attributes is great because you can for example categorize your mp3 files by genre, author, album and year. And my question is: Is it so difficult to create appropriate directories and then create playlists out of them? By the way, this story has already been discussed on Slashdot (the author of newdocms Manuel Arriaga posted it to Slashdot). ---------------------------------------------------------------- 'The filesystem paradigm has been around for a long time, litterally thousands of years, because it works, it is easy, and it is how people think.' ACNeal on Slashdot

Re: It is as complicated - Shift - 2003-01-08

Moreover KDE3.1 can know acces META Informations of the files (jpeg, gif, mp3, ogg, video,...) and you can use kfind to search for them. For example I can go into my mp3's directory an search file with meta data containing Madonna and I can found all Madonna's mp3. I can search for jpeg photos taken by a Canon camera in a thousand of other files. I can search for videos with 400x300 resolutions ... Just test kfind of KDE3.1 and test "Properties" of different files in konqui to understand what I am talking about :)

Re: It is as complicated - Georg - 2003-01-08

The hierarchical filessystem is a restriction in man machine-interaction. Human thinking is more associative than merely hierarchical. I don't want to emulate the associative approach using symlinks. It's to easy to get messed up. I *have* thousands of documents and will *ever* have them. I never do some cleaning in my computer and i don't throw old documents into the thrash bin. And I don't want to be forced to do so. Everyone does it (No!). It is hard for me to find the documents I need. It is a big deal. I'm very dissatisfied that finding documents on the local disks is soo slooow compared to search engines on the web. Data based approch can be partly automated by content aware assistants that gather metadata or physical features (colour vs black&white, resolution) from the documents (e.g. metatags in html), from open databases (CDDB, ID3-Tags). Other helpers: using templates and a list of "recently used storage constellations". It's a matter of GUI-Design. >Some people say that adding attributes is great because you can for example >categorize your mp3 files by genre, author, album and year. And my question is: >Is it so difficult to create appropriate directories and then create playlists >out of them? What do you do if you want to access your mp3s in another "sort order"? >Database approach is as much complicated as hierarhical file system if you have >many documents with similar or same attributes: then they don't fit to an open >window, you have to scroll down or click on the next page and next page etc. >Many of you have tried to search for a book(s) at a local library database and >got a thousands of entries. That can be the case. Counteragents: - assistants (see above) - powerfull search options (boolean, thesaurus based, fuzzy) - a little bit experience and discipline > It can be useful for libraries, universities You are right. Both are important target groups for KDE and they should get appropriate tools. > but not for the end user. I'm an end user. OK, one of the power sort ;-). And I really need it! Georg

Re: It is as complicated - David Johnson - 2003-01-08

Human thinking is more associative than merely hierarchical. That's funny, because when you look at human beings manipulating real world documents (books, letters, etc), you'll find that they use a hierachical system by and large. Books go on a book shelf. Magazines go in a magazine rack. All the old love letters go in a frilly box labelled "love letters". Take a look at an office an you'll see file cabinets, drawers, and folders. Associative metadata is good, but not at the expense of the hiearchy. Metadata is great for making new "views" of the data, but they shouldn't be the "model".

Re: It is as complicated - André Somers - 2003-01-08

So tell me, how would you store your books associative in a 3d space (where a book can only be at one place at a time)? Precisely: you don't, because you can't. However, this physical limit does not exist in storing documents in electronic systems, so why limit ourselves to it? I know some people like Yahoo's hierarchical webindex, but I guess Googles associative index is far more popular...

Re: It is as complicated - antialias - 2003-01-08

'The hierarchical filessystem is a restriction in man machine-interaction.' No. It is a requirement for a man-machine interaction. As I pointed before: do a search in any database or search engine and the results will be presented hierarchicaly. Scroll down, next page. What do you get? Another file manager/browser inside the database dialog. And when you do search in Google you get castrated hierarchy - links organized in pages and pages are named 1,2,3,4,5... Can you see or guess what is on the page nr. 10? No. 'Human thinking is more associative than merely hierarchical.' You are right. Human thinking is more associative, but that is a problem. We forget things, and therefor we use hierarchical system for organizing things. Associative thinking is not powerful enough because we forget or better said suppress things. ' I *have* thousands of documents and will *ever* have them.' Some people collect Barbies, and some do collect documents ;-) 'It is hard for me to find the documents I need.' If it is hard for you to find the documents you need, you really don't need them. Otherwise you will know where did you put them. We remember only things we are really interested in. 'What do you do if you want to access your mp3s in another "sort order"?' I move them around. I create another playlist. I know where my ogg files are, I know who the artist is, and I know what the genre is. Because I am interested in exactly that kind of music I have stored on my computer.

Re: It is as complicated - KDE User - 2003-01-09

> If it is hard for you to find the documents you need, you really don't need them. Otherwise you will know where did you put them. We remember only things we are really interested in. No. If you will never remember nor be reminded of a document, then and only then you do not need it. If you however at some point in time do recall the existance of a document it is normal not to remember where it is. A good example is a photo from your childhood.

Some replies... - André Somers - 2003-01-08

Hi, As the author of the article (altough it got edited quite a bit after posting), I would like to address some of the questions raised. Note: I am not the author of newdocms, nor am I affiliated with it. The idea is of course not completely new, there have been other attempts to make something like this. This project is very new though, and has received many, many suggestions for enhancements allready. He is open to suggestions, and is well aware that there are flaws in the way the system behaves at the moment. Still, I share his belief that the basic idea is very valid. A lot of the questions raised above are answered by just really reading the site. I urge everyone interested to do that, before critising the idea based on the screenshots. A few comments though: -The system is not depending on KDE. It uses a general backend server, and frontends for for instance GNOME and OpenOffice will be implemented too. KDE just provided a convient testing ground as I understand it... :-) -The hierarchical "category"tree that is mentioned is NOT just another tree to replace the one of the filesytem. It represents an ontology of the world instead. The first idea was that the user would build his own ontology, but there have been suggestions that it might be possible to use allready existing ontologies that are around. These ontologies make it possible for the system to perform fuzzy queries, that will enhance the searchresults if you are retreiving a document. -A pure hierarchical structure can never hold the same semantic value as using real metadata, because the data is not labeled. A very good example can be found on the newdocms webpage. -Multiple databackends, both for personal and for corporate use, have been envisioned. -Automatic inclusion of as much relevant metadata as possible from the files into the system is of course needed. That is largely a question of interface though, and KDE with it's KFilePlugins would be VERY usable for this! And yes, maybe if ways of automaticly generating enough metadata become available, it may be possible to stop bothering the user at all with entering it, if he want to. -The presentation of the data has not received the attention it deserves yet. Sure, a lot need to be done in terms of cleaning-up the interface and making it more userfriendly. That was not the purpose of this release. The purpose was to show the basic technology to the world, and get the attention of people wanting to help to develop this further and take the concept to a level where it becomes usefull for the average user. -This system will not replace your current filesystem, it will be build on top of it. André

Thoughts on this... - Geoff Rivell - 2003-01-08

I think the idea is good, but the implementation isn't. Perhaps this would be a better way (off the top of my head): Instead of a flat tree of unique numbers for storing those files, why not just let the user store the file like usual, and then be able to edit the meta-data (through properties dialog, maybe even on the SAVE dialog). You could have the files' meta-data stored in meta.db for each directory. And then when you want to do a search (or open) the dialog could have a "search meta-data" option. You select /Documents/ and do all the searches you have presented. This has many advantages: 1) You can use command line tools on those files without them needing to be modified and it would be much more simple to "kedit my_diary_1_02_03.txt" then seeing 124321341.txt at a command line. 2) Someone mentioned about fileserving earlier. This keeps everything right for Samba and the like. And client "meta-data aware" apps can use the meta.db while others can be on Windows and still get reasonable use (if thats what you call it <rim shot>). 3) It also means that SHOULD some simple minded user delete meta.db, that not everything is lost. This is a project for KDE and GNOME to work together on for it to be an advantage for Unix/clones. Hope this gets implemented :-)

Recommendations and wishes - Georg - 2003-01-08

- Powerful tools for ex- import - Manual, semi-automatic (proposal based on contents with man-machine interaction) or fully automatic generation of file-names. - The tool or its descendants should be capable to give access to all stored data, including bookmarks, addresses, emails and attachments, Web-Archives (.war-files). - Storing of metadata should be robust and fault-tolerant. So i think that "a little bit" redundancy would be OK. Why not store documents and metadata in RPM-like archives or folders, so that the databases can simply be recontructed? It would also open a nice way to send metadata-enriched documents via email. The recipient can make use of some or all of the given categries and attributes.

Great... but don't go it alone - Joe Theriault - 2003-01-08

I definitely think some FS abstraction concept like this is really necessary for Joe User... how many people's parents actually touch "My Computer" on their Windoze boxes? And this seems like a good way about it, but I do have one concern: that KDE will go it alone. That it will have a great system like this, but it will mash up the real file system in some way that it's unusable in any non-KDE app... like bash. Because if they're all to be found in $HOME/.kde/share/Newdocms with names like kword-01.09.03-12.kwd or something, it will be incredibly not useful outside a KDE open file dialog. Maybe a suite of command-line utilities or something is the way to go. I'm just afraid of a system that requires 100% KDE and hinders leaving it... that seems very anti-trust-like. Just my $0.02.

Re: Great... but don't go it alone - Daan - 2003-01-14

Perhaps you could generate a nice filename out of the metadata, even with a hiearchical structure. For example, if I save this webpage, it would be stored as, for example "~/webpages/fora/KDE/unknown/Great... but don't go it alone.html", thus "~/type/category/subject/author/title". This way, you can easily find a document back in HFS, but the new system still has great advantages: you can give queries to find everything about KDE, or everything written by John Petersen... and you don't have to make the directory structure yourself!

Re: Great... but don't go it alone - anonymous - 2003-01-14

Great idea! Probably impossible to do though.

of course we want it - socialEcologist - 2003-01-08

In fact everyone agress that it can be a good idea. The advantage of the open source model is that even if the majority doesn't want it, the minority that is interested in it can develop it. Just do a wonderful job and everyone will use it. This kind of things can send linux light years from windows and allow more intuitive uses of our computers and is in the direction of high-level semantic interactions with them. Show us that it works and rocks ! :)

Offtopic: (not so much) .app Folders for Linux - Mike Mitterer - 2003-01-08

Is there somewhere a discussion about .app Folders on Linux? Because we are talking about inovation - has anyone seen how easy these .app folders work under OSX? You can install the most applications by just drag and drop them to the harddisk. Everything a Application needs is in this folder. The lib's (most of them), the Application itself, the icons and so on... THIS would be a great inovation too. Mike

Re: Offtopic: (not so much) .app Folders for Linux - ibotty - 2003-01-09

the way you describe those app folders, there are some disadvantages: - every app has their libraries inside their main bundle meaning, that you'd have many (slighly different) versions of a lib on your filesystem. imagine every kde app, having a libqt.so.3 and all kde apps (and standard icons). a waste of resources. this is why you dont have all libraries in anApp.app/. there are libraries in another dir ( mac uses /Frameworks, if i recall correctly) nevertheless, on unix where most admins have a package management system, you dont need the "all resources in one place" app folder approach. but there are other advantages. app folders, or lets call them bundles, because all the following applies to frameworks (bundles with .framework suffix on mac) may store more than one binary in the bundle. they may store one i386 (linux) binary, one alpha (linux) and one i386 (win) binary. if your DE supports app folders, you could then start an app on your remotly mounted volume. and it would start the right binary. this is an advantage. there are many others... (this is why bundles are even used by gnustep on linux, etc) nevertheless, i dont think kde will use frameworks and app bundles in the near term. they simply are too much unfamiliar to most developers (and even admins). ~ibotty

Re: Offtopic: (not so much) .app Folders for Linux - Mitch - 2003-01-10

Maybe a solution with redundancy checking on a "graft" or "stow" like basis could help. Some overhead would of course be a consequence. -Mitch

Re: Offtopic: (not so much) .app Folders for Linux - Jo Edwards - 2004-05-13

This is actually a very user friendly concept but only works if there is a known "base system" with a known set of installed libs. One approach is described in http://klik.berlios.de/architecture/ It would indeed be nice if an action could be attached to a KDE folder to launch an app instead of opening the folder.

Not quite yet to the obvious - Eric Laffoon - 2003-01-08

There are good points on both sides here because there are solid pros and cons. People are by nature inert and will stick with what they know long beyond rational explanation. Some people have very few documents too. Newdocms has advantages and flexibilities that look very useful and it addresses nearly all concerns. But as good as it looks your bleeding edge early adopter tech types will still want their docs in an HFS and with clear names... so I see a limited adoption of this, however good the idea is. Many people have mentioned standard attributes too... Here is how I believe this could best be implemented and it would take a little more work but it would require less discomfort while providing the best of all worlds and perhaps adding some value. First, use the HFS. Make the metadata an extended attribute of the file name. It is possible to reference this with paths, though it would be nice to have a low level daemon update the database with moves and deletes. Data could be entered in the file save dialog. To enforce the metadata concept a .metarules file could provide information to the file save dialog about what the directory owner requires a string in or provide dropdown lists. Indexing could still occur from a central database as is currently indicated or it could add a parameter for where in the file tree this must exist to narrow the search. The metadata could be stored in a hidden folder such as ~/.filemeta and communications systems put in place for servers to register their databases to the user who could also select whether to include server and local in a search. Much of this would happen below KDE so it would be usable with very little modification by GNOME and others. This is a more ambitous and complicated solution than proposed but it would work. newdocms purists could configure save dialogs to automatically construct a name for a file. Intransigent HFS users could still realize a benefit by setting required field flags in data directories to remind them to create document search terms. People who want nothing to do with it need never look at it, interoperability is preserved and other desktops can adopt it easily. It is possible to have "File Open" and "Meta Open" dialogs or a directory/meta open dialog. Everybody wins. I can now use either or both whenever I want with any application because I get it automatically with the KDE standard actions.

good idea - but - godot - 2003-01-09

Sorry, could't test it for now (so you may stop reading here) I agree with the idea of a better organisation of stored files and (regarding my mom) know the problem of overfull desktops and unorganized harddisks (but "Search" is so powerful). Adding special information to documents is a good (and important) idea for a better organisation. The problem I see is, that everyone has its own way of storing data - and this doesn't seem to be solved with this approach. Again my mom's desktop: ever tried to find a special file there? No chance. If you ask where it is you get the answer seach for it (F3)... but what to look for? Not only "wrong" interpretation of HFS-Trees but wrong interpretation of Keywords is possible. A reviewer would perhaps interpretate "Auther" as original auther he writes about but not himself - perhaps because every document on his computer has been written by him. Next problem: where to know from, if every document has a valid author? Metadata is not NULL? Metadata has been updated to the document content? There are many things not only depending on open/save dialogs but going much further into the application. A word processor e.g. could answer the question if you have an artical, a simple note, a telefax, slides itself. Music (mp3, ogg) has its own system that should be read when saving files from your favorite browser. So why that? Because I believe that it is not only the HFS that doesn't provide a good possibility of organisation but also the user, that doesn't want to spend time on it (see full desktop mentioned by David). "Work's over so just save anywhere and try to reach the bus." I prefer the mentioned cross referenced-filesystem: User Interface should provide a folder like system where all data should appear everywhere it belong to. Let's say you have common KDE open-file dialog. On the left you would find some sort of what you find now like but you can choose between attributes: "Text" "Sound" "Video". Choosing Text you get something like: By MIME-Types, by Type (Article, Slide), by Author,... Then I can browse: "I" have wirtten a "TEXT"-file"2 Jears ago" containing "SLIDES" just as a document tree I am used too, but it should be able to reorganise to be able to find it in: Written for the "UNIVERSITY" about "COMPUTER GRAPHICS". Creating new folders would have the same framework as the nowaday system: Choose your subfolder in your favorite browser and do a "create new". One other important thing: The system must be transparent for any application: Gnome, KDE, emacs, vi(!). I just want to "cd" in it. I still want to "scp" my files to where I need them and copy them back - without the need of KDE or something similar. But this would imply a new virtual filesystem provided by a special system wide library. A filesystem capable of collecting, storing and offering meta information for special document paths in user-home-directories. There really is the need of a better organisation and your ideas could (and should) be part of it - but this is just a small step towards it. A good step, but as you see in other comments, too, a really complicated one. A long way to go but no reason to give up. Greetz, godot

Abstracting HCI and rethinking implementation - Troy Unrau - 2003-01-09

I'm am pleased that someone has beat me to this :) First off, I'll make a disclaimer - I'm a star trek fan. Second, in that show, and yes I'm aware it's fiction, they never access information via filenames. They run a search on given parameters, narrowing it down logically (voice acticated, blah!, that's after the concept is sound). If the information is personal, they access it something along the lines of 'Computer, access personal recording of trip to Risa' and it would start playing the first recording it found that matched that criteria. If they wanted a later recording, something along the lines of 'Forward two days' would jump it forward. This is how I dream a computer will work one day - everything like Google except cooler. If it's a personal document, it'll be tagged as such, if it's a private document, it'd be tagged as such, and if it was public access (like a web-page?) the same would hold true. I've been doing some studying and conceptualisation of this privately for the last little while (sorry guys on IRC, I miss you!) but have determined that it would be terribly hard to properly integrate this into an interface as it exists today. Frist off, the meta-tagging would have to be mostly automatic (which can be done with some smart routines in the save dialog). Second, the user would not be able to be aware of the underlying HFS at *all* for it to work smoothly. A quick-access designation could be assigned to commonly used documents that would bring it up faster than doing the normal meta data by adding 'among recent documents' or somesuch -- anyway - lots of work to do - so little coding skills :P sigh. working on that. I'm working on an RFC for now - if anyone's interested in more info/assisting/brainstorming - jiilik(a)bytebenders,com (figure it out :P) oh, and I definately want this to be open source if I can pull it off. Troy Unrau used to be troy@kde.org - too much spam there now

OT: spam - Navindra Umanee - 2003-01-09

How could you be getting all that spam? Didn't coolo install Spam Assassin? Seems to work fine here.

Re: OT: spam - Troy Unrau - 2003-01-10

All the spam I'm getting is a) in korean an b) is a result of a serucity hole that my ISP refuses to properly fix. (an email address that when sent to, ends up at the whole subscriber base). So the spam isn't channelling through the kde.org address - it the address beign forwarded to that's the problem. I have a new pop adress - just waiting for .forward to get fixed. Troy Unrau jiilik(a)bytebenders,com

I have been working on the same thing - James - 2003-01-10

I have been working on something like this useing java and a web interface, It allows meta-data to be added to databases/File systems. I am just useing Http as the network protocal which just happens to work with a web browser :-) I have no idea how I would intergrate something like this into the File System viewers.

More freedom concerning filenames - Georg - 2003-01-10

Putting all files in the ~/docs folder using crypting file names is an easy solution for the data mangement system but not for all users. I think that a more sophisticated approach should be developed, that support a thrid way: First way: Classic HFS. Second way: newdocms in the proposed form Third way: A mixed mode. Metadata and other info attached to documents outside of the ~/docs-folder with filename and path. The additional data are stored in the document itself (if possible (eg. html,xml,tiff,jpeg), or in an additional .db-file, that is normally invisible in KDE-applications. To maximize coherence the "main"-file and the .db-file can be stored in a RPM-Archive or a common folder. Example a metadata-enriched "picture.raw" would be moved into a folder "picture.raw", together with "picture.raw.db". A demon monitors consistency-leaks due to pure HFS-programs. This or another demon should also offer an Virtual FS, that allows browsing files by categories and attributes. This feature would be usefull for pure HFS-progs and KDE-apps as well.

Have a look at Reiser4 (the "next" Linux FS) then. - Dieter Nützel - 2003-01-11

Your FS become a database, like BeOS tried some years ago. [-] Snapshot of reiser4 source code can be found at http://www.namesys.com/snapshots/. It is set of patches against current Linus BK tree. Reiser4 is the next version of ReiserFS file system. It was re-written from the scratch. It supports: - full data journalling with "wandered logs" ("shadows" in DB parlance); - extent-based files; - delayed allocation of disk space and on-line optimization of disk layout across file boundaries; - plugins: infrastructure for easy extention of file system and utils functionality; - and a lot more, see http://www.namesys.com/v4/v4.html Snapshot contains reiser4 proper (fs_reiser4.diff), set of patches (described in READ.ME) with necessary changes to the core kernel, and utils package (in particlar, mkfs.reiser4). It is still crasheable. Do not put critical data on it. Nikita. [-] And here some notes about the speed: http://www.namesys.com/v4/fast_reiser4.html Regards, Dieter BTW Hans Reiser did a demonstration for Apple on it.

An idea - David Findlay - 2003-01-11

Maybe instead of having to enter a category every time, you could have a list of categories you have already set up and select one from the box, then a subcategory... etc. I'd like to see something new like this, but it'd have to be good. Thanks David

Sounds like a plugin for ReiserFS - Doc Funfrock - 2003-01-11

The management of metadata is a typical design goal for ReiserFS, because this FS is already designed like database. But it's a disadvantage to design such a system for a specific FS Doc Funfrock

You'll meet some psychological resistance... - Mike Forbes - 2003-01-12

... to this idea, but I love your implementation. Given your inclusion of the the ability to save w/ no extra information other than the MIME type, and the fact that the metadata is all stored in a database, I don't see it being as big a problem as I originally feared. My suggestions are few: a) Make it work with any common database backend (i.e., MySQL, Oracle, etc). b) Make it work with NFS c) Make sure to copyright your work under GNU... if necessary to protect the idea and ensure it remains opensource, patent any new technologies you've created for it.

Middle Ground - Alex - 2003-01-15

One thing that really annoys me is having to search for things that you would otherwise not have to. For example, file-open boxes that don't give you a place to simply type in the path/filename of whatever you want to open (the quickest option if you know it), but actually having to look through directories and clicking several times to get to it. I would never opt for a filename-less system because of this. Even if I don't remember what a file is called, or where I put it, all I have to do is ask myself a question: if I was going to save it now, where would I put it? There is something to be said in knowing the way that you think, and forming habits where our memory isn't sufficient. However, perhaps an 'index' tickbox on the save dialog box, that you can enter searchable information about the doc, and enter it into a central database which you can search later should the filename elude you would be a middle-ground solution, perhaps it could even suggest a filename based on that information for those who struggle? We seem far too obsessed with making computers require less thought to use, in general, not just data retrieval. Maybe I'm just being old fashioned, and maybe (most likely) it's the way forward for computers. But is it the way forward for mankind?

Assimilate existing metadata - Dirt Road - 2003-01-16

A system like newdocms is only as useful as the number of (properly tagged) documents it contains. Let's say I'm a technical writer with a big pile of documentation... the first thing I'd have to do is add all my existing documents to the database. Tagging all those files by hand would be daunting enough to keep many people from even starting. However, a lot of files essentially describe themselves to a point. An otherwise-untagged PNG or JPEG file could be assumed to be a picture, for example. When I come back looking for that picture, I can just look for pictures and not Kword files. Some digital cameras add a little extra info, perhaps dating it, and that info could be useful (say I'm looking for a picture I took on January 3 but I added it to the database on March 7 -- I know I took the picture some time in early January, and I don't care when I put it in the database). MP3s have lots of handy tags as well. Text files contain even more metadata that could be assimilated. If I write a paper using troff -ms macros, there are tags like .AU (author), .AI (institution), .TL (title), and some specifying the document type (report etc.). Many XML doc types, and even some binary formats, have at least as much info. All of that could be used when adding a file to the database; newdocms could simply take advantage of it and present the user with a suggested set of tags (which the user could approve or change). Being able to analyze incoming documents would go a long way toward general acceptance of something like newdocms. An adaptive system, that learns how each user likes to categorize things over time, could be difficult to implement but would be even more difficult to walk away from. :-) -- Dirt Road

Better FS organisation: Replace Tree with Forest ? - Eugen Dedu - 2003-01-16

My idea to face with the organisational problem of files was to replace the TREE relationship with a FOREST relationship. That way, using "directories" as metadata you can find easily (by "descending" in the forest) the document you are searching for. http://www.prism.uvsq.fr/~dedu/docs/kb/ provides more information, if you find it useful.