New Plans for KDE Multimedia

Stefan Westerfeld has posted a first draft of cool new plans for the new multimedia architecture for KDE2, based on aRts (the "analog realtime synthesizer"). Plans include merging the two existing media players (noatun and kaiman), new media types, infrastructure improvements, improved MIDI and more. The full draft (edited by our own Dre) is attached below.

KDE2.1 Multimedia Plan (draft)

KDE 2.0 was quite a step for multimedia on the UNIX® desktop. aRts, which has been integrated into KDE 2 over the past year, provides an always running soundserver and service-type media-playing. The notification system, backends and the
media player were completely rewritten. As a result users have a more powerful platform
for multimedia tasks.

Still, there remains a lot to be done. This document tries to provide a
planning and discussion base for coordinating "the way to go". The focus
lies on what can be achieved for KDE 2.1.

Discussion by the community is highly welcome. Also, please consider contributing to further development.

This draft has spontaneously been designed on IRC, by

  • Charles Samuels
  • Stefan Schimanski
  • Stefan Westerfeld

Overview

  1. The KDE Media Players
  2. New media types
  3. Organization & Documentation
  4. Midi/Sequencer
  5. Not in 2.1

1. The KDE Media Players

1.1 Merging noatun and kaiman

Kaiman and noatun are two seperate media players available
for KDE2.0. Kaiman was shipped with 2.0, and noatun is available via CVS. The plan
for 2.1 is to combine these players into one media player, the official "KDE Media Player".

Technically, the KDE Media Player will be based on noatun's plugin architecture. Kaiman would be
just one player making use of the noatun technology.

1.2 Pluggable effects

aRts allows much more than vanilla playback: filters can be used to affect the sound. Currently, this feature has been mostly unavailable
in media player(s), mostly for two reasons:

  1. there were few effects; and
  2. the binding of a GUI to effects was not yet implemented.

The fix for (1) seems obvious: write more effects. One starting point could
be porting the FreeVerb code to the aRts architecture.

The fix for (2) is less obvious. The problem is forumlating a method for GUI objects (which run in the player process) to connect to the
effects (which run in the sound server process). The clean way to do this
is to extend MCOP (the inter-process communication architecture used by aRts) to provide a signals & slots technology.

An example best illustrates this point. Say for instance there is a GUISlider aRts object, which
runs in the player process space, and a ExtraStereo effects plugin, which runs in
the aRtsd process space. The idea is to be able to write:

connect(slider,"value_changed",effect,"depth");

and thereby cause a connection between the slider and the effect. Conversely, the same mechanism can also be used to connect output GUI elements to
effects.

Besides replacing the need to poll, this approach also allows the GUI to have a flexible deisgn, using the same modular approach that artsbuilder
currently provides for sound.

1.3 Moving the tag reading code

The ID3 tag reading code, which provides information such as title and author from .mp3 files, is currently a noatun plugin. It should be made an aRts components to make it available to all applications using aRts.

2. New media types

2.1 mikmod

Work to make an aRts PlayObject for mikmod files is currently progressing. This
would allow the KDE Media Player to play .xm, .s3m, .mod and other digital audio codecs. The clear advantage is that the component would be available to all other apps which know how to talk to aRts.

2.2 MIDI via timidity

Another great project would be to make a timidity plugin for aRts. Timidity is a
package which renders MIDI notes to audio output using sampling. Currently kmidi is the KDE MIDI player. Converting kmidi to an aRts PlayObject would again
provide the user with a much more consistent experience and provide the playback capabilities to all applications aware of aRts.

3. Organization & Documentation

3.1 KDE multimedia webpage

KDE multimedia development needs to be more visible. There is no official
coordination point where people can see the development process and design overviews. To address this shortcoming, a KDE multimedia webpage (how does http://multimedia.kde.org sound?)
should be created.

Charles has volunteered to take care of this.

3.2 aRts documentation

Fortunately the multimedia chapter of the KDE 2.0 Development book should
become available to the public RSN. While this is a significant piece of
good developer documentation and a great start, more work
needs to go into a consistent documentation on http://www.arts-project.org,
for both developers and users.

3.3 kdelibs/arts -> arts

Packaging aRts seperately is probably a good idea. aRts itself by design does not
depend on Qt or KDE. People outside the KDE world should
still be able to use aRts as a sound server or for music tasks.

If - for instance - GNOME would start using aRts, having aRts mixed in the
CVS and in packaging with KDE is probably a bad idea.

4. Midi/Sequencer

4.1 Usability issues

aRts in the CVS already provides MIDI realtime synthesis. You can use midisend
to play instruments in artsbuilder. You can also combine it with Brahms or
artstracker to compose songs. The problem is that if you are not
a rocket scientist or don't study the code or collected READMEs for a while, you
will probably not manage to do it. That has to change.

A good start would be providing an instrument manager, where you can
graphically assign which MIDI channel should get which instruments, without
bothering with the details.

4.2 Interoperability

There are at least three sequencer-like programs which actively want to talk
to aRts:

  • Brahms, which has been in the CVS for some time now. Jan, it's author, is
    working on a new version right now - some aRts to MIDI support is already in place.
  • Anthem is in its infancy. Its author Pete definitely wants to have
    audio stuff like hd recording, and probably aRts to MIDI support should
    be added as well.
  • ArtsTracker is an experimental small tracker in kmusic.

Especially the areas

  • connecting to aRts
  • finding (virtual) MIDI channels and synthesis instruments
  • assigning instruments to channels
  • sending timestamped events

need to be standarized more for MIDI. Other interesting endeavors include:

  • harddisk recording; and
  • effect processing

Most of this only needs a small amount of code in aRts and the applications,
the problem is finding a sophisticated standard.

5. Not in 2.1
5.1 Media interfaces

aRts supports streams between objects. Besides the typed (float) audio streams,
it also supports untyped byte streams. These streams need to be extended to support notions of:

  • which encoding does the stream have?
  • what is the size of the frames?
  • which streams (audio/video) belong together?

Addressing these issues and others would make a good step towards further modularizing audio/video
codecs. It would also allow things like connecting from a decoder to a
renderer without knowing which conversions need to be done to make them
the same format (a connection manager could find a suitable chain of filters
to plug in there). Also http and other forms of streaming could be supported.

Not in 2.1 however.

Dot Categories: 

Comments

by Anonymous (not verified)

Does this mean there will be a media player for KDE similar to Microsoft's Media Player or RealPlayer?

by L.D. (not verified)

It would be great to have a high performance Multimedia player in Linux! audio and video.

I remember when I would be playing an mp3 and I would get multiple icq messages during the song after the song was over all of the icq sound events would play one after the other, it was terrible, at least it doesn't do that anymore.

by Anonimous (not verified)

What about esound?

by Charles Samuels (not verified)

esound doesn't hold a candle for arts. Well, except for the performance part, but optimization is always one of the last step in a project. Esound is very minimalistic, so they it achieved that state rather quickly :)

by Eric Laffoon (not verified)

It would be nice if KDE 2 also had an mpeg player that worked as well as zzplayer!

As for the rest, I may have to get involved. I'm thinking of the A/D boards bringing in the mics and instruments, striping the MIDI, laying down the tracks, automixing and burning the CD. Look for it on mp3... Old guy scortches subwoofer with KDE on a new open sound license ;)

by tritone (not verified)

What about the great avilib available at euro.ru/~divx which uses wine code to run DivX and other codecs available under windows. Would there be any legal issues with this since there seems to be some reverse engineering involved?
Even Quicktime should be able to be incorporated this way. What I'd like to see is a player being able to handle all sorts of media and with a interface consistent with the KDE look. I have been thinking about writing something like this myself, I really don't care about skins and stuff.

by gis (not verified)

What about the great avilib available at euro.ru/~divx which uses wine code to run DivX and other codecs available under windows.

Install the full mpeglib
and kaiman and noatun will be able to play DivX!

by Julian Regel (not verified)

Fact: Multimedia is currently a major weakness for Linux (and most Unices).

Neither KDE or GNOME have made much progress on integrating real multimedia support into their desktop environments. Surely, this would be a good time to make use of the KDE-GNOME discussion list? A standard multimedia architecture would save an awful lot of duplicated work. It would mean that CODECs would only need to be written once.

i agree

by Charles Samuels (not verified)

The gnomes have been cooperating with Stefan Westerfeld. I'm not sure how much, but they will use aRts in some way or another. See our mailing lists.

With the gnome's help, arts will become better quickly

So you read the article and came to the same conclusion as the author ?

Cool.

The AC above read your post and gave a response worthy of AOL ?
Not cool.

by Moritz Moeller-... (not verified)

Comeon, how can you be taken serious as a player, if kaiman/artsd (or noatun) use 7% of the CPU power of my Athlon, while xmms does not even show in xosview (<1%)

This is one point I would like to see improved, the handling for the most common case: One sound file is being played.

Another thing I would like to see is a GUI plugin for xmms skins and a crossfade effect for song changes. Should be quite easy to do with artsd.

by Charles Samuels (not verified)

CPU optimization is a concern, we'll get there, but aRts is still maturing. Give us time :)

Winamp/xmms skins on the other hand, well, there is a person already writing a skin loader for noatun:

http://noatun.derkarl.org/shots/winamp.png

It's not released to the public because it's not complete enough (the Kjofol skin loader took at least a month before I was confident enough to put in the cvs)

by Christian Mueller (not verified)

I'd like to know whether the ogg / vorbis
effort (http://www.ogg.org) would fit in here, and how.

Regards,
cm.

by Charles Samuels (not verified)

OggVorbis support seems to be completely stable and working. All of us like OggVorbis very much in fact.

BTW, it doesn't work off the CVS, you'll need to install libvorbis and the "kmpg_vorbis" plugin for it to work. there's a readme in the source somewhere (for mpeglib_artsplug)

by Anonymous (not verified)

Does this mean there will be a media player for KDE similar to Microsoft's Media Player or RealPlayer?

by Anonymous (not verified)

Does this mean there will be a media player for KDE similar to Microsoft's Media Player or RealPlayer?

by Pavel Vondricka (not verified)

I have a small and stupid problem, that is probably not worth any work, but perhaps someone will have a simple and good idea :-).
I have MP3s with ID3 tags in different encodings (ISO8859-1 and 2). It would be nice to see something like unicode here as well, but 30 chars are not enough. Of corse, the new revisions of ID3 do not have such problems, but what with the old files? Will I have to re-burn my CD´s again with new ID3 tags in order to get correct characters in the future?

by Dietmar Schnabel (not verified)

What we really need is a esound compatibility for arts (or the other way around it, it does not matter for me)

There are many nice games or programs like xmms which have to fight with arts to support sound on kde2. But there is already esound which provides one standard. Either gnome or kde people have to take care to integrate both together. The first environment providing integration will have better chances to be the environment of choice for at least me in future.. I am currently switching between both environments and like them both very much!!