New Plans for KDE Multimedia

Stefan Westerfeld has posted a first draft of cool new plans for the new multimedia architecture for KDE2, based on aRts (the "analog realtime synthesizer"). Plans include merging the two existing media players (noatun and kaiman), new media types, infrastructure improvements, improved MIDI and more. The full draft (edited by our own Dre) is attached below.

KDE2.1 Multimedia Plan (draft)

KDE 2.0 was quite a step for multimedia on the UNIX® desktop. aRts, which has been integrated into KDE 2 over the past year, provides an always running soundserver and service-type media-playing. The notification system, backends and the media player were completely rewritten. As a result users have a more powerful platform for multimedia tasks.

Still, there remains a lot to be done. This document tries to provide a planning and discussion base for coordinating "the way to go". The focus lies on what can be achieved for KDE 2.1.

Discussion by the community is highly welcome. Also, please consider contributing to further development.

This draft has spontaneously been designed on IRC, by

• Charles Samuels
• Stefan Schimanski
• Stefan Westerfeld

Overview

1. The KDE Media Players
2. New media types
3. Organization & Documentation
4. Midi/Sequencer
5. Not in 2.1

1. The KDE Media Players

1.1 Merging noatun and kaiman

Kaiman and noatun are two seperate media players available for KDE2.0. Kaiman was shipped with 2.0, and noatun is available via CVS. The plan for 2.1 is to combine these players into one media player, the official "KDE Media Player".

Technically, the KDE Media Player will be based on noatun's plugin architecture. Kaiman would be just one player making use of the noatun technology.

1.2 Pluggable effects

aRts allows much more than vanilla playback: filters can be used to affect the sound. Currently, this feature has been mostly unavailable in media player(s), mostly for two reasons:

1. there were few effects; and
2. the binding of a GUI to effects was not yet implemented.

The fix for (1) seems obvious: write more effects. One starting point could be porting the FreeVerb code to the aRts architecture.

The fix for (2) is less obvious. The problem is forumlating a method for GUI objects (which run in the player process) to connect to the effects (which run in the sound server process). The clean way to do this is to extend MCOP (the inter-process communication architecture used by aRts) to provide a signals & slots technology.

An example best illustrates this point. Say for instance there is a GUISlider aRts object, which runs in the player process space, and a ExtraStereo effects plugin, which runs in the aRtsd process space. The idea is to be able to write:

connect(slider,"value_changed",effect,"depth");

and thereby cause a connection between the slider and the effect. Conversely, the same mechanism can also be used to connect output GUI elements to effects.

Besides replacing the need to poll, this approach also allows the GUI to have a flexible deisgn, using the same modular approach that artsbuilder currently provides for sound.

1.3 Moving the tag reading code

The ID3 tag reading code, which provides information such as title and author from .mp3 files, is currently a noatun plugin. It should be made an aRts components to make it available to all applications using aRts.

2. New media types

2.1 mikmod

Work to make an aRts PlayObject for mikmod files is currently progressing. This would allow the KDE Media Player to play .xm, .s3m, .mod and other digital audio codecs. The clear advantage is that the component would be available to all other apps which know how to talk to aRts.

2.2 MIDI via timidity

Another great project would be to make a timidity plugin for aRts. Timidity is a package which renders MIDI notes to audio output using sampling. Currently kmidi is the KDE MIDI player. Converting kmidi to an aRts PlayObject would again provide the user with a much more consistent experience and provide the playback capabilities to all applications aware of aRts.

3. Organization & Documentation

3.1 KDE multimedia webpage

KDE multimedia development needs to be more visible. There is no official coordination point where people can see the development process and design overviews. To address this shortcoming, a KDE multimedia webpage (how does http://multimedia.kde.org sound?) should be created.

Charles has volunteered to take care of this.

3.2 aRts documentation

Fortunately the multimedia chapter of the KDE 2.0 Development book should become available to the public RSN. While this is a significant piece of good developer documentation and a great start, more work needs to go into a consistent documentation on http://www.arts-project.org, for both developers and users.

3.3 kdelibs/arts -> arts

Packaging aRts seperately is probably a good idea. aRts itself by design does not depend on Qt or KDE. People outside the KDE world should still be able to use aRts as a sound server or for music tasks.

If - for instance - GNOME would start using aRts, having aRts mixed in the CVS and in packaging with KDE is probably a bad idea.

4. Midi/Sequencer

4.1 Usability issues

aRts in the CVS already provides MIDI realtime synthesis. You can use midisend to play instruments in artsbuilder. You can also combine it with Brahms or artstracker to compose songs. The problem is that if you are not a rocket scientist or don't study the code or collected READMEs for a while, you will probably not manage to do it. That has to change.

A good start would be providing an instrument manager, where you can graphically assign which MIDI channel should get which instruments, without bothering with the details.

4.2 Interoperability

There are at least three sequencer-like programs which actively want to talk to aRts:

• Brahms, which has been in the CVS for some time now. Jan, it's author, is working on a new version right now - some aRts to MIDI support is already in place.
• Anthem is in its infancy. Its author Pete definitely wants to have audio stuff like hd recording, and probably aRts to MIDI support should be added as well.
• ArtsTracker is an experimental small tracker in kmusic.

Especially the areas

• connecting to aRts
• finding (virtual) MIDI channels and synthesis instruments
• assigning instruments to channels
• sending timestamped events

need to be standarized more for MIDI. Other interesting endeavors include:

• harddisk recording; and
• effect processing

Most of this only needs a small amount of code in aRts and the applications, the problem is finding a sophisticated standard.

5. Not in 2.1

5.1 Media interfaces

aRts supports streams between objects. Besides the typed (float) audio streams, it also supports untyped byte streams. These streams need to be extended to support notions of:

• which encoding does the stream have?
• what is the size of the frames?
• which streams (audio/video) belong together?

Addressing these issues and others would make a good step towards further modularizing audio/video codecs. It would also allow things like connecting from a decoder to a renderer without knowing which conversions need to be done to make them the same format (a connection manager could find a suitable chain of filters to plug in there). Also http and other forms of streaming could be supported.

Not in 2.1 however.