KDE 3.0 Multimedia Meeting

Stefan Westerfeld
has posted
a summary of a IRC discussion held by the KDE Multimedia team
last month. Essentially a KDE 3 roadmap for the multimedia team, the
discussion covers topics ranging from MCOP and OSS compatability to recording
and video embedding. A slightly edited version of his post follows.

 

The KDE3.0 Multimedia Meeting, IRC discussion
summary

This summary provides an overview of the things that we talked about
during the KDE3.0 multimedia meeting which took place on IRC (2001-09-04).

We talked our way through more or less the topics that were posted before on
the list. I will summarize this document to a list list of keywords for
the KDE3 planned features document (which is relevant for the freeze), but
this is the more verbose source of what will be done.

  1. Making the MCOP component model more open to the world
  2. Compatibility with legacy/broken OSS drivers
  3. Enabling i18n
  4. Recording
  5. Video embedding / streaming
  6. PlayObjects (special point ;)
  7. MIDI
  8. GSL Engine
  9. GSL other stuff
  10. Samples
  11. Environments
  12. Decisions & Open Discussion
  13. Conclusion
  14. Meeting Information

Making the MCOP component model more open to the
world

  • MCOP<->DCOP bridging: the plan is to provide access to MCOP based
    services from DCOP - thats useful for scripting, as well as development,
    and desktop integration.
    • ==> responsible: Nikolas Zimmermann

  • JMCOP: There is a working but unreleased partial Java MCOP implementation,
    so if you want to work on something like this, don't start from scratch -
    however, there is currently no fixed completion date for this, nor will
    it probably be distributed bundled with KDE3.0.
  • UCOM: There was a bit discussion about this one, but nobody seemed to
    have looked too much into UCOM yet.

Back to top

Compatibility with legacy/broken OSS
drivers

  • Threaded OSS support: some applications acquire better results with
    respect to compatibility with various possible kinds of hardware than
    aRts, because of their threaded implementation of accessing the sound
    card. The threaded oss patch is a patch that opens the driver in blocking
    mode and spawns a seperate thread.
     
    While there was discussion whether the observed increase in quality (less
    dropouts, better support on more hardware) was caused
    • additional buffering gained through this (especially for cards with
      small hardware buffer)
    • brokenness of various OSS drivers for completely non-blocking I/O
    • the usage of select(...) which might be badly implemented for some
      OSS drivers or a combination of those, it generally was considered a good
      idea to do it
    •  
      ==> responsible: Matthias Welwarsky
      [ I wrote him here because he wrote the patch, but since he wasn't
      at the meeting, we might need to change this ]

Back to top

Enabling i18n

  • All strings emitted by aRts should be internationalizable in KDE3.0 - they
    are currently not, because aRts does neither use libkdecore (so it can't
    use the standard i18n), nor has own means to deal with that.
     
    While workarounds might be possible (such as artsd only emitting error
    codes, and seperate in-the-gui-layer-translation), the right solution,
    which was also somewhat discussed on the lists will be adding own i18n
    capabilities to libmcop. Whether or not libc i18n can be used, or whether
    the kdecore code could be copied, is still to be evaluated. As there are
    no unicode strings in aRts (no QString), utf8 is probably the way to go
    for transport to the GUI.
    • ==> responsible: Jeff Tranter
      [ Volunteered on the lists recently ]

Back to top

Recording

  • The aRts C API didn't implement clientside recording until now, but it
    should. Likewise should the Arts::SoundServer interface. It might be good
    finally following the KDE2 plan to provide a KAudioStream class in
    libartskde, to provide recording/playing streams in a nice, canned,
    easy-to-use way. Including an end user sound recorder into kdemultimedia
    might be useful, too.
    • ==> responsible: Matthias Kretz

Back to top

Video embedding / streaming

  • There is an VideoPlayObject interface in KDE2.2 already, which supports
    playing videos via X11 embedding. This is a quite simple solution, but
    using it more in KDE applications will make viewing videos much more
    user friendly than it is now.
  • There was some discussion about videos and noatun. Noatun currently doesn't
    use the embedding interface, and this might be hard to do, because every
    skin needs to be changed for that. On the other hand, noatun is the
    official KDE media player, so it should probably be able to embed videos.
  • There was some discussion about eventually implementing video editing, but
    we currently lack mainly the codecs to do it. The codecs don't support
    precise frame based seeking and delivery of the raw image as data. Besides
    this, video support would greatly benefit from introducing more structured
    asynchronous data delivery (with timestamps and frames) into aRts.
  • Extending Brahms towards embedding on the other hand seems less hard, so
    that it might be reasonable simple to compose film sound tracks with Brahms
    eventually, or - if we have better codecs - even edit films the same way
    Brahms can edit music.

Back to top

PlayObjects

  • Malte does have a RealMedia PlayObject ready (based on the Real library, so
    the implementation is not entierly open source). Currently, there are
    unsolved licensing issues.
    • ==> responsible: Malte Starostik
       

  • There was some discussions on other ways to get codecs. Basically, writing
    a PlayObject for one of
    • ffmpeg
    • libmpeg3
    • xine
    • gstreamer

    could extend the fileformats aRts will be able to play. Martin Vogt is
    working on xine, combined with code to standarize rendering and output
    more, to make integration of new video codecs, and later, video editing,
    simpler. There is experimental libmpeg3 code in the CVS.

Back to top

MIDI

  • Midi sequencers (Brahms/Rosegarden/Anthem) have started to use aRts
    as way of outputting midi, whereas other applications (kmid) still use
    libkmid, and some things (for instance ALSA) are not supported by aRts
    at this time. For KDE3.0, aRts should provide a complete solution.
  • The midi interfaces should finally support accessing sound card hardware of
    all fashions (ALSA, OSS-hw-synth) - that will mean a rewrite of some stuff,
    but is necessary for making the midi interfaces reasonable complete.
  • Precise synchronization between midi/audio should be adressed.
  • Libkmid could finally be moved upon the new midi interfaces.
  • The midi libs should be mainstream (i.e. kdelibs) and stable after KDE3.
  • A midi PlayObject would be helpful, to enable media players like noatun
    to deal with midis.
    • ==> responsible (all midi issues): Antonio Larrosa, Stefan Westerfeld

Back to top

GSL Engine

  • GSL will be the new engine for scheduling synthesis modules. This will
    not cause a glib dependancy.

      ==> responsible: Tim Janik, Stefan Westerfeld

  •  

  • more lengthy explanation about what/why/how --

    aRts currently has an engine that decides which modules are to be
    calculated when, given a certain flow graph - this is called scheduling
    (and implemented in kdelibs/arts/flow/synthschedule.cc)
     
    The GSL engine is a complete replacement of the algorithm with the
    following characteristics:

    • it should be quite a bit more efficient in the common case
    • it supports dispatching calculations in multiple threads, which for
      instance allows to fully exploit multi processor systems
    • it allows putting module calculations in a seperate thread from the
      aRts main thread, thereby avoiding dropouts from lengthy aRts operations,
      such as for instance calling open() on an NFS drive
    • it is designed in a way to support advanced scheduling features that
      aRts currently does not support, that is
      • feedback loops (circles)
      • sample precise timing
    • it should allow achieving very very low latencies reliable (where very
      very low is something like 3ms) - aRts currently can't do this
    • it is implemented in plain C, using some glibisms, however it can be
      integrated into aRts with a glib-compat source in a way that it does
      not bring additional dependancies (no glib required)

    The GSL engine should first act as a drop-in replacement - however to take
    advantage of its capabilities, it will be necessary to make sure that the
    modules are implemented in a way that they can run in a seperate thread
    without interfering with anything else.
     
    That means, that there will two types of modules: the old modules (which
    will still be running in the aRts main thread), and the hard-rt-modules
    (which will be able to run in the engine threaded).
     
    You will get the very very low latency benefit only if you use only hard-
    rt-modules, but everything should work with any mix of modules.
     
    Where is GSL from?
     
    GSL is designed by Tim Janik and me, and contains a lot of know how about
    music apps (Tim Janik has been working for years on Beast - best.gtk.org,
    and I have been working for years on aRts).
     
    GSL code is in the CVS and working (although yet somewhat slow) already.

  • Back to top

    GSL other stuff

    • GSL provides a sample cache which supports partial caching and preloading,
      the integration of that will get rid of the long outstanding issue that
      aRts always needs to load whole WAV files (or other formats), and support
      GigaSampler like "read-from-HD-and-play" style sample sets of huge size.
    • GSL provides filter design code for butterworth, tschebyscheff-1/2
      and soon elliptic low/high and soon bandpass/stop filters, which can be
      wrapped in new aRts modules (aka Arts::Synth_HIGHPASS).

    Back to top

    Samples

    • While aRts is good at creating synthetic sounds and using these as
      instruments in a sequencer (i.e. Brahms), playing samples will greatly
      enhance the possibilities.
    • I have a prototype of a .PAT loader - you can import a whole timidity
      style patchset with that, which means that we have a useful GM-sample-set
      for aRts. There is also AKAI loading code in kmusic/ksynth. GSL provides
      another native format. All of these probably should use the GSL sample
      cache for efficient, preloaded, replaying.
    • Combined with an aRts PlayObject, media players (noatun, kaboodle) gain
      not only the capability to do similar replay than kmid (output midi via
      card synth or external midi), but also kmidi (output midi as calculated
      audio data from sample sets).
      • ==> responsible (all points): Antonio Larrosa, Stefan Westerfeld

    Back to top

    Environments

    • The work started with mixers/environments should be completed - for instanceit would be very useful if the mixer could save/restore itself properly, and
      if it could also contain insert-stereoeffects and a master volume. Also,
      the mixer look more like a real-mixer(tm), for instance groups or panning
      is not there right now.

        ==> responsible: Matthias Kretz

    •  

  • Visualizations (led-bars) would be nice, but currently have the technical
    problem that they overload the MCOP layer easily, this can probably only
    be fixed by changing MCOP to deal more intelligently with change
    notifications.

      ==> responsible: Matthias Kretz

  • Back to top

    Decisions, Open Discussion

    • aRts will break binary compatibility (KDE does so, so we should, too).
    • the general consensus seems to be that moving aRts in an extra CVS module
      (so that it is easier to obtain, compile, develop, use, package without
      KDE) seems to be a good thing do to
    • whether to merge the SoundServerV2 interface into SoundServer (and so on),
      is still an open issue:
      • advantages: simpler code, cleaner interfaces
      • disadvantages: wire compatibility (running KDE2 and KDE3 apps on artsd)
        will break - or is at least very hard to keep

    Back to top

    Conclusion

    There are about three major areas which the work focuses on. First of all,
    it's important to continue to improve everyday usage for casual users. Mostly,
    recording, i18n and the threaded OSS support fall in that category.

    On the other hand, a lot of things should be simply unified, made more
    consistent, and merged. A media player which can play everything might sound
    a lot better to the user than noatun, kaboodle, kmid, kmidi and aktion.

    Finally, there is a lot of work to be done especially for music applications,
    which includes the gsl engine, more support for samples, midi improvements,
    and environments/mixers.

    I hope that with the way we split up the work/responsibility on several
    developers, we'll be able to achieve the goals listed here in time for
    KDE3.0.

    Back to top

    Meeting Information

    The original IRC log can be found
    here.
    Meeting participants (in no particular order) were:

    • Carsten Niehaus
    • Stefan Schimanski
    • Andreas Burman
    • Robert Gogolok
    • Achim Bohnet
    • Michael Seiwert
    • Stefan Westerfeld
    • Matthias Kretz
    • Nikolas Zimmermann
    • Neil Stevens
    • Tim Janik
    • Jan Wuerthner
    • Daniel Haas
    • Malte Starostik
    • Martin Vogt
    • (... there was a lot of joining and leaving on the channel, so maybe
      I forgot somebody ...)

    Cu... Stefan
    --

    -* Stefan Westerfeld,
    [email protected] (PGP!),
    Hamburg/Germany
    KDE Developer, project infos at http://space.twc.de/~stefan/kde *-

    Dot Categories: 

    Comments

    I have two Slackware 10.1 systems, one with 2.6.15.4, one with 2.6.15.5. Both use KDE 3.5.1.

    artsd works fine for root user on both systems. For ordinary users, artsd crashes.

    WTF?

    I guess I'm the only person that doesnt like arts out there. I have tried and tried with it, but no matter what I do, it always ends up crashing a few minutes after I get it going. sometimes it makes it through a few songs, but then, right as i begin to think, maybe it's working, it craps out on me again. i use slackware linux and have tried using the packages that slackware-current has out, and compiling everything myself. even when i tried using artsd on a different machine running red crap 7.1 it still crashed on me after a few minutes. i think the best thing KDE could do for the entire project would be to switch to esd, which has never ONCE failed on me on any computer for any reason.

    just my $.02

    I've had this problem before. I have a Sis motherboard with some kind of Trident/Sis audio chipset builtin. At first I used the standard OSS kernel sound modules and esd wouldnt break up playing mp3s but artsd would when I used noatun. It would fuzz out after 5 seconds, which was annoying so I disabled artsd. A couple of weeks ago I switched to the Alsa sound drivers and now I dont get anymore sound breakups with artsd. So definetly try the Alsa sound drivers if you want artsd to run with KDE...

    by Clinton Ebadi (... (not verified)

    Um, last I checked, GNOME 2.0 was going to use aRts as well. ESD _really_ sucks. I can't get it to run anymore. I know several other people who can't get it to run as well. It hasn't been updated in a year or two either. It is _dead_. As dead as dead can be. I suggest that you file a bug report for aRts in order to get help fixing your problem. Don't sit back and complain; do something about it. Its not that hard to file a bug report you know? Except when the package broke for a while (good old Debian unstable...), aRts has worked fine for me (of course, I use WindowMaker and only run aRts for apps like brahms).

    I had that kind of problem before. Then I turned off realtime priority in the kontrol center. It has worked flawless for many months since then. The problem is that the crappy sound driver in the kernel can't handle realtime priority.

    (Yes, I do know about ALSA. I will try it again when it comes with Linux.)

    by Alain (not verified)

    I am very glad that midi will be enhanced, thanks !

    With my old sound card Kmid didn't work, kmidi was good, but I didn't reached to replace Kmid by Kmidi in Konqueror. With my new sound card, Kmid works, but without sound (!!), and I don't still try Kmidi. And Kmid crashes Konqueror when I klick on it... I even don't see a button to stop music... Please be care with the integration in Konqueror...

    I also hope that Kmid will play karaoke as well as karawin, with little balls showing the speech coming (see www.karawin.com)... for KDE 3.1 !

    by Terkum (not verified)

    What about making arts compatible to _native_ alsa. I do not want OSS compatibility when using alsa.

    by Stefan Westerfeld (not verified)

    There is _native_ alsa support in aRts (artsd -a alsa or kcontrol -> sound -> soundserver -> sound i/o -> method -> alsa). Right now, it works for alsa-0.5.0 only, but Takashi Iwai is looking into porting it to alsa-0.9.0. If you just can't wait for it, feel free to port it yourself and send me a patch.

    by qwerty (not verified)

    Nothing on Linux comes close to Cubase at the moment, even though I feel Cubase is flawed in its design. This is a pity as it means I still use Windows for midi.

    However, all this stuff sounds brilliant. What KDE needs is a robust, well designed framework of re-usable components for audio/video/midi which can be combined easily.

    by loopkin (not verified)

    can't aRts provide an interface to esd, so that all the apps out here written to use esd can work with aRts ? (nearly all of the commercial apps written for Linux do so...)
    all-in-one media player is a good idea though, but u'd rather not use noatun, which is crappy, heavy and unstable piece of software (especially compared to xmms), and has odd features, such as the systray icon, which is not really needed....

    by David Findlay (not verified)

    noatun is good, but on video I've noted it to be much more jumpy on video than mplayer or avifile. Could we have a better name for it, as well as a version that plays all file formats, and is fast?

    by ART (not verified)

    What about computer telephony devices?

    by Jean-Michel POURE (not verified)

    Dear all,

    This are really great features ahead.

    Do you also plan to integrate H323 libraries to offer embedded video conference support. There is also a need for a better support of instant messaging tools (Jabber, http://sourceforge.net/projects/konverse/).

    Today, I use GnomeMeeting which is a mature product for video-conference. Do you plan something like in the future?

    Best regards,
    Jean-Michel POURE

    by mohamd (not verified)

    hi
    i am mohamd and don't family with h323 i want create a video conference with h323 for windows please gudie me

    by -=MaGGuS=- (not verified)

    Check out the open source libraries for H.323 protocol called OpenH323 here:
    http://www.openh323.org/
    latest versions could be found here:
    https://sourceforge.net/projects/openh323/
    And MyPhone - my Windows H.323 client, based on these libraries here:
    https://sourceforge.net/projects/myphone/

    -=MaGGuS=-

    by Laura (not verified)

    Hi,

    I've the same problem that you had some months ago.I would like to use H323, only for voice, but I don't know how to beguin. I have the code but I don't know how to execute and use it. Can you help me, please?