[vorbis-dev] Parallelism

[Jeff Squyres]

On Wed, 16 Aug 2000, Segher Boessenkool wrote:

> What would be the advantages of a parallel encoder? If you need the
> extra speed, you're probably encoding lots of tracks, so just start a
> bunch of encoders at once? The present vorbis example encoder (not
> optimized) is better than realtime, so that's no problem either.

Speed is the main purpose.

However, you can do better than just starting a bunch of encoders at
once.  Starting a bunch at once:

- is inconvenient for the user to do
- requires many windows (potentially on different machines), or clever use
  of & 
- may require distribution and collection of the input/output data (unless
  you have a global filesystem [e.g., NFS], but the overhead caused by
  that can be detrimental to overall performance)
- can be sub-optimal; you are creating artificial boundaries on the tracks

Granted, this could all be automated with shell scripts, but most users
won't know how to do this.

The real benefit of parallelism is encoding a whole shload of tracks at
once -- you can farm out data to threads/nodes as those threads/nodes
become available (e.g., a master/slave kind of paradigm).  For example --
encode some tracks on 8 nodes.  When the input data for the first track
runs out and a node is asking for more, give it the first set of data from
the next track (even though the other 7 are still working on the last
track).  And so on.

The point of this scheme is that all the nodes (I'm using the term "node"
loosely here -- "node" == CPU) stay busy as close to 100% of the time as
possible.  Starting a bunch of encoders, each on one or more tracks, could
be less efficient (particularly in a cluster of workstations that are
getting used for something else -- heavily loaded/slower machines will be
able to do less work than lightly loaded/faster machines).  

It could certainly be convenient to do something like:

        ogg encode -parallel *.wav

(assuming that the output filenames would be automatically derived from
the input filenames)

But there is the valid point that the encoder is already fast.  If it's
already fast enough, I would agree that parallelism is not necessary.  
This is just my $0.02.  :-)

{+} Jeff Squyres
{+} squyres at cse.nd.edu
{+} Perpetual Obsessive Notre Dame Student Craving Utter Madness
{+} "I came to ND for 4 years and ended up staying for a decade"

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/