[vorbis-dev] Parallelism

[Jeff Squyres]

On Fri, 18 Aug 2000, Timothy Wayper wrote:

> [rationale snipped]
> 
> However is forcing every implementation to use MPI a good idea? MPI is
> great for a cluster, or when the nodes are running on different
> CPU's/operating systems, but it requires complex end-user
> configuration. My feeling is that it is a bit heavy-weight for most
> people; surely the most common case of parallelism is multiple CPU's
> within the same machine, which can be handled with far less overhead
> using "native" threads and synchronization primitives? Maybe there
> should be a very thin abstraction layer to allow both MPI and "native"
> implementations? Maybe my impression of MPI is incorrect though,
> please correct me if it's easier to use than I think!

You're probably right.

I think that if we can agree that parallelism is a Good Thing, and
possibly get a proof-of-concept showing that it at least *could* work,
then we can discuss exactly *how* to do it.

There is also nothing that says that you can't mix threads and MPI -- use
threads for on-machine parallelism, and MPI for off-machine parallelism
(or perhaps make it user configurable -- you may not always want to use
all the CPUs on an SMP for your encoding.  For example, if you have a
4-way SMP, you may want to use 3 for your encoding, but leave one
available for your "real" work).

configure can look for pthreads/whatever as well as look for MPI.  
Finding either/both will enable the VORBIS_HAVE_PARALLEL #define to 1 (and
VORBIS_HAVE_THREADS and/or VORBIS_HAVE_MPI, as appropriate) which can turn
on/off chunks of code.  Hence, if users don't have MPI, they can still get
the benefit from threads.  And if users don't have threads (do any modern
OS's not have threads these days?), they can still use MPI.  And if they
have neither, they get a stock serial version of the encoder.  
Justification through righteous shell scripting.  ;-)

There's issues here, of course (many MPIs -- including mine -- are not
thread safe), but probably not unsolvable.

{+} Jeff Squyres
{+} squyres at cse.nd.edu
{+} Perpetual Obsessive Notre Dame Student Craving Utter Madness
{+} "I came to ND for 4 years and ended up staying for a decade"

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/