[Flac-dev] Proof-of-concept multithreaded FLAC encoder (take 2)

[Frederick Akalin]

Hey again FLAC devs,

I managed to hack out another proof-of-concept multithreaded FLAC
encoder that is more amenable to streaming and also uses a fixed
buffer size.  The performance is pretty much the same as my earlier
version; that is, I can encode a 636 MB wave file in ~7s with 8
threads on an 8-core 2.8 GHz Xeon (I erroneously stated 3.0 GHz in my
last e-mail) vs ~32s with 1 thread.  I had mentioned near-linear
speedup in my last e-mail and to clarify that, it looks like 2x the
threads gives a ~1.7x time reduction (up to processor count).  I'm
actually suprised that this version isn't faster than the first one;
even turning off outputting to a file didn't help performance much,
either.

In this version, I create an encoder just write out the metadata.
Then I mmap() the input file and spawn encoding threads.  Each
encoding thread will look up the value of a shared sample counter,
increment it by some multiple of the blocksize (all atomically),
encode that many blocks, read a shared (output sample counter, output
offset) pair until the output sample is in its range, and then write
to the shared output file handle, updating the pair atomically.  Also,
I had to hack up libFLAC a bit more:

- I wrote a version of the stream encoder finish function that does
  not reset the private variables.  This actually isn't necessary for
  correctness but re-initializing the encoder for each frame hurts
  performance quite a bit.

I used pthread locking primitives for synchronization but I wrote it
so that it can easily be replaced by non-blocking constructs (which is
beyond the scope of this version) -- an earlier version used
non-blocking constructs but moving to pthread locks didn't seem to
affect performance so the bottleneck isn't there.

Updated patch file: http://www.akalin.cx/patch-libFLAC.2.in
Source file for this version: http://www.akalin.cx/mt_encode2.c

This should also compile cleanly with gcc 4.x with -Wall -Werror -g
-O2 -ansi.

Comments and thoughts on this version are welcome.

-- 
Frederick Akalin
http://www.akalin.cx