[Flac-dev] Proof-of-concept multithreaded FLAC encoder

[Brian Willoughby]

Frederick,

This is great news!  Thanks for your effort.

Your proof-of-concept raises a few questions for me:

1) I know that the ratio of uncompressed to compressed data is  
unpredictable, but I never really considered whether the input block  
size or the output block size is constant.  I'm assuming that if  
you're breaking the uncompressed input file into multiple pieces,  
then the uncompressed block size must be constant, while the output  
block size becomes variable.  I suppose the FLAC format allows for  
some variation in block size anyway.  My question (finally) is: Are  
there any anomalies at the points in the stream where you concatenate  
the compressed blocks from the multiple threads?  Do you end up with  
partial blocks in order to fit things together?

2) How hard would it be to enable md5 calculation?

3) Do you accelerate decode as well as encode?  I'm thinking that the  
variable block size would require each thread to scan its block for  
the start of a new block header, and also continue processing past  
the end of its region to grab any partial block skipped by the  
adjacent thread.  Scanning the entire file for block offsets would  
probably expend a significant amount of time, so it might be better  
for each thread to scan within its own region.  There's probably no  
way to weave the output together, though, until all threads have  
completed - at least the ones processing earlier parts of the time line.

I agree that it would be useful to have a parallel file- or block- 
based API.  It would be more effort to maintain, but there is a gain  
for the added work, now that multiple processors is more common, even  
on laptops.  The flac-mt command-line also seems like a good idea.

Brian Willoughby
Sound Consulting


On May 6, 2008, at 11:31, Frederick Akalin wrote:
> I managed to hack out a proof-of-concept multithreaded FLAC encoder
> based on the example libFLAC one.  It turned out to be fairly
> straightforward to get near-linear speedup; I can encode a 636 MB wave
> file in 6.8s with 8 threads on an 8-core 3.0 GHz Xeon vs. 31.4s with a
> single thread.
>
> Basically I mmap() the input file, divide up the mmap()ed region into
> nearly equal pieces, parcel them out to encode threads, and write out
> the output chunks (asynchronously) in order as soon as they're ready.
>
> I turned off md5 calculation in the example libFLAC encoder for ease
> of comparison and also verification as it interacts badly (crashes)
> with FLAC__stream_encoder_set_current_frame_number().  I also zero out
> the min/max frame size fields in the metadata, which should be the  
> only
> byte difference between the files the multithreaded encoder outputs  
> and
> the ones the example libFLAC one outputs (well, and md5).
>
> I think there are a number of misconceptions floating around.  I don't
> think the existing encoding APIs (stream_encoder.h) should be
> retrofitted to support multithreading, but maybe a separate file- or
> block-based API could be written that shares most of the code with the
> existing stream APIs.  I agree that the stream-based API should always
> be primary, but there's no reason that it couldn't co-exist with an
> alternate parallel-friendly API.  As an end-result I envision a
> utility similar to the flac command-line utility (flac-mt?) that
> supports multithreading, but with only a subset of the functionality
> of flac, according to feasability and demand.
>
> As a first step, a function that simply encodes a block of memory to
> an output buffer with a given frame number etc. would be helpful, as
> well a simple function to build and write out a streaminfo metadata
> block to memory.  Maybe also exposing FLAC's implementation of md5
> calculation in the form of a function that md5s a memory block, too.
> I might have time to hammer something out for review.  What do you
> guys think?