[Flac-dev] Proof-of-concept multithreaded FLAC encoder

[Brian Willoughby]

On May 6, 2008, at 14:20, Frederick Akalin wrote:
> On Tue, May 6, 2008 at 2:03 PM, Brian Willoughby  
> <brianw at sounds.wa.com> wrote:
>>  3) Do you accelerate decode as well as encode?  I'm thinking that  
>> the
>> variable block size would require each thread to scan its block  
>> for the
>> start of a new block header, and also continue processing past the  
>> end of
>> its region to grab any partial block skipped by the adjacent thread.
>> Scanning the entire file for block offsets would probably expend a
>> significant amount of time, so it might be better for each thread  
>> to scan
>> within its own region.  There's probably no way to weave the output
>> together, though, until all threads have completed - at least the  
>> ones
>> processing earlier parts of the time line.
>
> I looked into it briefly, but haven't had time to get it working.  The
> tricky part is dividing up the input file equally given the variable
> block size and the (potential) lack of a seektable block.  The best
> solution might just be to scan the output file and push (frame number,
> byte offset, byte size) onto a queue which the decoding threads would
> pop off, but that's a little more complicated.  I'm still mulling it
> over. :)

I may be biased because I work with hour-long recordings, but it  
seems like the block size would be small compared to the overall file  
size.  In other words, if you break the encoded file into equal-sized  
regions, then have each thread scan its region for the first block  
header, then it might be close enough to equal to work.  Then again,  
the output region sizes could vary greatly, especially if the input  
dynamics are vastly different from region to region.

I'm also assuming that scanning the entire compressed file for block  
offsets would take a while, but again that's probably because I work  
with files on the order of a gigabyte in size.  Obviously, more  
people work with "song" sized FLAC files, but you should at least  
consider what will happen with your algorithm when it is presented  
with a gigabyte of data.

>>  I agree that it would be useful to have a parallel file- or block- 
>> based
>> API.  It would be more effort to maintain, but there is a gain for  
>> the added
>> work, now that multiple processors is more common, even on  
>> laptops.  The
>> flac-mt command-line also seems like a good idea.
>
> Cool!  I'll start working on something then.

Hopefully Josh is also in agreement.

Brian Willoughby
Sound Consulting