[flac-dev] Disk fragmentation

[Janne Hyvärinen]

I have been running a lot of large compression tests to see how well 
Martijn van Beurden's new presets do and was once again reminded how 
real the fragmentation problem still is.

To have decent speed it's necessary to run multiple encoders in 
parallel. In my setup FLAC was compressing four files at the same time 
and each instance writes tiny bits of data to disk at once. A 
fragmentation analyzing tool reported that my files were in 300 
fragments on average. Thanks to this decoding the files happened in a 
fraction of the speed that I should be getting.

There would be one very simple solution to help against this issue, the 
use of setvbuf() command to buffer the file writes to larger chunks.
Unfortunately it looks like the call has to be done inside libflac. The 
simple test change shown at the end of the mail reduced average 
fragmentation to two fragments per file and decode speeds were as good 
as without fragmentation.

I don't know how big issue this is in *nix environments but added 
buffering would definitely be a nice change for the Windows frontend.


diff --git a/src/libFLAC/stream_encoder.c b/src/libFLAC/stream_encoder.c
index 6f46d78..bd95634 100644
--- a/src/libFLAC/stream_encoder.c
+++ b/src/libFLAC/stream_encoder.c
@@ -1288,6 +1288,8 @@ static FLAC__StreamEncoderInitStatus 
init_FILE_internal_(
      if(file == stdout)
          file = get_binary_stdout_(); /* just to be safe */

+    setvbuf(file, NULL, _IOFBF, 10*1024*1024); /* 10MB output buffer to 
help reduce disk fragmentation */
+
      encoder->private_->file = file;

      encoder->private_->progress_callback = progress_callback;