[flac-dev] Disk fragmentation
cse at sci.fi
Tue Sep 23 08:42:16 PDT 2014
I have been running a lot of large compression tests to see how well
Martijn van Beurden's new presets do and was once again reminded how
real the fragmentation problem still is.
To have decent speed it's necessary to run multiple encoders in
parallel. In my setup FLAC was compressing four files at the same time
and each instance writes tiny bits of data to disk at once. A
fragmentation analyzing tool reported that my files were in 300
fragments on average. Thanks to this decoding the files happened in a
fraction of the speed that I should be getting.
There would be one very simple solution to help against this issue, the
use of setvbuf() command to buffer the file writes to larger chunks.
Unfortunately it looks like the call has to be done inside libflac. The
simple test change shown at the end of the mail reduced average
fragmentation to two fragments per file and decode speeds were as good
as without fragmentation.
I don't know how big issue this is in *nix environments but added
buffering would definitely be a nice change for the Windows frontend.
diff --git a/src/libFLAC/stream_encoder.c b/src/libFLAC/stream_encoder.c
index 6f46d78..bd95634 100644
@@ -1288,6 +1288,8 @@ static FLAC__StreamEncoderInitStatus
if(file == stdout)
file = get_binary_stdout_(); /* just to be safe */
+ setvbuf(file, NULL, _IOFBF, 10*1024*1024); /* 10MB output buffer to
help reduce disk fragmentation */
encoder->private_->file = file;
encoder->private_->progress_callback = progress_callback;
More information about the flac-dev