[flac-dev] exhaustive-model-search issue results in multi-gigabyte FLAC file

[Miroslav Lichvar]

On Sun, Jul 21, 2013 at 09:19:46PM +1000, Erik de Castro Lopo wrote:
> Miroslav Lichvar wrote:
> > >     https://git.xiph.org/?p=flac.git;a=commit;h=6f7ec60c7e7f05f5ab0b1cf6b7b0945e44afcd4b
> > 
> > I don't like this fix. It will probably hurt performance with 16-bit
> > data and it won't fix the problem in the assembly code.

> I have committed an improvement on the above fix.
> 
>     https://git.xiph.org/?p=flac.git;a=commit;h=f34f31dac0032887887b5bbcb0944de055b757d0
> 
> that reverts to the use of a FLAC_uint32 accumulator for files of less
> than 24 bits per sample.
> 
> I still have no proof that this overflow cannot occur for 16 bit files.

I think it could happen with 16-bit files too if a very long block is
used.

My understanding of the problem is that after the prediction there are
samples in the residual signal which are wider than the original
samples, i.e. the bps argument in precompute_partition_info_sums_ is
wrong. I think such results should be ignored and never allowed to be
encoded to prevent problems in the decoder.

At the moment I don't see how it can be done efficiently. I'll try to
look into it more later. As a temporary workaround for the encoder I'd
suggest to use something like this instead of the bps < 24 check. I'm
not sure if 4 extra bits is enough to cover all possibilities.

--- a/src/libFLAC/stream_encoder.c
+++ b/src/libFLAC/stream_encoder.c
@@ -3515,7 +3515,7 @@ unsigned evaluate_fixed_subframe_(
                        rice_parameter_limit,
                        min_partition_order,
                        max_partition_order,
-                       subframe_bps,
+                       subframe_bps + 4,
                        do_escape_coding,
                        rice_parameter_search_dist,
                        &subframe->data.fixed.entropy_coding_method
@@ -3598,7 +3598,7 @@ unsigned evaluate_lpc_subframe_(
                        rice_parameter_limit,
                        min_partition_order,
                        max_partition_order,
-                       subframe_bps,
+                       subframe_bps + 4,
                        do_escape_coding,
                        rice_parameter_search_dist,
                        &subframe->data.lpc.entropy_coding_method


-- 
Miroslav Lichvar