[flac-dev] Possible overflow of _candidate_bits in stream_encoder.c

[Martijn van Beurden]

Recently I was trying some new approaches to improve FLAC compression, when
I stumbled on a possible overflow. The reason this has not come up earlier
is because the encoder only hits this point when the estimate of the
rice_parameter is very much off.

To trigger this overflow, one has to force rice_parameter to 0 in for
example the function evaluate_lpc_subframe in libFLAC/stream_encoder.c.
When encoding noisy material, which needs a high rice parameter, it can
happen that the return value of that function overflows. When this happens,
the wrong order might be picked, and the file blows up to enormous
proportions. In my case, about 30 times the size of the original WAV file.

When analysing this problem, I found more or less the same overflow in
flac/analyse.c, where in flac__analyse_frame, frame_bytes is multiplied by
8.

Should I send a patch to change all affected uint32_t to uint64_t? Or is
this benign enough not to matter? As far as I can tell, such a patch should
only touch private functions, no public ones.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/flac-dev/attachments/20200702/5b21514f/attachment.html>