[flac-dev] Duplicate QLP coefficient restricting code

[Martijn van Beurden]

Hi all,

I was investigating the behaviour of the -p switch of flac. This 
switch should enable exhaustive search for the optimal qlp 
coefficient precision, but the resulting files are usually 0.5% 
larger then when not using the switch. I stumbled upon this code 
in stream_encoder

> if(encoder->protected_->do_qlp_coeff_prec_search) {
>     min_qlp_coeff_precision = FLAC__MIN_QLP_COEFF_PRECISION;
>     /* try to ensure a 32-bit datapath throughout for 
> 16bps(+1bps for side channel) or less */
>     if(subframe_bps <= 17) {
>         max_qlp_coeff_precision = flac_min(32 - subframe_bps - 
> lpc_order, FLAC__MAX_QLP_COEFF_PRECISION);
>         max_qlp_coeff_precision = 
> flac_max(max_qlp_coeff_precision, min_qlp_coeff_precision);
>     }
>     else
>         max_qlp_coeff_precision = FLAC__MAX_QLP_COEFF_PRECISION;
> }
> else {
>     min_qlp_coeff_precision = max_qlp_coeff_precision = 
> encoder->protected_->qlp_coeff_precision;
> }

However, a little further, I found similar (but slightly 
different)code

> /* try to keep qlp coeff precision such that only 32-bit math 
> is required for decode of <=16bps streams */
> if(subframe_bps <= 16) {
>     FLAC__ASSERT(order > 0);
>     FLAC__ASSERT(order <= FLAC__MAX_LPC_ORDER);
>     qlp_coeff_precision = flac_min(qlp_coeff_precision, 32 - 
> subframe_bps - FLAC__bitmath_ilog2(order));
> }

The difference here is the ilog2 function. I presume this is a 
mistake, because subtracting the order (32 - 17 - 8) leaves few 
possible precision steps to try. Because the first 'quote' of 
the code is always followed by the second, I attach a patch that 
makes the first match the second.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-bug-when-using-p-switch-during-compression.patch
Type: text/x-patch
Size: 0 bytes
Desc: not available
Url : http://lists.xiph.org/pipermail/flac-dev/attachments/20140728/4b1e500d/attachment.bin