[flac-dev] 64-bit residuals

[Martijn van Beurden]

Op do 24 mrt. 2022 om 18:01 schreef Brian Willoughby <brianw at audiobanshee.com>:
>
> Can you briefly explain the consequences of this unexpected residual?
>

Of course. The WAV file I've used now does not cause problems when
compressed with any release version of libFLAC as far as I can tell.
For now, this is more of a decoder-side problem: encoders other than
libFLAC or a future libFLAC might create such residuals, as it is not
limited in the spec.

The problem files in the github issue have been created with an
experimental version of libFLAC with some compression improvements
I've been working on the past two years. This experimental encoder
implements a so called iteratively reweighted least squares solver
(IRLS) to find a least absolute deviation (LAD) solution to the
problem of finding an optimal predictor, whereas current libFLAC uses
an approach that more or less resembles a least squares solution. When
using a LAD solution, the average residual is smaller at the cost of
the maximal residual being much larger. Therefore I do not expect this
to happen with released versions of libFLAC, not even with crafted
material.

The _ultrawide functions would be used when the formula subframe_bps +
qlp_coeff_precision + FLAC__bitmath_ilog2(order) - quantization_level
returns a value larger than 32. This will never happen on 16-bit
files, not often on 24-bit files and quite often on 32-bit files, if
encoding and decoding 32-bit files is ever implemented.

If at some point this experimental branch is merged into a release
version, the test suite will need to be expanded to cover this.