[flac-dev] 64-bit residuals

[Martijn van Beurden]

Op do 31 mrt. 2022 om 09:23 schreef Miroslav Lichvar <mlichvar at redhat.com>:
> Can you post some examples of the impact on compression ratio? Do the
> samples actually use all of the 32 bits? I have no experince with
> this.

I did some tests with ffmpeg with the patch I linked earlier. This
patch limits the residuals to "31-bit ints" and the LPC predictions to
32-bit ints. I varied the residual limit between "30-bit int", "31-bit
int" and 32-bit int. Limiting to "31-bit int" is needed for backward
compatibility with ffmpeg. I compressed a test corpus with this
patched ffmpeg, and measured file size of the compressed corpus and
the total number of verbatim frames, which indicates how often the
encoder uses verbatim frames as fall-back in case it cannot find a
predictor with residuals that fall within the stated limits.

Test corpus was created from 16-bit files upscaled by SoX with the
hilbert filter to fill the added 16-bit. Corpus consists of 20 tracks
of mostly music encompassing a wide range of genres, from solo piano
through heavy metal, and also including two non-music recordings.

set of 32-bit WAV: 1749MB
32-bit FLAC with 32-bit residuals, no stereo decorrelation: 1215MB
(2.4% verbatim)
32-bit FLAC with 31-bit residuals, no stereo decorrelation: 1219MB
(4.5% verbatim)
32-bit FLAC with 30-bit residuals, no stereo decorrelation: 1234MB
(27% verbatim)

To assess the influence of limiting the LPC predictions to 32-bit ints
and limiting stereo decorrelation, I also tested with 31 bit per
sample files.

31-bit FLAC with 32-bit residuals, no stereo correlation: 1160MB (0.0% verbatim)
31-bit FLAC with 32-bit residuals: 1141MB (0.0% verbatim)
31-bit FLAC with 31-bit residuals, no stereo decorrelation: 1161MB
(0.6% verbatim)
31-bit FLAC with 31-bit residuals: 1141MB (0.3% verbatim)
31-bit FLAC with 30-bit residuals, no stereo decorrelation: 1165MB
(6.2% verbatim)
31-bit FLAC with 30-bit residuals: 1144MB (3.7% verbatim)

>From this data I think I can conclude the following:
- Limiting residuals to "31-bit ints" hampers compression by 0.25%.
>From the results with 31-bit FLAC files it seems capping the residual
at 32-bit ints does negligible harm to compression, but capping them
to 31-bit does harm it a bit. So, 0.25%-point is calculated as
(1219-1215)/1749
- Limiting predictions to 32-bit ints hampers compression by another
0.25%-point. Comparing the results of 31-bit FLAC with 32-bit FLAC and
combining that with the previous conclusion, it seems the 2.4%
verbatim frames in 32-bit FLAC with 32-bit residuals seem to come
mostly from this limit.
- Limiting stereo decorrelation seems to hamper compression by
1.1%-point. This is calculated from the difference between the 31-bit
FLACs with and without stereo decorrelation.

So, from this limited performance testing, it seems imposing these
limits causes compression to fall by 1.6%-point. It is a little more
than I expected, but performance is still quite reasonable.

Kind regards,

Martijn van Beurden