[Flac] format spec unclear on how to quantize the LPC prediction

[Sebastian Gesemann]

Hello!

I just noticed how the LPC prediction is quantized by browsing the
source code of the java port jFlac:

	:
	int sum = 0;
	for (int j = 0; j < order; j++)
		sum += qlpCoeff[j] * data[startAt+i-j-1];
	data[startAt + i] = residual[i] + (sum >> lpQuantization);
	:

I checked the C version (libFLAC) and it matches. However, it was a
surprise because this way of quantizing the prediction introduces a bias
towards negative infinity which results in residual samples that have a
bias towards positive infinity and therefore a shifted probability
distribution: residual of +1 is as common as 0 and more common than -1
etc...

This is not too bad I guess. But it CAN be accounted for -- for example
via an alternate entropy coding which uses a mapping from all possible
residual values to non-negative numbers by this interleaving:

	0	-> 0
	+1	-> 1
	-1	-> 2
	+2	-> 3
	-2	-> 4
	+3	-> 5
	etc

This way the symbols 0, 1, 2, ... have non-increasing probabilities.

Anyhow, this quantization issue is not mentioned in the format
specification. Naturally, I would have assumed to quantize by properly
rounding like this:

	data[startAt + i] = residual[i] + ((sum + a) >> lpQuantization);
where
	a = (1 << lpQuantization) / 2;

but I guess now it's too late to change that behaviour of the FIR predictor.


Cheers!
SG

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://lists.xiph.org/pipermail/flac/attachments/20071014/ad270058/signature.pgp