[Vorbis-dev] float->int rounding problem in setup_tone_curves()

[Martin Dufour]

Hi,

We have severe audio quality issues when we compile Vorbis with
Microsoft VC 2005, while it works fine with VC 2003. I have isolated the
problem to a rounding issue in setup_tone_curves() ; the "lo_curve" and
"hi_curve" are different depending on the compiler. For example, here
are the values for the last band (#17) at 32 kHz prior to { ceil(),
floor() }:

VC2003: lo_fcurve=15.997361 hi_fcurve=16.000000 
VC2005: lo_fcurve:15.997360 hi_fcurve:15.999999

See how the floor() for hi_curve is problematic, as it will resolve to
15 or 16 depending on the compiler. This makes me think that perhaps a
round-to-nearest would be more appropriate (and indeed, seems to make
sense for other bands as well).

Why is hi_curve floored?

Thanks,
Martin Dufour
Audiokinetic