[vorbis-dev] slightly different audio output

[Segher Boessenkool]

> -I debuged the code and found that mdct_backward() returns quite different
> values between float and int setting. (before MDCT, there error is almost
> off-by-one, at most 1%).

Yes.  The integer mdct code in libvorbis is pretty bad.  There are
comments in the code that say so, even :)

> -I'm using MDCT code from RC3 (not sure if the code in CVS is better?). I
> have tried with 14-bit TRIGBITS and then modified the code to include
> Segher's magic union:
> http://www.xiph.org/archives/vorbis-dev/200202/0081.html. And run it at
> TRIGBITS = 14, 20, 25, 30.

Do you use rounding or truncating? ...

> -the output from mdct_backward() seems to converge to a set of values when
> increasing TRIGBITS but it doesn't converge to the result from floating
> point version though.

... If you use truncating, the libvorbis mdct algorithm is pretty bad.

> -has this int implementation of MDCT been proved to yield the same (or
> almost the same) result with floating point version when using at high
> TRIGBITS? Maybe my workaround is not correct..

No, but it _has_ been proven to yield bad results (in some cases).

<p>If you have 1024-or-so-bit integer math, converting floating point code
to integer code is easy.  In all other cases, you need some
serious engineering work, to make it high-quality.

I'm sure you'll enjoy it,

<p>Segher

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.