[vorbis-dev] what script or command makes the codebooks? (fwd)

Erik Kruus kruus at on2.com
Tue Dec 19 11:34:14 PST 2000 


---------- Forwarded message ----------
Date: Mon, 18 Dec 2000 15:50:05 -0500 (EST)
From: Erik Kruus <kruus at on2.com>
Reply-To: vorbis-dev at xiph.org
To: vorbis-dev at xiph.org
Subject: Re: [vorbis-dev] what script or command makes the codebooks?

I've been through the entire vq table process during the last week.  Some folks
have already suffered through listening to .ogg files encoded with a 75 kbps
[stereo, 44kHz input] mode.  I ran another training run that took 30 hours
or so on a PIII-600 on a small, but reasonably-sized training set, and 

Monty gave me a file outlining what he had done to start me off, and
at the end, I attach a section.  There are some things you need to do because
the current vq/ is not quite compatible with the current lib/.

I do my retraining with the Ugliest-Bash-Script-Ever-Written, and have it
mostly automatic.

On Mon, 18 Dec 2000, Gregory Maxwell wrote:

> On Sun, Dec 17, 2000 at 11:07:28PM -0600, Mike Coleman wrote:
> > Gregory Maxwell <greg at linuxpower.cx> writes:
> > > No. It's easy to generate training data, it's just a define in the source.
> > 
> > Yeah, but I'm trying to see exactly how these particular codebooks were
> > produced.  (Actually, it would be kind of nice if each .vqh file included a
> > "pedigree" inside a comment that described exactly how it was produced (args,
> > etc), and maybe even gave a list of training data sources together with
> > md5sums.
> 
> They almost do, nothing that nice though.. Their params (in the vqhs) and
> the training set is all you need to reproduce them. 
> 
> The process runs the data through several commands, so its hard to track the
> data and it's assume that you know what you are doing if you are using those
> tools.:)
> 
> Steps, in rough,
> 
> Change #if 0 in lib source to make it generate training data,
These are compile-time switches.  I added targets to my Makefile.am
to build with -DTRAIN_LSP and/or -DTRAIN_RES (and/or -DTRAIN_RESENT,
if that's more what you need).

> 
> Train the LSP data (no info here because I havn't done it since Vorbis
> switched to lattice for LSP).
You'll probably need to change some code in latticetune.c for this to work.
(I had to:  the q_sequencep treatment was a bit off, and you don't
interleave that data).

> 
> Regenerate training data with LSP books because residue is affected by LSP
> books.
Here I mucked with vorbisenc.c to add a new mode with a certain
bit-rate window.

> 
> Run residue training data into residuesplit (parms in mode_*.h but not
> stated in an obvious way), which splits the residue into partitions.
Do this on a LARGE partition with many gigs of free space, to get a feel
for how big those intermediate files will be.

> 
> Lattice build each partition (parms can be extracted from the vqhes).
latticetune.c is the source.  Use the version that does interleaving.

> 
> Latticepare, to reduce for targeted bitrate.
'entries' or something is initialized badly if your input tables are
already sparse.  There's a trivial fix, but I pared my tables differently
for low-threshold ones, and kept the full table for the high-residual
cases.  I'll probably use latticepare myself for future experimentation,
it's just that I haven't yet done enough myself to be convinvced I changed
it correctly enough.  (I recommend you tune them without cutting them down
in size, if you want to use latticepare.  I use rather large training
sets, so I asked my scripts to zero all the never-used entries for me.)

> 
> Huffbuild to produce, aux book.
Works fine.

> 
> Latticehint the books.
Works fine.  (I think I rewired some of the vq code to use the lib/
search versions, which actually do fall back to a brute-force search.
This is nicer than the vq/ version, which will fail, at least for
some programs, because brute force is not included).
Oh yah, latticehint'ing the trivial lattice (a single quant-level)
is not handled correctly.  Don't try it.  Hard-wire those guys
directly to the first item (the '0') in secondstages[] (?name?)

> 
At this point, I did a bunch of Perl or sed commands to produce the
final mode_XX.h file, and use yet more Perl/'C' hackery to read the
"final" mode_XX.h file in at run time into the encoder to actually
use the new trial mode.  All [my code] totally unclean, and no way it's good
enough to be included in the vorbis trunk, unfortunately.  There's
been so much to do, that I just stop when it works.  Right now I'm
way behind in updating my docs from beta2-floundering-newbie to
something slightly less confused-sounding and appropriate to beta3.

If you want to see what I've been using, and promise not to hang
yourself when you see the sheer ugliness of how it "works", send
me an e-mail.  I'ld be more than glad for someone to streamline
the whole process in a way that had a nonzero chance of being
accepted into the cvs tree!

> It's been some time, so sorry I havn't been much help.. :( But with Monty
> very busy, I figured something is better then nothing..

So you now have two somethings.  I'll let you judge whether it's
better or worse than nothing.

Erik.

> 
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> Ogg project homepage: http://www.xiph.org/ogg/
> To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
> containing only the word 'unsubscribe' in the body.  No subject is needed.
> Unsubscribe messages sent to the list will be ignored/filtered.
> 
************ 96 ****************

residuesplit residue_0.vqd 0,128,32 res0_96_128 0,,32 ,1.5,32 ,2.5,32 ,6.5,32 ,1

residuesplit residue_1.vqd 0,1024,32 res0_96_1024 0,,32 9,2,32 ,2,32 21,4,32 ,4,

0
0 1
0 1 2
0 1 2 3 5
0 1 2 3 5 8 11
0 1 2 3 5 8 11 14 17 20 24 28 31 35 39 43 47 51 56 61 66 71 72 78 84 90 96 102

0
0 1
0 1
0 1 3
0 1 3
0 1 3 5 8
0 1 3 5 8 11 14
0 1 3 5 8 11 14 17 20 24 28 31 35 39 43 47 51 56 61 66 71 72 78 84 90 96 102

huffbuild res0_96_128aux.vqd 0,4,2
huffbuild res0_96_1024aux.vqd 0,32,2

latticebuild res0_128_128_1.vql >! res0_96_128_1.vqh
latticehint res0_96_128_1.vqh >! temp.vqh
restune temp.vqh res0_96_128_1.vqd >! res0_96_128_1.vqh
latticehint res0_96_128_1.vqh >! temp.vqh
mv temp.vqh res0_96_128_1.vqh

latticebuild res0_128_128_2.vql >! res0_96_128_2.vqh
latticehint res0_96_128_2.vqh >! temp.vqh
restune temp.vqh res0_96_128_2.vqd >! res0_96_128_2.vqh
latticehint res0_96_128_2.vqh >! temp.vqh
mv temp.vqh res0_96_128_2.vqh

latticebuild res0_128_128_3.vql >! res0_96_128_3.vqh
latticehint res0_96_128_3.vqh >! temp.vqh
restune temp.vqh res0_96_128_3.vqd >! res0_96_128_3.vqh
latticehint res0_96_128_3.vqh >! temp.vqh
mv temp.vqh res0_96_128_3.vqh

latticebuild res0_128_128_4.vql >! res0_96_128_4.vqh
latticehint res0_96_128_4.vqh >! temp.vqh
restune temp.vqh res0_96_128_4.vqd >! res0_96_128_4.vqh
latticehint res0_96_128_4.vqh >! temp.vqh
mv temp.vqh res0_96_128_4.vqh

latticebuild res0_128_128_5.vql >! res0_96_128_5.vqh
latticehint res0_96_128_5.vqh >! temp.vqh
restune temp.vqh res0_96_128_5.vqd >! res0_96_128_5.vqh
latticehint res0_96_128_5.vqh >! temp.vqh
mv temp.vqh res0_96_128_5.vqh

latticebuild res0_96a_1024_1.vql >! res0_96_1024_1.vqh
latticehint res0_96_1024_1.vqh >! temp.vqh
restune temp.vqh res0_96_1024_1.vqd >! res0_96_1024_1.vqh
latticehint res0_96_1024_1.vqh >! temp.vqh
mv temp.vqh res0_96_1024_1.vqh

latticebuild res0_96a_1024_2.vql >! res0_96_1024_2.vqh
latticehint res0_96_1024_2.vqh >! temp.vqh
restune temp.vqh res0_96_1024_2.vqd >! res0_96_1024_2.vqh
latticehint res0_96_1024_2.vqh >! temp.vqh
mv temp.vqh res0_96_1024_2.vqh

latticebuild res0_96a_1024_3.vql >! res0_96_1024_3.vqh
latticehint res0_96_1024_3.vqh >! temp.vqh
restune temp.vqh res0_96_1024_3.vqd >! res0_96_1024_3.vqh
latticehint res0_96_1024_3.vqh >! temp.vqh
mv temp.vqh res0_96_1024_3.vqh

latticebuild res0_96a_1024_4.vql >! res0_96_1024_4.vqh
latticehint res0_96_1024_4.vqh >! temp.vqh
restune temp.vqh res0_96_1024_4.vqd >! res0_96_1024_4.vqh
latticehint res0_96_1024_4.vqh >! temp.vqh
mv temp.vqh res0_96_1024_4.vqh

latticebuild res0_96a_1024_5.vql >! res0_96_1024_5.vqh
latticehint res0_96_1024_5.vqh >! temp.vqh
restune temp.vqh res0_96_1024_5.vqd >! res0_96_1024_5.vqh
latticehint res0_96_1024_5.vqh >! temp.vqh
mv temp.vqh res0_96_1024_5.vqh

latticebuild res0_96a_1024_6.vql >! res0_96_1024_6.vqh
latticehint res0_96_1024_6.vqh >! temp.vqh
restune temp.vqh res0_96_1024_6.vqd >! res0_96_1024_6.vqh
latticehint res0_96_1024_6.vqh >! temp.vqh
mv temp.vqh res0_96_1024_6.vqh

latticebuild res0_96a_1024_7.vql >! res0_96_1024_7.vqh
latticehint res0_96_1024_7.vqh >! temp.vqh
restune temp.vqh res0_96_1024_7.vqd >! res0_96_1024_7.vqh
latticehint res0_96_1024_7.vqh >! temp.vqh
mv temp.vqh res0_96_1024_7.vqh

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to 'vorbis-dev-request at xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.

More information about the Vorbis-dev mailing list