[vorbis-dev] VQ generation utilities ready + usage instructions

[Monty]

The VQ codebook generation utilities are more or less ready to go.  Minor
things are left to do, but those will have to be handled as I to them :-)
The next thing to do is complete the interface to get compression modes and
codebooks to the library for encoding (minor stuff), and then arrange how
exactly the MDCT residue is filtered through the codebooks. 

VQ codebook util instructions:

***********
Input data:

First, ya gotta have a data set to train a codebook on.  Training data files
are text files of comma delimited values, one frame per line (each line should
have the same number of columns.  Essentially two forms of data are taken:

1) data sets where each column is equal to or greater than the preceeding
column; LSP coefficient sets are like this

2) data sets where the values are not monotonically increasing; essentially
'everything else'.  Raw residual data falls into theis category.

********************************
Training a codebook on the data:

The trainer for each specific purpose is custom; lspvqtrain is used for LSP
data, residuevqtrain for residue (not yet checked in; we're still playing with
it :-). genericvqtrain is a generic trainer that trains based on a mean squared
error.

each trainer takes the same command line syntax (run with no args to see it):

xxxxvqtrain basename [options] inputdata.vqd [options] [inputdata.vqd]...
...to train a set from scratch or:

xxxxvqtrain basename-n.vqi [options] [inputdata.vqd]...
...to continue training a set

'basename' is used as the root name for theoutput file.  In the case of a new
set, the file produced is basename-0.vqi (the output is a 'vq intermediate'
file).  When continuing to train a set, specify the name of the intermediate
file to continue training; if the input is 'basename-0.vqi', the output file
will be 'basename-1.vqi'.  Note that the utilities, for the sake of simplicity,
do (generally) rely on extentions and file naming conventions expressed above.

upported options:
         -p[arams]     <entries,dim,quant>
                        desired codebook entries
                        dimension of the codebook (how many input scalars per
                          codebook vector)
                        bits of resolution to represent quantized codebook entry
                          values in packed codebook
         -s[ubvector]  <start[,num]>
                        for the following input data file, start reading scalars
                          at column <start> instead of column zero
                        instead of reading vectors until the end of the line, 
only
                          read <num> vectors from each line.
         -e[rror]      <desired_error>
                        codebooks are trained to have roughly equal probability of
                          occurrence for each entry; this specifies minimum 
                          variance from the ideal probability. .05 (5%) is 
default
         -i[terations] <maxiterations>
                        maximum iterations to make while trying to converge

Options are order sensitive.  The codebook may be trained on more than one 
input data file.  Data points from the specified trainind files are stored in 
the .vqi file; additional points may be added by subsequent training runs.

************************************************************
Building a codebook header (.vqh) file from the trained set:

Once you have a .vqi file all trained up,

vqbuild file.vqi

will build a .vqh output that represents a finished codebook (a .vqh file is a valid C header for direct #include use if so desired).

********************
Metrics and testing:

vqmetrics can be used to run test data (.vqd files) through a codebook to see how it behaves.  The .vqd file need not be the file on which the codebook was trained.

usage: vqmetrics codebook.vqh [codebook.vqh]... datafile.vqd [datafile.vqd]...

vqmetrics can cascade codebooks automatically; residue from one codebook pass
is fed into the next.  Note that all codebooks (.vqh) must come before data
(.vqd) files.  Data may come from stdin.

Aside from some global metrics sent to stderr, metric datas go to files (the
basename is taken from the last codebook name appearing ont he command line).
These files can be sucked directly into Gnuplot:

       basename-me.m:       breaks all entries down into their scalar elements
and for each, plots the scalar value and mean error as an error bar (use 'with
errorbars' in gnuplot)

       basename-mse.m:      as above, but sqrt(mean squared error)

       basename-worst.m:    as above, but uses the worst-case high and low mark
matches for each scala as error bars, thus showing the full occurrence range
matched against that scalar value.

       basename-distance.m: plots a curve representing the mean match
'distance' from the actual entry error value (assuming the actual entry value
is the centroid of the cell the entry represents in N-space).  The distances
are sorted low to high and each entry's error is plotted with a width according
to probability of occurrence.

*****************************
Generating a codebook cascade

Codebooks may be cascaded; a cascade book is a codebook generated from the
residue of a previous codebook.  To produce the residue, use vqcascade (which
supports the exact same command line as vqmetrics) and send the output data to
stdout.  A two stage codebook generation might look like (from start to
finish):

genericvqtrain codebookA -p 64,8,8 training-data.vqd
vqbuild codebookA-0.vqi
vqcascade codebookA.vqh > cascade-data.vqd
genericvqtrain codebookB -p 64,8,8 cascade-data.vqd
vqbuild codebookB-0.vqi

test it out using:

vqmetrics codebookA.vqh codebookB.vqh training-data.vqd

***********************************
Generating a partitioned data space

Codebooks may be used in a straight cascade, or a codebook can be used to
partition a data space into smaller spaces; the codebook splits the input data
into multiple smaller sets, each containing the input data points belonging to
that entry.  The vqpartition utility performs this partitioning:

vqpartition codebook.vqh inputdata.vqd [inputdata.vqd]

Like the other utils, it can take more than one input data file (.vqd) and take data from stdin.  However, only one codebook can appear on the command line.

The output is split into N data files, where N is equal to the number of
entries in the codebook.  Each output file is named basename-n.vqd, where in
the above example, the base name is 'codebook'.

*****************************
Hopefully that's enough to get folks going.  Of course, this description ignores all the internal details of the VQ codebooks, training and workings. 

Monty

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/