[Flac-dev] Non-audio applications

[Josh Coalson]

--- Daniel O'Connor <doconnor at gsoft.com.au> wrote:
> Hi,
> I work for a company which makes meteor and wind radar
> (http://www.gsoft.com.au).
> 
> On occasion (ie during meteor showers such as the Leonids) we
> configure
> the system to save raw data as it comes out of the acquisition
> system,
> the data rate for this varies (depends on acquisition parameters and
> number of coherent integrations etc), but usually it is around
> 600kb/sec.
> 
> The data consists of 16 bit samples, one for each digital channel (in
> phase and quad for each receiver) at each sampled height. Typically
> there are 10 channels and 30 range gates, so for a single transmitter
> pulse tick you get 10 * 30 = 300 16 bit samples.
> 
> In an effort to compress the data I looked at lossless audio
> compressors
> (as well as gzip etc) and I ended up trying flac and shorten. 
> 
> 
> Program       Compressed Size         Compression time       
> Decompression time
> gzip           358 Mb (84.9%)          147.96 sec              21.28
> sec
> bzip2          330 Mb (78.2%)          589.01 sec             232.13
> sec
> shorten -c 10  326 Mb (77.4%)           91.39 sec              55.89
> sec
> shorten -c 300 313 Mb (74.3%)          102.08 sec              53.04
> sec
> flac 5chn      328 Mb (77.8%)          131.22 sec              66.54
> sec
> flac 8chn      332 Mb (78.7%)          139.57 sec              66.74
> sec
> flac 10chn     327 Mb (77.5%)          126.32 sec              n/a
> sec

In every case I've ever seen, default FLAC compresses more
than default shorten.  The anomaly here is due to the number
of channels; it appears that you really do have 300 uncorrelated
channels of data (uncorrelated according to the linear predictor
I mean).  FLAC only supports 8 channels, but I would bet that
the compression would be a lot better if you separated the 300
channels stream into several streams (<=8 channels each) and
compressed them with FLAC.  That may not be practical for your
application.

> Note the 10 channel version was done by me changing the format.h file
> and recompiling, but alas I couldn't get decompression to go :(

Yes, the hard limit of 8 channels is binding everywhere; it
would take some more hacking at the code to make it handle more
channels.

> The command line I used for flac was
> /usr/bin/time flac --lax --endian=little --channels=5 --bps=16
> --sample-rate=64320  --sign=signed --force-raw-format reallyraw -o
> reallyraw.flac-5chn
> 
> I needed the --lax otherwise it generated a
> FLAC__STREAM_ENCODER_NOT_STREAMABLE error.

Yes, by default the encoder tries to encode according to a subset
of parameters that makes for easier decoding.  You need --lax to
turn that off.

> I am basically letting you guys know about a possibly more
> interesting
> than normal application for your code and soliciting any tips you may
> have :)

You could try splitting the signal like I mentioned.  It might
take some experimenting to find channel combinations that have
some exploitable correlation.  See the format page for the kinds
of inter-channel and intra-channel decorrelation FLAC does, it
might give you some more ideas:

  http://flac.sourceforge.net/format.html

Josh


__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com