[flac-dev] More than 150 MB / second encoding + "nanozip"

[Nouvelle Collection]

Thanks for the info. I will look at encode.ru .

In the meantime I discovered "nanozip" : http://www.nanozip.net/

On  http://compressionratings.com/sort.cgi?aud1.brief+4nf_pf, the result is
amazing : 100 MB compressed in ... 0.44 sec !
with compressed size = 60% of the original size !
i.e. ratio = similar to FLAC, and very very fast...

nanozip   0.09a    -m.5g -cf    *****    63 710 155 bytes    60.2 %    0.44
sec    0.44 sec

Unfortunately, "nanozip" is closed source. Do you have any idea on the
algorithm behind that makes this possible ?



2014-03-17 21:22 GMT+01:00 <neheb at hushmail.com>:

> flac -0 would work. I was under the impression that it just packed and did
> not compress but that was wrong. For FLACCL as well.
>
> FLACCL is faster and compresses better. My test file was encoded in 10
> seconds using FLACCL and 26 using regular FLAC. If GPU is not possible,
> you're probably SOL.
>
> FLACCL -0 --fast-gpu --no-md5: 805893523 bytes - 10.687 seconds
> FLAC -0: 838208659 bytes - 27.170 seconds //no idea how to remove the MD5
> checksum from the encode. The MD5 function is not SSE-accelerated and is
> pretty slow.
>
> One other variant is to use a general purpose compressor(maybe LZ4 or even
> Zhuff) and apply a filter to get the size down. I tried this and got
> miserable results, mainly because the filter that i used was slow. You may
> want to post on http://encode.ru if you wish to take this path. The site
> is not in Russian despite the last two letters.
>
> On 3/17/2014 at 10:08 AM, "Nouvelle Collection" <
> nouvellecollection at gmail.com> wrote:
> >
> >Hello neheb,
> >
> >Thanks for your answer.
> >Unfortunately, I won't be able to use something that needs a GPU,
> >because
> >some users may not have a GPU on their computer.
> >What's the fastest option when running "standard FLAC" in order to
> >have
> >fastest compression time ?
> >
> >> As an aside: the author of LZ4 has also created Zhuff which is
> >basically
> >LZ4 with an entropy coder. Encoding
> >> speeds are slightly slower but decompression is much faster.
> >Compresses
> >better as well.
> >> See: http://fastcompression.blogspot.com/p/zhuff.html
> >Very cool ! I tried it and I think it will probably fit my needs.
> >I just need to find a Python-Zhuff binding :)
> >
> >Best regards
> >
> >
> >
> >
> >
> >2014-03-17 1:11 GMT+01:00 <neheb at hushmail.com>:
> >
> >> FLACCL might work: http://www.cuetools.net/wiki/FLACCL
> >>
> >> The problems with it are: slow initialization time. The OpenCL
> >kernel must
> >> be compiled during the first run. Which leads me to the second
> >drawback.
> >>
> >> It requires a GPU. Core i5s include an integrated GPU which can
> >be
> >> utilized.
> >>
> >> The latest version of FLACCL is at:
> >> http://www.cuetools.net/install/CUETools_2.1.5.zip
> >>
> >> Recommended command line would probably be: -1 --lax --no-md5 --
> >fast-gpu
> >>
> >> --slow-gpu may produce faster results on your end. Needs to be
> >tested. -1
> >> could also be increased if needed. --no-md5 could also be
> >removed but the
> >> code that calculates the MD5 hash is slow. I also think that an
> >MD5 hash of
> >> the audio data is pointless since FLAC includes CRCs for every
> >block.
> >>
> >> Also, there is currently a bug in the flac.cl kernel which
> >sometimes
> >> produces wrong encodes at higher compression levels. The fixed
> >kernel can
> >> be downloaded from:
> >>
> >http://sourceforge.net/p/cuetoolsnet/code/ci/default/tree/CUETools.
> >Codecs.FLACCL/flac.cl?format=raw
> >>
> >> For a slightly smaller size, you can change the line "cbits =
> >min(cbits,
> >> clz(order + 1) + 1 - shared.task.obits);" to "cbits = min(cbits,
> >clz(order)
> >> + 1 - shared.task.obits);". I checked with the author to make
> >sure this is
> >> correct.
> >>
> >> Also: BIG NOTE: The best results that I could get on my machine
> >was around
> >> 110MB/s (based on my own calculations). It may or may not be
> >acceptable for
> >> your purposes.
> >>
> >> As an aside: the author of LZ4 has also created Zhuff which is
> >basically
> >> LZ4 with an entropy coder. Encoding speeds are slightly slower
> >but
> >> decompression is much faster. Compresses better as well. See:
> >> http://fastcompression.blogspot.com/p/zhuff.html
> >>
> >> On 3/16/2014 at 10:27 AM, "Nouvelle Collection" <
> >> nouvellecollection at gmail.com> wrote:
> >> >
> >> >Hello,
> >> >
> >> >Is there some version of FLAC that allows very very fast
> >encoding
> >> >(i.e.
> >> >able to process at least 150 MB / second of .wav input data on a
> >> >standard
> >> >computer : laptop computer, Core i5/i7,  Windows 7 64 bit, 8 GB
> >> >RAM) ?
> >> >(It's ok to have a compression ratio which is a little bit lower
> >> >than
> >> >traditionnal FLAC)
> >> >
> >> >I'm looking for something which is between FLAC (very good
> >ratio,
> >> >slower
> >> >than LZ4)   and  LZ4 (very very fast compression : 400 MB / sec,
> >> >but lower
> >> >compression ratio than FLAC because it's not dedicated to
> >audio).
> >> >
> >> >Best regards.
> >>
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/flac-dev/attachments/20140317/b279628d/attachment.htm