[Tremor] lowmem-no-byte decoding problem

[akarui]

Monty wrote:

>What platform is this?  alloca(0) is usually legal, although like
>malloc, it is probably a good idea to guard it.
>
>For now, I would expect that simply replacing alloca(foo); with
>if(foo)alloca(foo); to be sufficient. 
>
>An older encoder release wasted a single zero-size codebook stage on
>very low bitrate files.  The files are valid and the wasted entry is
>harmless (it increases file size by a byte or two) and you've
>possibly found one of these files.
>  
>
I'm currently working on both x86 and ARM platforms.  What's actually 
happening is that since 'used_entries' is 0, the value that gets passed 
to via the formula:

work=alloca((s->used_entries*2-2)*sizeof(*work)); (line 189, codebook.c)
and
s->dec_table=_ogg_malloc((s->used_entries*(s->dec_leafw+1)-2)*s->dec_nodeb); 
(line 194, codebook.c)

turns out to be a negative number.  On the x86 platform the crash 
happens at alloca, on the ARM platform, it happens at _ogg_malloc since 
it's trying to allocate around 4GB (since the negative value is 
interpreted as an unsigned integer).

I apologize for not explaining in detail in my previous post; I wasn't 
quite sure if this was a known problem or not.

>Codebooks may be sparse; they are not required to physically occupy
>the entire codeword space they declare.  A lattice codebook might
>claim to be, for example, several thousand total codewords, but only
>actually represent a few hundred.  The used_entries variable
>represents the actual codebook size.
>  
>
Thank you for the clarification.  I wasn't aware that it was valid to 
have 0 used_entries.  For this particular case (with the negative values 
being calculated during memory allocation), I added an 'early exit' to 
_make_decode_table when 'used_entries' is 0.  Everything seems to work 
without any problems; the patch file attached to this message fixes this 
bug.

Thanks again for the clarification.

akin
-------------- next part --------------
--- ./lb/Tremor/codebook.c	2005-10-04 21:54:57.000000000 +0900
+++ ./lowmem-no-byte/Tremor/codebook.c	2005-10-04 13:22:05.000000000 +0900
@@ -183,6 +183,10 @@
 			      oggpack_buffer *opb,int maptype){
   int i;
   ogg_uint32_t *work;
+
+  if( s->used_entries == 0 )
+    return 0; /* no entries in the codebook, so there's nothing to decode */
+
   if(s->dec_nodeb==4)
     work=s->dec_table=_ogg_malloc((s->used_entries*2-2)*4);
   else