[theora-dev] quantize after fdct, _dequant table, and idct

[Alexander Lubyagin]

1) What are you doing "mathematically" in a procedure x86/x86enquant: oc_enc_quantize_sse2()?
This - the assembler code, and I do not understand mathematically - that's going on there.

--- A:
      120      121       28       73      -20      -99      -98     -100
      123      122      112      108       73      -32     -102      -98
      123      123      117      121      100       74      -15      -97
      124      125      125      124      123      116       74      -33
      125      125      125      125      124      124      100       74
      124      124      124      125      125      124      121      114
      125      125      124      125      125      125      120      117
      125      125      125      125      125      125      123      113

--- _dct:
 2767 1165 -439 154 15 24 -87 -53
 -1254 1080 -245 0 125 -15 -101 -67
 -512 260 212 -207 200 -49 -80 -55
 -111 -111 300 -166 125 -17 -70 -45
 -54 -69 127 18 -31 60 -82 -30
 -50 -30 40 83 -74 72 -68 -35
 -31 -33 37 48 -40 40 -28 -44
 -21 -13 29 12 1 -3 11 -44

--- _dequant:
 24 32 32 36 32 32 40 36
 36 36 40 40 40 44 48 56
 48 48 48 48 48 56 56 56
 52 56 60 56 60 60 60 56
 60 56 60 64 64 64 60 60
 64 60 56 60 64 64 64 64
 64 64 64 64 64 60 64 64
 64 64 64 64 64 64 64 64

--- _qdct:
 115 36 -39 -14 34 -14 4 -7
 7 -3 -1 -3 5 0 0 0
 3 -4 6 -1 -1 -1 -1 2
 -3 4 0 -2 -1 -2 -1 2
 0 1 -1 0 0 1 1 -1
 0 -1 -1 -1 -1 1 -1 1
 0 0 -1 1 -1 -1 0 -1
 1 0 0 0 -1 -1 0 -1

So, _qdct[i][j] = _dct[i][j] / _dequant[i][j] ?


2) Second question - you are using:
24 20 20 20 20 16 24 20
20 20 28 24 24 28 32 40
32 32 28 28 32 44 40 40
36 40 44 44 44 44 44 44
44 44 44 48 48 48 44 44
48 44 44 44 48 48 48 48
48 48 48 48 48 44 48 48
48 48 48 48 48 48 48 48
, or
24 32 32 36 32 32 40 36
36 36 40 40 40 44 48 56
48 48 48 48 48 56 56 56
52 56 60 56 60 60 60 56
60 56 60 64 64 64 60 60
64 60 56 60 64 64 64 64
64 64 64 64 64 60 64 64
64 64 64 64 64 64 64 64
for luma plane,

and
28 28 28 36 32 36 52 40
40 52 60 56 52 56 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
for chroma plane?

3) Third question. How do you restore the original matrix after fdct and quantize ...
Used dequantize + idct?