[theora-dev] quantize after fdct, _dequant table, and idct

Alexander Lubyagin lubyagin at yandex.ru
Tue Apr 5 04:25:43 PDT 2011

```1) What are you doing "mathematically" in a procedure x86/x86enquant: oc_enc_quantize_sse2()?
This - the assembler code, and I do not understand mathematically - that's going on there.

--- A:
120      121       28       73      -20      -99      -98     -100
123      122      112      108       73      -32     -102      -98
123      123      117      121      100       74      -15      -97
124      125      125      124      123      116       74      -33
125      125      125      125      124      124      100       74
124      124      124      125      125      124      121      114
125      125      124      125      125      125      120      117
125      125      125      125      125      125      123      113

--- _dct:
2767 1165 -439 154 15 24 -87 -53
-1254 1080 -245 0 125 -15 -101 -67
-512 260 212 -207 200 -49 -80 -55
-111 -111 300 -166 125 -17 -70 -45
-54 -69 127 18 -31 60 -82 -30
-50 -30 40 83 -74 72 -68 -35
-31 -33 37 48 -40 40 -28 -44
-21 -13 29 12 1 -3 11 -44

--- _dequant:
24 32 32 36 32 32 40 36
36 36 40 40 40 44 48 56
48 48 48 48 48 56 56 56
52 56 60 56 60 60 60 56
60 56 60 64 64 64 60 60
64 60 56 60 64 64 64 64
64 64 64 64 64 60 64 64
64 64 64 64 64 64 64 64

--- _qdct:
115 36 -39 -14 34 -14 4 -7
7 -3 -1 -3 5 0 0 0
3 -4 6 -1 -1 -1 -1 2
-3 4 0 -2 -1 -2 -1 2
0 1 -1 0 0 1 1 -1
0 -1 -1 -1 -1 1 -1 1
0 0 -1 1 -1 -1 0 -1
1 0 0 0 -1 -1 0 -1

So, _qdct[i][j] = _dct[i][j] / _dequant[i][j] ?

2) Second question - you are using:
24 20 20 20 20 16 24 20
20 20 28 24 24 28 32 40
32 32 28 28 32 44 40 40
36 40 44 44 44 44 44 44
44 44 44 48 48 48 44 44
48 44 44 44 48 48 48 48
48 48 48 48 48 44 48 48
48 48 48 48 48 48 48 48
, or
24 32 32 36 32 32 40 36
36 36 40 40 40 44 48 56
48 48 48 48 48 56 56 56
52 56 60 56 60 60 60 56
60 56 60 64 64 64 60 60
64 60 56 60 64 64 64 64
64 64 64 64 64 60 64 64
64 64 64 64 64 64 64 64
for luma plane,

and
28 28 28 36 32 36 52 40
40 52 60 56 52 56 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
60 60 60 60 60 60 60 60
for chroma plane?

3) Third question. How do you restore the original matrix after fdct and quantize ...
Used dequantize + idct?
```