[theora-dev] DCT in Theora and DCT "by hands"

[Alexander Lubyagin]

Hello.

I try to perform DCT using oc_enc_fdct8x8_x86_64sse2(), and by Python-script dct.py.
Results differ. Why?

Original pixel data block:
Morig = [
[28,28,28,28,28,28,28,28],
[28,28,28,28,28,28,28,28],
[28,28,28,29,29,29,28,28],
[28,29,28,29,29,29,28,28],
[28,29,28,28,28,28,28,28],
[29,28,29,28,28,28,28,28],
[29,28,29,29,29,29,28,28],
[29,29,29,29,29,29,29,29],
]

Result by libtheora:
 *    -3187        3       -3        2        0       -1        1       -1
 *       -5       -1        0        1        0       -1       -1       -1
 *        4        0        3        0        1        2        2        2
 *       -4        4        5       -1       -1        1        1        2
 *        2        0        2        0       -1       -2       -3       -2
 *        2        1       -4        3        2        0        2       -1
 *        2        1        2        0        1        1        1        1
 *       -1        0        0        1        0        0        0       -1

Result by dct.py:
  -1479     55     50     47     39     30     21     10
     52     -2     -1     -1     -1     -1     -1      0
     52     -2     -1     -1     -1      0      0      0
     44     -1      0     -1     -1      0      0      0
     40     -1     -1     -1     -1     -1     -1      0
     32     -1     -2      0      0      0      0      0
     21      0      0      0      0      0      0      0
     10      0      0      0      0      0      0      0
----

Source code of dct.py:
#!/usr/bin/python

# pixels
Morig = [
[28,28,28,28,28,28,28,28],
[28,28,28,28,28,28,28,28],
[28,28,28,29,29,29,28,28],
[28,29,28,29,29,29,28,28],
[28,29,28,28,28,28,28,28],
[29,28,29,28,28,28,28,28],
[29,28,29,29,29,29,28,28],
[29,29,29,29,29,29,29,29],
]

from math import sqrt,cos,pi
from sys import stdout
M = 8
N = 8

def dct(a):
  print "---"
  F = []
  for u in xrange(N):
    F = F + [[]]
    for v in xrange(M):
      s = 0.0
      for i in xrange(N):
        for j in xrange(M):
          #print a[i][j]
          li = 1
          if (i == 0): li = 1/sqrt(2)
          lj = 1
          if (j == 0): lj = 1/sqrt(2)
          d = li*lj*(a[i][j]-128)
          #d = (a[i][j]-128)
          d *= cos(((pi*u)*(2*i+1))/(2*N))
          d *= cos(((pi*v)*(2*j+1))/(2*M))
          s += d
      s *= sqrt(2.0/N)
      s *= sqrt(2.0/M)
      F[u] = F[u] + [s,]
      stdout.write(" %6d" % s)
    stdout.write("\n")
  return F

dct(Morig)

# END

Source code based on http://www.cs.cf.ac.uk/Dave/Multimedia/node231.html
formulas for 2D-DCT.

-- 
 Alexander Lubyagin