[Theora-dev] Theora camera/fast computers

[Andrey Filippov]

> The noise level (including the chroma subsampling artifacts) is pretty
> high, but doesn't change between frames in the static parts of the
> scene.

Several different effects. Color artifacts - yes, I know about them and
will improve. In the camera data from the sensor is temporalily stored in
the external (to FPGA) SDRAM as Bayer pattern. Later they go to the
compressor in 20x20 overlapping tiles (corresponding to macroblocks), but
currently only 18x18 are used - the same algorithm that was in earlier
model 313 camera that had much weaker FPGA. Then there is a direct
Bayer->YCbCr 4:2:0 converter with no RGB intermediate data
(http://cvs.sourceforge.net/viewcvs.py/elphel/camera333/fpga/x333/csconvert333.v?view=markup)
In that converter only 3x3 pixels are used to restore data for each pixel.
When I was designing 333 memory controller (the "heart" of the system) I
implemented 20x20 tiles (so 5x5 core for each pixel), but for now used the
old code focusing on the compressor first. Now it will be rather easy to
implement mopre advanced algorithm for the Bayer->YCbCr converter. BTW,
4:2:0 is sufficient for the color sensors with mosaic filters.

In the dark areas (there is non-linear table-based converter on the input
data that gets 10 bits per pixel data and outputs 8 bbits) is also
possible to get visible some fixed-pattern-noise. I have hardware FPN
correction (each pixel can be on-the-fly added and multiplied with the
preloaded array of coefficients - a pair for each pixel), but some
software is needed to prepare and manipulate this data.

The other effect is that (for test purposes - to see accumulating errors
if any) each INTRA is followed by 255 INTER frames and the quantization
coefficients for INTER frames produce lower quality. So I could see some
trail after the moving objects - but it is still within threshold of the
quantization. Maybe it can make sense to use some of INTER frames (i.e. 1
in 10) with higher quality that the rest - they will still be smaller than
INTRA but can better cover up the trails. It will be rather easy to try
with the camera that can support 2 alternative QIS - tables are loaded in
advance (currently - total 8) to FPGA and then each frame can use all
"low", all "high" or according to the loaded map with 32x32 pixel
resolution.

The other thing - I noticed that even on the completely static patterns -
it took several INTER frames to improve results of INTRA - and I suspect
the effect of not the block average values before the DCT, but I didn't
make any experiments/calculations with it yet.

> That, the resolution, and watching it at 7 fps, gives it a really
> interesting 'frozen in time' feeling.
>
>> This will come. Currently when there is no restriction to the frame rate
>> placed (in a web interface when the frame rate input field is blank or
>> 0)
>> systems tries to run sensor as fast as possible. With the default sensor
>> clock of 48MHz and required "margins" around the used window of
>> (1280+4)*(1024+4) it all results to that frame rate. It is possible to
>> program the sensor clock and also change "virtual frame" of the sensor
>> by
>> adding invisible rows and/or columns.
>
> So it just works out that way? Interesting. How fast can the sensor go?
> And can you bin the pixels to read out the full frame at lower
> resolution?

Yes, but then it is not optimal for the color images or it will need
different algorithms as the pixels will not be even. The
http://theoracamera.elphel.com has 1280x1024 sensor installed, but
normally they are
http://www.micron.com/products/imaging/products/MT9T001.html that can
support real binning, not just decimation. You may look at different
images/sensors through 3 online model 313 cameras
(http://camera1.elphel.com - 1280x1024, http://camera3.elphel.com -
1600x1200, http://camera2.elphel.com - 2048x1536) - the same will be with
the model 333.

> Well, that's not existing software. IF you bought a dual 3 GHz Xeon, AND
> Ruik gets a good speedup with the idct, AND some simple paralellization
> scheme works, THEN you might hit 30 fps by mid April. I guess you'll get
> to find out!

That can be an expensive system, but it is fun to have a FOSS-based system
that has significantly higher performance than any of the proprietary ones
:-)

>
> I'd experiment with paralellization first to see what scales the best.
> Off the top of my head you could try: serial stream decode but parallel
> frame reconstruction (best for smp) or entirely parallel decode dropping
> anything outside the crop (might work on a cluster of mac minis).
>
>> I'm trying to be able to demonstrate the camera at it's best :-). BTW, I
>> made a clip of the falling water droplets on a narrow frame of 1280x64
>> pixels at 360fps.
>
> Cool! Put that one online. :)
Here it is - not really impressive - water was dripping from the roof from
melting snow and the droplets were out of focus (exposure was 1ms):
http://www.elphel.com/3fhlo/samples/333_samples/360_fps2.ogg

It is turned 90 degrees - sensor can run much faster with horizontal
stripe, than vertical.


Andrey