[theora] Marketing stuff

[Aaron Colwell]

comments inline

Maik Merten wrote:
> Aaron Colwell schrieb:
>   
>> In my opinion I believe that these claims are anecdotal at best. Has
>> anyone actually
>> done a true bake off between Theora and the codecs mentioned? Has anyone
>> done PNSR
>> vs bitrate or similar comparisons across a large and diverse body of
>> content? For
>> me these types of comparisons are far more compelling than a zoomed in
>> area of a single
>> frame where the deblocking filter happened to do a good job.
>>     
>
> First of all PSNR is of course no usable quality metric. Nothing but
> human eyes can judge quality, which is what makes tuning codecs a
> non-trivial thing.
>
>   
While I agree that human eyes are the best, PSNR even with all of its 
flaws has been commonly used
in academic papers, codec evals, etc. to compare codec performance. 
While it isn't the best metric it
is A metric that can be informative. I'd prefer some sort of 
double-blind human test, but obviously that
is much more difficult to coordinate.
> I'm doing claims regarding some codecs that happen to not have an
> in-loop deblocking filter. From today's point of view they are dated but
> still widely used.
>
> At extremely low bitrates the lack of advanced deblocking obviously is a
> problem:
>
> http://people.xiph.org/~maikmerten/sign_irene_484_mpeg4-2.png
>
> http://people.xiph.org/~maikmerten/sign_irene_484_theora.png
>
> (no postprocessing applied, QCIF, 25 fps, ~130 kbps)
>
>
>   
I'm also concerned with terms like "Extremely low bitrate", which 
apparently maps to ~130kbps in
your mind. For someone like me who has spent quite a bit of time in the 
past with mobile & modem speed
media delivery, this isn't "extremely low" at all. Info like frame-size, 
frame-rate, and bitrate are key to
deciphering what "extremely low bitrate" really means in this context.

My agenda here isn't to say Theora sucks compared to proprietary codecs. 
I'm just trying to suggest
that Theora's worthiness should be backed by some form of "objective 
metrics"(flawed yes, but provides
some insight) that other codecs use to judge their performance. I think 
it's also important to expose
performance over a range of use cases. Each codec has its own sweet 
spot. Where is Theora's? What
does it do well? Where does it tend to break down? What is the 
bitrate/quality tradeoff I have to make
if I pick Theora over say MPEG4-Part 2? How good is it at hitting the 
target bitrate specified? These
are all questions that content providers are going to want answers to 
when deciding whether Theora is
the right codec for them.
> The obvious problem with this small comparision is the "no
> postprocessing applied", which is a pretty uncommon setup for MPEG-4
> Part 2 as most players *do* apply some postprocessing (so quality also
> does depend on the player software). Then again most players *don't*
> apply postprocessing on Theora albeit the library offers powerful
> built-in postprocessing (which is not enabled by default). Finding a
> common and fair ground for comparisons sadly isn't trivial.
>
>   
I think this is a separate but important discussion. It would first be 
valuable to do an evaluation where
the codecs don't have the aid of a post -filter. Then you could do a 
post-filter bake off to determine what
post-filters work the best for each codec. Then you could rerun the 
initial evaluation with each codec using
its optimal post-filter.
> To put it into a nutshell:
>
> I did some tests to make sure I am not making completely misguided
> statements - but to be 100% fair I perhaps indeed should not mention any
> competing codecs until there's an exhaustive set of experimental results.
>
>   
I didn't doubt that you've done some tests. My point is we don't have 
any way of knowing that nor
do we have any idea what specific use cases (bitrate, frame size, frame 
rate, CBR vs VBR, 1 vs 2-pass)
your claims apply to.

My intent isn't just to complain here. I am willing to help if people 
decide they want to do this sort of analysis.
This should be relatively easy to do with minor modifications to ffmpeg 
& ffmpeg2theora.

Aaron