[CELT-dev] Mixing of compressed streams

[Benjamin M. Schwartz]

On 11/28/2011 05:28 PM, Mike Hooper wrote:
> Is it possible to mix (sum) two or more compressed audio streams with
> either CELT or Opus?

Of course it always possible: just decompress, mix, and recompress!

The usual question is: how much CPU does this cost, and can we save some
by not fully decoding the streams?
The answer is:
1.  It doesn't cost a lot of CPU.  Opus is designed to be CPU-efficient
for both encode and decode.
2.  The best way to save CPU is probably to optimize the encoder and
decoder, which presently have very little in the way of performance
optimizations.
2a.  Once the encoder and decoder are highly optimized, then you can
probably save an addition 10-20% of CPU time by implementing a
transform-domain mixer, provided that the two streams are in the same mode.

The problem with 2a is to convince yourself that the CPU time is really
more expensive than the engineers' salaries.

Maybe your question is: can we mix two streams without experiencing
"tandem loss"?  The answer is no.  Even the "2a" mixer cannot be lossless
in CELT or Opus.  However, Opus should tandem with itself very nicely ...
and experience has shown that in lossy compression, tricks to try to
improve quality during tandem compression rarely do much better than
simply re-encoding.

Maybe a more interesting interpretation of your question is: is it
possible to mix two CELT or Opus streams without adding any latency?  The
answer is yes ... but you need to do all the engineering for 2a above, and
it only works if both clients stick to the same mode.  ("no added latency"
is a tricky concept, though, because senders typically have a random clock
offset, so it's more like "no more latency than the inter-sender clock
shift".)

Hypothetically, a no-added-latency, transform-domain mixer for Opus could
even do simple stereo panning on its inputs.

--Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
Url : http://lists.xiph.org/pipermail/celt-dev/attachments/20111128/81397a40/attachment.pgp