[opus] Antw: Re: Antw: [EXT] Opus merging streams

[Sampo Syreeni]

On 2022-04-06, Andrew Sonzogni wrote:

> I've used this one but it's not THAT good: (chan1 + chan2 + chan3) / 3
> 
> The output signal may peak or be buggy some times.

That is an age old question. In general, if you assume the channels are 
statistically independent, and you want each channel to contribute 
equally and maximally, you scale each by the square root of the number 
of channels. That solution mathematically mostly goes through under the 
central limit theorem, and exactly so if each of the sources are 
Gaussian distributed. In general you can assume that with sampled real 
life signals, because of reverberation and another application of the 
central limit theorem, but in case of "digital" signals and in 
particular if you sum many channels which are coherent (contain more or 
less the same stuff), you'll fall off the ladder.

Personally I'd advocate for a pure sum with no scaling. But that sort of 
thing is kind of opinionated: it only really works if you have dynamics 
to spare and also an absolute scaling of the incoming waveforms over 
your whole signal chain. That is, each of the source waveforms at 
digital full scale are supposed to represent an absolute amplitude of, 
say, 96 dB(Z).

When you do this sort of a gain architecture, anything coming in goes 
out just as it was, unless modified. If it comes in at 24-bit fullscale, 
it'll literally break not your ears but also your windows. So you'll 
never actually put in anything near fullscale. You *might* put in 
combinations of sounds which then -- if incoherent with each other -- 
will usually scale by the square root law. Which is fine, as long as you 
keep your output fullscale to something which will break stuff. Say, do 
24 bits and set the digital fullscale at something like 130dB SPL in 
your monitors.

This by the way is how it was and is done in cinematic audio editing. 
The full dynamics shatter your bones and ass, in absolute SPL range. 
Anything lesser is just added linearly into the mix. The added bonus of 
this setup is that you actually get to use amplitude as an extra 
variable which isn't normalized away by someone turning a knob or 
putting in an extra compressor in the signal pathway. You can actually 
compose with volume, instead of assuming everybody takes your dynamics 
away at will. (Obviously in many cases they will do just that, but trust 
you me, composing in absolute amplitude leaves *so* much more dynamics 
and nuance for the rest of them idjits to churn up.)

> For your information, I'm using an ARM M4F with Opus configured like 
> this (40ms, 16kHz, 16 bitrate, 0 compres).

Just unpack and sum. You really, *really* don't want to see how an 
optimized algorithm for something like this would be. It'd take a 
cross-compile into a fully functional language, some heady CS-minded 
meta-coding concepts, and a week of computation to find the precise 
weaving-in of the functional forms. At best. It's nowhere *near* the 
best use of your time, especially since you can optimize the simple 
decode-sum-recode(?) cycle already.


As an example of the latter kind of optimization, if you have a number 
of unsynchronized opus streams decoding, you can rather easily land the 
output in circular buffers of double the block size. The code for 
updating such a buffer is classic, as is reading from it with arbitrary 
delay upto half the buffer size. The code to SIMD-vectorize arbitrary 
byte spanning reads from such a buffer is also classic, and can be found 
even from the better implementations of memcpy(). The only thing you 
need to do then is to weave your code together so that it more or less 
advances a word/cacheline/whatever-unit a time in each buffer, linearly, 
and sums over the inputs synchronously, without unduly polluting your 
CPU's data cache. It's nasty even then, but highly doable, and that's 
about as fast you can go with any library imaginable (any deeper 
algorithm would require recoding the Opus library in a functional 
language, and weaving its internals together with your buffering algo).
-- 
Sampo Syreeni, aka decoy - decoy at iki.fi, http://decoy.iki.fi/front
+358-40-3751464, 025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2