[CELT-dev] adding celt support to netjack some questions.

[Gregory Maxwell]

On Mon, Nov 24, 2008 at 3:33 PM,  <torbenh at gmx.de> wrote:
> i am currently adding celt support to netjack.
> very nice to see a free low-latency codec :)

Greetings. I have written a yet-not-officially released tool which
interconnects multiple Jack servers using CELT.  It would be released
already but I got off on a tangent adding the ability to rewire the
remote jack system. (I.e. so you can say plug Jack_B mic_l to Jack_A
mixer_in_4 and it would handle all the routing on both sides to pass
through the interconnect.)

So, in addition to being a occasional CELT developer, I may have some
helpful thoughts for your application.

> i currently dont require robustness against packet loss,
> because the sync code of netjack does not handle packet loss very
> gracefully. how much bandwidth is wasted for this feature ?

It's inherent in the format and not deactiavtable. It costs only a
very small amount.  You want it anyways, more below.

> is it sensible, to have the data downsampled berfore encoding , in
> order to reduce bandwidth ? i suspect that just using celt to do this
> is more sensible, and leaves me with higher quality of the signal.

The CELT library officially supports integer sample rates between
32,000 Hz and 64,000 Hz.  If your jack is running at higher speeds you
will need to resample externally to CELT.

I do not expect CELT to ever truly support higher sampling rates, as
humans can not (safely) hear above ~22kHz, fruitcake audiophile
be-damned.   The correct way to support 88 and 96kHz is to resample to
44.1kHz or 48kHz.  The proper way to handle 44/48kHz is to simply run
CELT at the correct rate.

> the signal data is obtained from individual jack ports.
> i would need one additional step to make the frames interleaved.
> how much bandwidth would i save if i was using one encoder, for
> all channels, instead of n encoders for n channels ?
> considering the signals are not really correlated.

You should run N encoders. Pack their output into a single packet. You
will save considerable bandwidth from the packing (At typical CELT
latencies IP+UDP+etc overhead is significant).

> there might be transmission of several stereo or otherwise
> correlated signals though.

The CELT encoder currently supports stereo and mono modes. The stereo
mode is incomplete.  In my own app I have a separate add stereo add
mono mode. I think at this point it's overkill.

> But i would be happier with an API
> that supported individual buffers for each channel.

By buffers what exactly do you mean?  You'll suffer significant
overhead if you don't share packets. :)

> can i hook up the decoder to an already running stream ?

Yes. This is a property of the robusness to packet loss. It will take
'several' packets to synchronize and produce decent quality.  Note:
CELT never resyncronizes perfectly. So if you compare two decoders one
which started before the encoder and one which started after their
output will probably never be identical. The difference should not be
perceptible in any case.

> netjacks protocol is stateless. and i was using downsampling and
> bit rate reduction to do compression only. which are stateless
> things to do.

As you've seen, CELT is not stateless. But the robusness to loss means
that the state is soft— you can resume at any point.

> at first glance i did not see a function which tells me the
> latency of the encoding/decoding (its the overlap i guess)
> i need this to make the latency compensation code work.

CELT_GET_LOOKAHEAD should tell you the additional latency beyond a
single frame. but CELT delay is always 1.5x the frame_size today, I do
not expect this to change.

You can select any even frame size, but power-of-two are recommended
(sizes with large prime factors have reduced quality right now).
Since you should probably make the CELT frame size either be equal to
or an integer factor of the Jack frame size in order to reduce latency
sticking to power-of-two CELT frames should be acceptable for your
purposes.

> how RT-safe is the code ?
> does it use malloc during encoding/decoding ?

CELT never mallocs during encode and decode. It mallocs only during
mode setup or the creation of an encoder/decoder.

The CELT library is setup to make it fairly easy to replace malloc
with an RT-safer one at compile time for even these calls. I'd like to
make it easier to do at runtime, but I don't see an obvious way to do
it without making a mess of the API.

I have done extensive profiling of the worst case encode and decode
computational load. Nanosecond precision executing timing on core2 RT
patched linux is part of my regular test suite.  There should be
surprises waiting for you. Low complexity mode results in both lower
average CPU consumption (roughly 50% for the encoder) and something
like a ~20% reduction in worst/average ratio (from memory), but I
doubt that full complexity CELT will result in serious CPU constraints
in your usage.

> well... i will make the code use several encoders for now, and await
> some responses.

Sounds good.