<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Jean-Marc Valin wrote:
<blockquote cite="mid1100586541.3957.15.camel@localhost" type="cite">
<blockquote type="cite">
<pre wrap="">OK, I'm actually about ready to start working on this now.
If people in the speex community are interested in working with me on
this, I can probably start with the speex buffer, but I imagine
there's going to be a lot more work needed to get this where I'd like
it to go.
</pre>
</blockquote>
<pre wrap=""><!---->
And where would you like it to go? ;-)
</pre>
</blockquote>
Heh. I guess after playing with different jitter buffers long enough,
I've realized that there's always situations that you haven't properly
accounted for when designing one. <br>
<br>
<br>
<blockquote cite="mid1100586541.3957.15.camel@localhost" type="cite">
<blockquote type="cite">
<pre wrap="">At the API level, It seems pretty easy to make the speex
implementation become speex-independent. Instead of having
speex_jitter_get call any particular speex_decode or speex_read_bits,
etc functions, it could instead just return the "thing" it got, and a
flag. I.e.
</pre>
</blockquote>
<pre wrap=""><!---->
It's not as simple as it may look -- otherwise that's what I would have
done. These are some of the things that you can't do easily if you "just
return the thing":
- Allow more than one frame per packet, especially if the frames don't
end on a byte boundary
- Let the jitter buffer drop/interpolate frames during silence periods
- Anything that requires the jitter buffer to know about what is being
decoded.
</pre>
</blockquote>
I think the only difficult part here that you do is dealing with
multiple frames per packet, without that information being available to
the jitter buffer. If the jitter buffer can be told when a packet is
added that the packet contains Xms of audio, then the jitter buffer
won't have a problem handling this.<br>
<br>
This is something I've encountered in trying to make a particular
asterisk application handle properly IAX2 frames which contain either
20ms of 40ms of speex data. For a CBR case, where the bitrate is
known, this is fairly easy to do, especially if the frames _do_ always
end on byte boundaries. For a VBR case, it is more difficult, because
it doesn't look like there's a way to just parse the speex bitstream
and break it up into the constituent 20ms frames.<br>
<br>
The problem isn't so much that the jb can't return the right thing, but
that internally it can't know if it just passed back a packet that
contained 40ms of data or 20ms of data, so later it can't know if it's
lost a frame or not.<br>
<br>
The other things can be handled based on the return value of the _get
method: dropping frames, interpolating, etc.<br>
<br>
<blockquote cite="mid1100586541.3957.15.camel@localhost" type="cite">
<blockquote type="cite">
<pre wrap="">We could then have a second-level API wrapper around this for speex
which would then call speex_decode, etc, as necessary.
Basically, I'd like the jitterbuffer to do all the history, length,
etc calculations, but not actually know anything about what it's
managing.
</pre>
</blockquote>
<pre wrap=""><!---->
I would suggest the opposite. You can just think of the current
implementation as being callback-based. If you look at the
implementation of speex_decode, it merely looks for a callback function
in a struct (mode definition). It would not be very hard to provide
similar (callback structures) wrappers for other codecs. I'm willing to
modify the current implementation to make that easier (though it's
already not very hard).
</pre>
</blockquote>
I can see how you'd do that, but I don't think that would work for me.
I really don't want the jitterbuffer to handle decoding at all, because
in some cases, I want to dejitter the stream, but not decode it.<br>
<br>
For example, I will be running this in front of a conferencing
application. This conferencing application handles participants, each
of which can use a different codec. Often, we "optimize" the path
through the conferencing application by passing the encoded stream
straight-through to listeners when there is only one speaker, and the
speaker and participant use the same codec(*). In this case, I want to
pass back the actual encoded frame, and also the information about what
to do with it, so that I can pass along the frame to some participants,
and decode (and possibly transcode) it for others.<br>
<br>
(*) (Yes, I do understand that this violates the expectations of the
codecs, but so far, it seems to work well for GSM, and for speex, since
the changes generally occur surrounded by silence, and there is a huge
benefit in scalability, and also in clarity because there is no
generational loss; the only drawback is that there is _sometimes_ some
distortion when the contexts change).<br>
<br>
<br>
<blockquote cite="mid1100586541.3957.15.camel@localhost" type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">In asterisk and iaxclient (my project), the things I'd pass into the
jitterbuffer would be different kinds of structures. Some of these
may be audio frames, some might be control frames (DTMF, etc) that we
want synchronized with the audio stream, etc. In the future, we'd
also want to have video frames thrown in there, which would need to be
synchronized.
</pre>
</blockquote>
<pre wrap=""><!---->
I'm not sure of the best way to do that. Audio has different constraints
as video when you're doing jitter buffering. For example, it's much
easier (in terms of perceptual degradation) to skip frames with video
than with audio, which means that the algorithm to handle that optimally
may be quite different. Don't you think?
</pre>
</blockquote>
Yes. While I don't plan on doing video in the first pass, the API for
this would be that when you pass in your "thing", you also pass along:<br>
<br>
a) The timestamp for that "thing"<br>
b) Some flags for the "thing". This might be a "stream number" for the
thing, or it might just be a flag saying this "thing" is audio, or this
"thing" is a control frame which must never be dropped, etc, or this is
a video frame (and maybe a keyframe), etc.<br>
<br>
A jitterbuffer for an audio/video system needs to be integrated, of
course, so that the audio and video are synchronized. In my first
implementation, I haven't decided if I want to do more than video, but
presently, in asterisk, all frames go through the jitterbuffer, and it
makes some sense to include, e.g. DTMF frames in there, HANGUP frames,
etc. (imagine leaving a voicemail, if DTMF or HANGUP isn't
jitterbuffered, and your have a _lot_ of jitter, you could lose part of
your message. If DTMF frames don't go through there, they may be
processed out-of-order, etc.).<br>
<br>
<br>
<blockquote cite="mid1100586541.3957.15.camel@localhost" type="cite">
<blockquote type="cite">
<pre wrap="">So, I guess my questions (for Jean-Marc mostly, but others as well):
1) Is it OK with you to add this extra abstraction layer to your
jitter buffer?
</pre>
</blockquote>
<pre wrap=""><!---->
I think there might be better ways to abstract the codec out of that
(callbacks and all).
</pre>
</blockquote>
callbacks or flags, I think, would work the same way, except for the
above. I think it could work for your use as well, though, if you had
some way to tell the jb out-of-band how long the "packet" you sent
along was (how many milliseconds).<br>
<br>
-SteveK<br>
<br>
</body>
</html>