[xiph-rtp] Chaining

[Aaron Colwell]

On Tue, Aug 30, 2005 at 11:01:36AM -0700, David Barrett wrote:
> Aaron Colwell wrote:

> I'm using Theora and Speex, as you guessed.  But I'm changing all the 
> video settings on the fly -- framerate, frame size, encoding quality, 
> etc.  (Audio too, but to a lesser degree.)

Ok well then like I said, other then the fact that you are changing frame
size, you don't actually need an encoding ID since you could use the same
codebook.

> 
> RTP timestamps are another discussion that I haven't followed closely. 
> I actually didn't realize there were any restrictions on them (ie, that 
> they need to be a multiple of anything).  I just have my decoder accept 
> arbitrary timestamps and sync up with the audio, even if the video 
> framerate is irregular.  I ignore the decoder timestamps.

The RTP timestamps must always be in the units specified by the rtpmap SDP
field. In general most payload formats require the packets to represent an
integer number of RTP timestamp units. This mainly avoid rounding
problems when doing codec frame to timestamps conversions and vice-versa. It
can also effect loss handling and A/V sync if the RTP timestamp sample rate is
not high enough to represent the highest sample rate in the stream.


> I guess I still don't see what this flood gate o' wackiness is that 
> chaining opens.  (I'm sorry if this was discussed in detail and I fell 
> behind.)  I see that the SDP gets horribly complicated if you need to 
> download a bunch of codebooks via HTTP (effectively solved by inline 
> codebook ack/retransmit).  And perhaps it's harder to write a player 
> that accepts irregular framerates, framesizes, and so forth.  But this 
> doesn't seem as bad as has been implied.  What am I overlooking?

There are several problems with allowing chaining to be supported in a general
sense.

- The client has no way to determine if it can actually play the stream
  properly. A chain might show up for a codec that it doesn't support. That
  is a bad user experience for the client. A slight variation on this is the
  case where you have an embedded netradio device. It may be able to handle
  certain codec parameters, but not others because of resource constraints. It
  will have to blindly start playback of a stream and then potential have to
  quit just because it encountered a stream it couldn't playback. This would
  be a very bad experience and is something that the user would not be able to
  figure out what was happening.

- The server might guess a bad value for the RTP timestamp sample rate. Say it
  guesses 16000Hz and then a 44100Hz sample rate chain comes along. There may
  not be enough information for the client to handle loss gracefully or ensure
  proper A/V sync. You could try to use a sample rate that has integer 
  multiples for both 44100 and 48000 sample rates, but if I remember correctly
  that sample rate is very high and doesn't provide a reasonable amount of time
  before the timestamp loops. That in itself has a bunch of complicated issues
  associated with it when loss occurs.

- Managing coordination of codebook downloads in the middle of playback is
  non-trivial. Making sure the client is notified of the new codebooks 
  "in time" is not straight forward. How is notification done? How much time
  to we have to let the client know before hand? That question directly impacts
  the amount of delay the server must introduce when switching between live
  feeds that use different codebooks. 

> All this talk on sample rates is scaring me.  I didn't realize it was a 
> requirement to be *absolutely* regular with framerate.  I thought we 
> were talking about *average* framerates.  I assumed anything stating 
> framerate is purely advisory, and the decoder should be prepared to 
> handle frames that come in on any sample frequency (ie, not puke if it 
> gets a frame before or after what it's expecting).

It is not an advisement. You are telling it the rate at which you are sampling
the video at.

> 
> After all, at the end of the day, these samples are coming from a live 
> source which *itself* isn't always producing a perfectly regular stream. 
>  How could I possibly enforce absolute regularity if my camera actually 
> generates only ~30FPS instead of a mathematically perfect 30FPS?

When you specify that the frame rate is 30fps you are effectively realigning
your images to that grid. Even if that isn't the rate that you are sampling at.
If you are specifying 30 fps as your frame rate then your capture code should
make sure that it doesn't generate more than 30 samples for each second. If
it does then you'll get sync problems because your capture app is not honoring
the 30fps contract that you set up with the encoder. This is true for most
video codecs that I'm aware of, H.263, MPEG4, RV.

> 
> 
> >I think keeping this sort of functionality is reasonable. We would just 
> >have to
> >add an encoding ID to the currently proposed SDP format. We'd have to add 
> >back
> >the chainingID field, but I think we should call it something like
> >"encoding ID" and make it 8-bits or less. I don't think anyone will need 
> >more
> >than 255 encodings.
> 
> I agree, 255 is probably enough (and 256 is even better).  You might 
> make the statement that an inline codebook transmission for an "encoding 
> ID" that is already in use overrides the old codebook.  Thus the encoder 
> can manage the "encoder-space" effectively.

I don't think overriding is a good idea. It assumes that state was actually
properly transmitted to the client. You don't have any guarentee that the
client gets the change request. Say for some reason, the change information
packets get completely lost. The client will have no clue that anything changed
and will suddenly have problems decoding packets for that encoding ID.

I hope this clears things up a little.

Aaron

> 
> -david
> 
> _______________________________________________
> xiph-rtp mailing list
> xiph-rtp at xiph.org
> http://lists.xiph.org/mailman/listinfo/xiph-rtp
>