[opus] Opus FEC

[Timothy B. Terriberry]

Young, Milan wrote:
> * Adding FEC seems to change the encoded audio bit-stream itself, i.e., it doesn't just add additional protection bits, but also changes the encoded bits. This is easy to show by comparing two decoded versions of the same audio file: one encoded with no FEC, and another with FEC. Is this expected or is it a bug?

If you encode with the same target rate with FEC on and off, then while 
FEC is on, the encoder will reduce the rate of the non-FEC portion in 
order to achieve the same total rate. So yes, this changes the resulting 
bitstream.

> * Related to the above, what does the FEC information consist of? How does the decoder use it? Does it simply repeat preceding packets, or interpolate preceding and subsequent packets, or does it do something smarter?

The FEC information consists of "LBRR frames" in the SILK layer. The 
bitstream syntax for these is largely the same as a regular SILK frame. 
See <http://tools.ietf.org/html/rfc6716#section-4.2.5> for some details.

The decoder uses it to replace the last Opus frame of a dropped packet. 
This is done by calling opus_decode() with a pointer to the next packet 
and the decode_fec flag set to 1 (see 
<https://mf4.xiph.org/jenkins/view/opus/job/opus/ws/doc/html/group__opus__decoder.html#ga7d1111f64c36027ddcb81799df9b3fc9>). 
If no FEC is available in the packet, normal PLC is used instead.

Although it would be legal to have it simply be a repeat of the 
preceding packet (as long as the preceding packet was an active frame), 
the encoder actually requantizes it with a reduced bitrate. This lets it 
do smarter rate allocation than a simple approach like RFC 2198 or RFC 
6354 redundancy, which always repeats whole packets. Nothing prevents 
you from using one of those codec-agnostic methods with Opus, however.

>   * Based on my tests, the FEC in opus seems to have a limit, i.e., it adds correction but only up to approx 4---10% expected packet loss, depending on the file. What are the parameters/limits of the FEC?

You can find the details in silk_setup_LBRR() in silk/control_codec.c. 
Basically, there is a base bitrate at which LBRR gets activated for each 
audio bandwidth (NB, MB, and WB). For small amounts of packet loss (less 
than 25%), this threshold gets boosted (by up to 25% when there is no 
reported packet loss). As long as the target rate is above this 
threshold, LBRR gets turned on.

When it's enabled, it boosts the quantization gain on LBRR frames 
(w.r.t. normal frames) by a factor that decreases as the packet loss 
percentage goes up. At a packet loss percentage of 0, this gain gets 
increased by a factor of approximately 3. As the packet loss percentage 
goes up, the increase in quantization goes down until it hits a factor 
of approximately 1.37 at 12.5% packet loss.

>    * Does FEC add any algorithmic delay? i.e., does it require the decoder to wait until post-loss packets are received in order to generate a replacement for the lost packets? If so, what is the maximum delay and can it be variably set?

A decoder must wait for one future packet to fill in a missing hole 
using FEC. This cannot be variably controlled (i.e., the maximum 
additional delay is 1 packet). Of course, a decoder is always free to 
ignore FEC if it does not want this additional delay, or 
opportunistically use whatever happens to be in its jitter buffer, etc.