[opus] Question about FEC and ogg/opus

[Timothy B. Terriberry]

Boris Grozev wrote:
> The ogg/opus draft describes[1] how to fill in gaps by generating
> zero-byte frames, but I do not understand how (and if) FEC can be used.
> Is this possible, and if so, what is the recommended way of doing it?
> Which component(s) would be responsible for handling FEC: the muxer
> (e.g. by extracting FEC data from a previous packet and inserting this
> instead of a zero-byte frame), the ogg player (by calling the opus
> decoder with decode_fec=1 whenever necessary), or the opus decoder itself?
>
> I failed to find any information about this specific topic. Please point
> me to relevant documentation if it exists.

I don't think we have any.

You could take a packet with FEC, extract the FEC data, and repackage it 
as a normal SILK frame in the proper place, but this requires 
re-encoding the indices with the entropy encoder (no actual 
re-quantization or searches would need to be done, so no quality would 
be lost). I am not aware of any code that exists to do this. Possibly 
such code should live in libopus (so it could re-use the existing 
encoder functions). While you're implementing that, you could also strip 
out the FEC data from the original packets, meaning you would only store 
FEC data in the saved file in places where there were actual losses. As 
long as you are not planning to re-broadcast the stream over a lossy 
channel later, that seems like a good idea. This approach also does not 
require upgrading existing players. The downside is it requires 
understanding the internals of an Opus packet.

Alternatively you could leave it up to the decoder. I did not add FEC 
decoding support to libopusfile, but only because I wanted to get 
something out the door, and it was not needed for the main use-case 
(streaming radio over http). If someone wanted to add it there, it would 
automatically mean a bunch of players would start handling this how you 
want, but certainly not all of them (Firefox, VLC, anything libav/FFmpeg 
based, etc., none of which, AFAIK, will look for FEC when they encounter 
a PLC packet). The simple case (where both the PLC packet and the 
following packet with FEC are on the same Ogg page) should be easy. The 
case where there is a page break between them would require some 
thinking to do properly, but should be doable.

I'm not sure where you draw the distinction between "decoder" and 
"player". Certainly libopus does not have enough information to solve 
this problem (it only sees one packet at a time, so it cannot look into 
the future), and libopusfile does not export enough information for a 
subsequent layer to fix up the loss after the fact, so I think 
libopusfile (or the analog in other software stacks) is the right layer 
to handle this if you want to do it from the decode side.