[theora-dev] [Cortado] How to support seeking in on-the-fly generated Theora stream?

[David Schueler]

Hello all!

i want to use Theora in a little video portal, because its free and open 
source and i want to contribute some code to get the Java Theora player 
(Cortado) more feature-rich.
But now I'm stuck and hope that someone can point me into the right 
direction.

I have videos stored in several formats (mostly H.264 or MPEG4) and use 
ffmpeg2theora to recode them on the fly. The recoded output is send via 
HTTP stream directly to the Cotrado player which stores the data and 
begins playing if enough data is buffered.
In practice this looks like this is a PHP file:
passthru("ffmpeg2theora -o - ".$source." 2> /dev/null");

I recoded the Queue.java (Queue class) that the queue array is not a dump 
FIFO anymore and called it QueueSeek.java.
Now the Buffer array is filled at the end and a variable points to the 
current play position which gets increased on every loop and sends the 
data to the next sink pad (at OggDemux).

QueueSeek Buffer:

           +--# data added at the end
+0-1-2-3-4-v-------
|#|#|#|#|#| | | | -> unlimited capacity
+----|-------------
     v
current play position
data will be send to demuxer
but not removed from the FIFO

Now i managed that the fill level of the buffer is the difference between 
the size of the data in the FIFO and the current play position. So the 
play begins if the data is enough and stops if the "watermark" gets below 
the lowest level.

     + current play position
+0-1-V-3-4-5-6-----
|#|#|#|#|#|#|#| | 
+----|-------|-----
     |buffer-|
      level=4

Now i want to seek. And thats where the problems start to begin. I 
understood how the demuxer works. It gets the data from the Queue, stores 
it in its own buffer and seeks for a "OggS" pattern which marks the start 
of a new Theora page. Then is checks if that whole page is present in the 
buffer and calculates a checksum. If all went well the page is processed 
in a way i did not look about any further.
But if i seek in my new Queue - which means just setting the play position 
to any value - then the demuxer stumbles, because the checksum and the 
length of the page  are not correct.
To get around this i did a little trick:
On my new FIFO i seek though the incoming data and look for the OggS 
pattern. If it is present the data will be split into two segments and 
both will be inserted info the FIFO. Now every page starts at the 
beginning of a FIFO buffer:

+------0-------------1-------------2------------3-----
| OggS...data | OggS...data.|....data    | OggS...
+-----------------------------------------------------

As you can see, sometimes the data is lager than the 4096 byte of one 
buffer, so it takes more than one buffer for a page. I've taken care of 
that in the seeking method by rewinding the play position until the buffer 
starts with an OggS pattern.
But now a second problem exists: The seeking should start if one page is 
completely submittet. And thats where i'm stucking now.

The pads of the elements are separate threads which work independently 
from each other. 

----+        +-----------[QueueSeek element]---------+         +----
prev|        |                                       |   to    | sink pad
src O--push->O sink pad --> [FIFO magic] --> src pad O--next-->O of the
pad |        |                                       | element | demuxer
----+        +---------------------------------------+         +----

Thats why many code blocks are synchronized on the queue object, that only 
one thread can access the queue at one time. Now, i want so synchronize 
the sink- and sourcepad threads of the QueueSeek class on another object, 
which is called "permissionToSeek". The sinkpad should wait until it gets 
a permission to seek.
I tried to achieve this with wait() and notifyAll().
the sinkpad has a method which gets called if a seek request arrives. It 
synchronizes on the permission object and waits for the srcpad to grant 
it:

private void doSeek(Event event) {
    // wait until we get the permission to seek
    synchronized (permissionToSeek) {
    try {
        Debug.info("[BUFFER] sink -> doSeek() waiting for 
permissionToSeek");
        // wait until we are allowed to seek
        permissionToSeek.wait();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    /* ... code to seek ... */
}

The sinkpad checks if the next buffer begins with OggS pattern and calls 
the notifyAll() method:

protected void taskFunc() {
    boolean runSeekIfWanted = false;
    synchronized (queue) {
        /* ... */
        if (playPosition+1 < queue.size()) {
            obj = queue.elementAt(playPosition+1);
            if (obj instanceof Buffer) {
                Buffer bb = (Buffer) obj;
                if (bb.data[0] == 'O' &&
                    bb.data[1] == 'g' &&
                    bb.data[2] == 'g' &&
                    bb.data[3] == 'S') {
                    runSeekIfWanted = true;
                    Debug.info("[BUFFER] src setting permissionToSeek");
                }
            }
        }
        /* ... */
    }
    /* ... main method code ... */
    synchronized (permissionToSeek) {
        if (runSeekIfWanted) {
            Debug.info("[BUFFER] src granting permissionToSeek");
            permissionToSeek.notifyAll();
        }
    }
}

The problem is that the applet stops in the doSeek() method at the wait() 
and waits forever. The whole applet does not respond anymore:

[INFO] [BUFFER] playPosition = 324, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 678325 -> 64% full
[INFO] [BUFFER] src granting permissionToSeek
[INFO] [BUFFER] playPosition = 325, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 679936 -> 64% full
[INFO] [BUFFER] playPosition = 326, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 679962 -> 64% full
[INFO] [BUFFER] src setting permissionToSeek
[INFO] doSeek(): aPos=0.05273069679849341
[INFO] doSeek(): Element: [pipeline].sendEvent [Event] type: SEEK, format: 
5, position: 52730
[INFO] Element: [pipeline] sendEvent(): [Event] type: SEEK, format: 5, 
position: 52730
[INFO] Element: [pipeline] doSeek(): [Event] type: SEEK, format: 5, 
position: 52730
[INFO] Element: [pipeline] doSendEvent(): [Event] type: SEEK, format: 5, 
position: 52730 to Pad: httpsrc:src
[INFO] [BUFFER] sink -> doSeek() waiting for permissionToSeek

If i do a wait for 5 seconds to get around this issue, i get discontinue 
messages in the console, and i don't know why:

[INFO] theora: got discont
[INFO] theora: got discont
[INFO] theora: got discont

So my first question is:
Is my whole thinking right with providing full pages to the demuxer? Is 
that the right way for seeking in a Theora video stream?
Why does my applet wait forever? I suppose the srcpad does not fire the 
notifyAll() method but i have no idea why or where the srcpad gets stopped 
or interrupted.

I will provide the whole code project in a .tar.bz here, so everyone can 
use and experiment with my code changes:
http://www.wapkamera.de/cortado.tar.bz

Hopefully someone is able to help me, because i just started in coding 
Java and i don't know the Theora video stream structure in detail.

Best regards to everyone.

David