[Theora-dev] [patch] ffmpeg2theora A/V sync

[David Kuehling]

Hi,

ffmpeg2theora currently assumes that the first frame in the input file
also corresponds to the first audio sample in the input file, which
might not be true for many file formats.  I have especially bad problems
with MPEG-TS streams that I recorded via DVB-T (sync off by 0.5 seconds
etc.).  Encoding chapter-ranges from DVD might yield similar problems.

The attached patch fixes that, using audio/video timestamps
(AVPacket::pts) for implementing sync correction by duplicating/dropping
frames.  This also helps when transcoding files that have missing frames
(again, this can sometimes happen with DVB-T streams tue to bit-errors
etc.).

Not sure whether the patch works in all cases, as I don't have much
experience with libavcodec...  

Maybe command-line switches for tuning/disabling the A/V sync algorithm
might be a good idea?

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

-------------- next part --------------
Index: Makefile.am
===================================================================
--- Makefile.am	(revision 8373)
+++ Makefile.am	(working copy)
@@ -1,4 +1,4 @@
-AUTOMAKE_OPTIONS = 1.6 dist-bzip2 no-dist-gzip
+AUTOMAKE_OPTIONS = 1.6 dist-bzip2 
 
 SUBDIRS = kino_export
 
Index: ffmpeg2theora.c
===================================================================
--- ffmpeg2theora.c	(revision 8373)
+++ ffmpeg2theora.c	(working copy)
@@ -150,6 +150,18 @@
 	int64_t frame_number=0;
 	double fps = 0.0;
 
+	/* 20041216/DK variables that help us keep track of A/V sync */
+	double v_pts_out = 0.0;
+	double v_pts_in = 0.0;
+	double a_pts_out = 0.0;
+	double v_pts_delta = 0.0;
+	double v_pts_delta_new = 0.0;
+	double a_pts_delta = 0.0;
+	double framesync = 0.0;
+	const double v_pts_smoothness = 8;
+	const double framesync_thresh = 0.6;
+	int j;
+
 	for (i = 0; i < this->context->nb_streams; i++){
 		AVCodecContext *enc = &this->context->streams[i]->codec;
 		switch (enc->codec_type){
@@ -520,14 +532,46 @@
 						len -= len1;
 					}	
 					first=0;
+
+					/* 20041216/DK Check and correct A/V sync */
+					v_pts_in = (double)pkt.pts / AV_TIME_BASE;
+					v_pts_delta_new = v_pts_in - v_pts_out;
+					if (v_pts_out == 0.0)
+					   v_pts_delta = v_pts_delta_new;
+					else
+					   v_pts_delta = ((v_pts_smoothness*v_pts_delta + v_pts_delta_new) / 
+									  (v_pts_smoothness+1.0));
+					framesync = ((v_pts_delta - a_pts_delta) * this->fps);
+					if (framesync < -framesync_thresh)
+					{
+					   fprintf (stderr, "\nDropping frame at %.1f: sync off by %.3f frames\n", 
+								v_pts_out, framesync);
+					   framesync = -1;
+					}
+					else if (framesync > framesync_thresh)
+					{
+					   fprintf (stderr, "\nDuplicating frame at %.1f: sync off by %.3f frames\n", 
+								v_pts_out, framesync);
+					   framesync = 1;
+					}
+					else
+					   framesync = 0.0;
+
 					//now output_resized
-					if(theoraframes_add_video(&info, output_resized->data[0],
-					this->video_x,this->video_y,output_resized->linesize[0],e_o_s)){
-				//this->output_width,this->output_height,output_resized->linesize[0],e_o_s)){
-						ret = -1;
-						fprintf (stderr,"No theora frames available\n");
-						break;
+					for (j = 0; j < 1 + framesync ; j++)
+					{
+					   v_pts_out += (1/fps);
+					   if(theoraframes_add_video(&info, output_resized->data[0],
+												 this->video_x,this->video_y,output_resized->linesize[0],e_o_s)){
+						  //this->output_width,this->output_height,output_resized->linesize[0],e_o_s)){
+						  ret = -1;
+						  fprintf (stderr,"No theora frames available\n");
+						  break;
+					   }
 					}
+
+					if (ret == -1)
+					   break;
 					if(e_o_s){
 						break;
 					}
@@ -564,6 +608,9 @@
 						ret = -1;
 						fprintf (stderr,"No audio frames available\n");
 					}
+					/* 20041216/DK keep track of audio timing for A/V sync */
+					a_pts_delta = (double)pkt.pts / AV_TIME_BASE - a_pts_out;
+					a_pts_out += (double)samples_out / this->sample_rate;
 					if(e_o_s && len <= 0){
 						break;
 					}