1 \documentclass{article}
2 \usepackage[usenames]{xcolor}
4 \title{Decoder structures}
10 At the time of writing we have a get-stuff-at-this-time API which
11 hides a decode-some-and-see-what-comes-out approach.
13 \section{Easy and hard extraction of particular pieces of content}
15 With most decoders it is quick, easy and reliable to get a particular
16 piece of content from a particular timecode. This applies to the DCP,
17 DCP subtitle, Image and Video MXF decoders. With FFmpeg, however,
20 This suggests that it would make more sense to keep the
21 decode-and-see-what-comes-out code within the FFmpeg decoder and not
24 However resampling screws this up, as it means all audio requires
25 decode-and-see. I don't think you can't resample in neat blocks as
26 there are fractional samples and other complications. You can't postpone
27 resampling to the end of the player since different audio may be
28 coming in at different rates.
30 This suggests that decode-and-see is a better match, even if it feels
31 a bit ridiculous when most of the decoders have slightly clunky seek
34 Having said that: the only other decoder which produces audio is now
35 the DCP one, and maybe that never needs to be resampled.
38 \section{Multiple streams}
40 Another thing unique to FFmpeg is multiple audio streams, possibly at
41 different sample rates.
43 There seem to be two approaches to handling this:
46 \item Every audio decoder has one or more `streams'. The player loops
47 content and streams within content, and the audio decoder resamples
48 each stream individually.
49 \item Every audio decoder just returns audio data, and the FFmpeg
50 decoder returns all its streams' data in one block.
53 The second approach has the disadvantage that the FFmpeg decoder must
54 resample and merge its audio streams into one block. This is in
55 addition to the resampling that must be done for the other decoders,
56 and the merging of all audio content inside the player.
58 These disadvantages suggest that the first approach is better.
60 One might think that the logical conclusion is to take streams all the
61 way back to the player and resample them there, but the resampling
62 must occur on the other side of the get-stuff-at-time API.
67 Thinking about this again in October 2016 it feels like the
68 get-stuff-at-this-time API is causing problems. It especially seems
69 to be a bad fit for previewing audio. The API is nice for callers,
70 but there is a lot of dancing around behind it to make it work, and it
71 seems that it is more `flexible' than necessary; all callers ever do
74 Hence there is a temptation to go back to see-what-comes-out.
76 There are two operations: make DCP and preview. Make DCP seems to be
79 \lstset{basicstyle=\footnotesize\ttfamily,
81 keywordstyle=\color{blue}\ttfamily,
82 stringstyle=\color{red}\ttfamily,
83 commentstyle=\color{olive}\ttfamily}
87 done = player->pass();
88 // pass() causes things to appear which are
89 // sent to encoders / disk
93 And preview seems to be
98 done = player->pass();
99 // pass() causes things to appear which are buffered
100 sleep_until_buffers_empty();
105 get_video_and_audio_from_buffers();
111 \texttt{Player::pass} must call \texttt{pass()} on its decoders. They
112 will emit stuff which \texttt{Player} must adjust (mixing sound etc.).
113 Player then emits the `final cut', which must have properties like no
116 Maybe you could have a parent class for simpler get-stuff-at-this-time
117 decoders to give them \texttt{pass()} / \texttt{seek()}.
119 One problem I remember is which decoder to \texttt{pass()} at any given time:
120 it must be the one with the earliest last output, presumably.
121 Resampling also looks fiddly in the v1 code.
124 \section{Having a go}
128 virtual void pass() = 0;
129 virtual void seek(ContentTime time, bool accurate) = 0;
131 signals2<void (ContentVideo)> Video;
132 signals2<void (ContentAudio, AudioStreamPtr)> Audio;
133 signals2<void (ContentTextSubtitle)> TextSubtitle;
141 virtual void pass() = 0;
142 virtual void seek(ContentTime time, bool accurate) = 0;
144 shared_ptr<VideoDecoder> video;
145 shared_ptr<AudioDecoder> audio;
146 shared_ptr<SubtitleDecoder> subtitle;
150 signals2<void (ContentVideo)> Data;
156 \item Video / audio frame or \texttt{ContentTime}?
157 \item Can all the subtitle period notation code go?
163 \item Remove \texttt{get()}-based loops and replace with \texttt{pass()} and signal connections.
164 \item Remove \texttt{get()} and \texttt{seek()} from decoder parts; add emission signals.
165 \item Put \texttt{AudioMerger} back.
166 \item Remove \texttt{during} stuff from \texttt{SubtitleDecoder} and decoder classes that use it.
167 \item Rename \texttt{give} methods to \texttt{emit}.
168 \item Remove \text{get} methods from \texttt{Player}; replace with \texttt{pass()} and \texttt{seek()}.