[Icecast] queue-size
Milton Huang
mph at emotrics.com
Wed Dec 29 22:16:11 UTC 2021
Man, that was awesome. I think that formula should be put in the doc pages
somewhere, as it really clarifies what that code is doing.
Thanks for all the details!
On Fri, Dec 24, 2021 at 4:45 AM Philipp Schafft <
phschafft at de.loewenfelsen.net> wrote:
> Good morning,
>
> On Wed, 2021-12-22 at 20:28 -0800, Milton Huang wrote:
> > Wow. This has been helpful to me to really understand what is going
> > on. I'd suspected the 'audio_bitrate' was just for show.
>
> happy to hear. :)
>
>
> > So, to again make sure I'm understanding, I assume I can get the
> > actual (average) transfer rate of data to a particular user for a
> > particular session by taking the access.log BYTES = total bytes of
> > data sent (other than headers) and dividing by TIME = seconds that
> > the connection lasted.
>
> Generally this is the way to go. However with the limit of BYTES and
> TIME being big enough. See below.
>
>
> > This varies depending on many of the factors you mention,
> > particularly the amount of information in the signal. And for those
> > reasons, it can be misleading if measured in a very short window.
>
> Correct.
>
>
> > I just looked at one of my logs and saw a huge spread for the 1
> > second duration... Don't know how to post pictures though...
>
>
> That is right. There are a number of factors here that need to be kept
> in mind for very short connections:
>
> First of all Icecast sends the burst (of wich we know the estimated
> size from the configuration, but again: exact size is unknown to us as
> it depends on the signal at given point in time):
>
> The burst is sent as fast as possible. The time it takes depend on the
> connection's performance (speed, delay, and jitter), as well as the
> state of Icecast and the operating system and their configuration.
>
> E.g. if you have a burst of 128kByte (= 1024kBit = 1MBit), a connection
> speed of 10MBit/s, a delay of 64ms, an jitter of +/- 10ms, and a TCP
> Window of 64kByte a initial estimation would be:
>
> TIME_Burst = (1MBit / 10Mit/s) + 3*64ms +/- 4*10ms
> Connection speed delay for SYN-ACK, Jitter matching
> ACK, and final ACK delays + one for
> data jitter
> -> 100ms + 192ms +/- 40ms
> -> 292ms +/- 40ms (252ms .. 332ms)
>
> This ignores the internal operation of Icecast and the operating system
> (such as effects of multitasking).
>
>
> After the burst the actual data transmission begins. Let's call that
> TIME_Streaming, and BYTES_Streaming.
>
> After the streaming the client disconnects (we ignore "being kicked by
> the admin" here):
>
> Depending on whether the client sends a RST or times out (by stalling
> the connection; we also ignore the buffer state of the client as we are
> interested in the bitrate, not the playback time):
>
> TIME_Termination = 1*delay +/- 1*jitter + (0 .. ~Queue_Size/~bitrate)
> BYTES_Termination = 0 .. ~Queue_Size
>
> (Please note that we are only using approximations here for the reasons
> already discussed.)
>
> Let us use the Example of Queue_Size = 512kByte (= 4096
> kBit), bitrate = 112kbit/s:
>
> TIME_Termination = 64ms +/- 10ms + (0 .. (~ 4096 kBit / 112kbit/s)
> ->
> 64ms +/- 10ms + (0 .. ~36000ms)
> -> 54ms .. ~36074ms
> BYTES
> _Termination = 0 .. ~Queue_Size
> -> 0 .. 512kByte
>
>
> The access.log now logs the sum:
> TIME = TIME_Burst + TIME_Streaming + TIME_Termination +/- TIME_e,
> BYTES = BYTES_Burst + BYTES_Streaming + BYTES_Termination +/- BYTES_e.
>
> However the TIME in access.log has a limited resolution of 1 second.
> Which means that it can not be better than +/- 0.5s as it can not
> provide "steps in between". In Addition there is clock-
> desynchronisation from the operating system (basically the clocks from
> all the cores tick a little bit differently) adding another +/-0.5s,
> and finally there are rounding errors adding another +/-0.5s.
>
> So TIME_e = 3*+/-0.5s = +/-1.5s.
> However in reality it is normally close to: ~TIME_e = -1s .. 0.
>
> BYTES_e is basically 0 as this is just a count. Most real world effects
> are already covered above.
>
> Let us put our example together:
>
> TIME = (292ms +/- 40ms) + TIME_Streaming + 54ms .. ~36074ms +/- 1500ms,
> TIME_Burst TIME_Termination TIME_e,
>
> -> 346ms + TIME_Streaming + (-1486ms .. 37614 ms)
>
>
> BYTES = 128kByte + BYTES_Streaming + (0 .. 512kByte) +/- 0
> BYTES_Burst BYTES_Termination BYTES_e
>
>
> BITRATE = BYTES / TIME
>
> 128kByte + BYTES_Streaming + (0 .. 512kByte) +/- 0
> = --------------------------------------------------
> 346ms + TIME_Streaming + (-1486ms .. 37614 ms)
>
>
> Let us calculate this for a TIME_Streaming = 5s and for
> TIME_Streaming = 500s with a bitrate of 112kBit/s (as used above):
>
>
> 128kByte + (5s * 112kBit/s) + (0 .. 512kByte) +/- 0
> BITRATE_5s = ---------------------------------------------------
> 346ms + 5s + (-1486ms .. 37614 ms)
>
> 202752 Byte + (0 .. 524288 Byte)
> = --------------------------------
> 5346 ms + (-1486ms .. 37614 ms)
>
> -> 36.87kBit/s .. 1471.50kBit/s
> -> (36.87kBit/s .. 1471.50kBit/s)/2 = 754.19kBit/s
> Expected: 112kBit/s
>
> and:
>
> 128kByte + (500s * 112kBit/s) + (0 .. 512kByte) +/- 0
> BITRATE_500s = -----------------------------------------------------
> 346ms + 500s + (-1486ms .. 37614 ms)
>
> 7299072 Byte + (0 .. 524288 Byte)
> = ---------------------------------
> 500346 ms + (-1486ms .. 37614 ms)
>
> -> 106.00kbit/s .. 122.51 kBit/s
> -> (106.00kbit/s .. 122.51 kBit/s)/2 = 114.26kBit/s
> Expected: 112kBit/s
>
>
> Conclusion:
>
> As clearly can be seen the shorter the amount of time for which is
> measured the bigger the likely error is. This depends on both effects
> of the real system as well as that calculations in form of
> lim t -> 0 x/t tend to amplify nose.
>
> The values from above can also be enhanced by estimating how much of
> the burst has been sent to a given client and subtracting that on both
> sides of the fraction. However this also comes with the downside of
> being only a estimation and therefore is mostly of statistical value.
> However it is expected to work well for connections with a connection
> time still relatively low so that the burst has an important impact but
> also long enough that other effects can be ignored (or otherwise
> compensated).
>
>
> If anyone finds errors in my calculations, feel free to point them out
> (on or off list).
>
>
> As always, in the hope that this was useful and did not cause more
> confusion.
>
>
> With best regards,
>
> > On Tue, Dec 21, 2021 at 5:40 PM Philipp Schafft <
> > phschafft at de.loewenfelsen.net> wrote:
> > > Good evening,
> > >
> > > On Tue, 2021-12-21 at 14:33 -0800, Milton Huang wrote:
> > > > Philipp,
> > > >
> > > > Thank you for your detailed and enlightening answers.
> > >
> > > you're very welcome.
> > >
> > >
> > > > Just to make sure I understand how 'bitrate' works here. Assuming
> > > we
> > > > are just working with audio to keep it simple. Let's also ignore
> > > > metadata,
> > >
> > > Ok. Just a word of warning to the general audience: This is a very
> > > limited view on the problem and might not apply to real deployments
> > > or
> > > fail to be valid at random.
> > >
> > >
> > > > as my use case is that I am generating an electronic music stream
> > > > from ffmpeg and broadcasting that using Icecast, so I can
> > > > control/calculate that.
> > >
> > >
> > > > I presume that if we specify `audio_bitrate` for a particular
> > > mount,
> > > > that is a 'target' rate that Icecast will attempt to transmit
> > > audio
> > > > data at.
> > >
> > > No, see below.
> > >
> > >
> > > > You mentioned that the actual rate can vary. Am I right to assume
> > > > this is because it is dependent on the source (ffmpeg) rate for
> > > > filling the 'queue', which in turn is used to fill each buffer of
> > > > each connected client?
> > >
> > > No. The bitrate depends on the codec, the encoder parameters, and
> > > the
> > > audio. E.g. bitrate can drop to virtually (or actually) zero for
> > > moments of silence as there just is no entropy to encode.
> > >
> > >
> > > > If so, wouldn't it generally be best for the mount
> > > `audio_bitrate` to
> > > > be set to the same bitrate that the source is generating/sending
> > > at?
> > >
> > > It is generally recommended to set as few options on the Icecast
> > > side
> > > as possible. Also see below.
> > >
> > >
> > > > I have the option of generating and compressing my mp3 stream at
> > > > either a fixed bitrate or to use a LAME VBR setting, and from
> > > what I
> > > > am now understanding it seems that using a fixed bitrate could be
> > > > better to avoid leading or lagging the queue buffer.
> > >
> > > Generally encoding with fixed bitrate tends to harm the quality of
> > > the
> > > signal. How much error is introduced depends again on the codec,
> > > the
> > > encoder parameters, and the signal.
> > >
> > > A quality based encoding is normally best as this results in the
> > > encoder trying to keep the signal on the same quality level.
> > > Bitrate is
> > > approximately a function of entropy*quality. The more constant you
> > > try
> > > to keep it the more quality of the signal will change with changes
> > > in
> > > entropy. Change of the amount of entropy per time in a signal is a
> > > very
> > > common event. E.g. you have a radical change between voice and
> > > music,
> > > but also between different kinds of music/compositions (fade-
> > > in/intro/fade-out/outro, instrumental-only, instruments+voice,
> > > simpler/more complex parts of a track, ...).
> > >
> > > With flexible transports such as IP based networks I see little use
> > > of
> > > constant or semi-constant bitrate modes. The main applications that
> > > comes to mind are fixed bitrate transports channels such as radio
> > > channels (e.g. GSM slots).
> > >
> > >
> > > Back to Icecast:
> > > Ignoring bursts and format specific headers, as well as syncing for
> > > a
> > > moment (which is exactly what happens once a listener is fully
> > > connected) Icecast sends data out as soon as it receives it.
> > > Icecast does not implement any bitrate control. There are no delays
> > > (which would be the only way for Icecast to do this, as it can not
> > > send
> > > data before it received it). All bitrate related options are
> > > cosmetic
> > > only (e.g. for user friendly display in directory services).
> > >
> > > Please note that this holds true for all official Icecast versions.
> > > Forks or custom versions may differ here.
>
>
> --
> Philipp Schafft (CEO/Geschäftsführer)
> Telephon: +49.3535 490 17 92
>
> Löwenfelsen UG (haftungsbeschränkt) Registration number:
> Bickinger Straße 21 HRB 12308 CB
> 04916 Herzberg (Elster) VATIN/USt-ID:
> Germany DE305133015
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xiph.org/pipermail/icecast/attachments/20211229/14843e7b/attachment.htm>
More information about the Icecast
mailing list