<div dir="ltr">Man, that was awesome. I think that formula should be put in the doc pages somewhere, as it really clarifies what that code is doing.<div><br></div><div>Thanks for all the details!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 24, 2021 at 4:45 AM Philipp Schafft <<a href="mailto:phschafft@de.loewenfelsen.net">phschafft@de.loewenfelsen.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Good morning,<br>
<br>
On Wed, 2021-12-22 at 20:28 -0800, Milton Huang wrote:<br>
> Wow. This has been helpful to me to really understand what is going<br>
> on. I'd suspected the 'audio_bitrate' was just for show.<br>
<br>
happy to hear. :)<br>
<br>
<br>
> So, to again make sure I'm understanding, I assume I can get the<br>
> actual (average) transfer rate of data to a particular user for a<br>
> particular session by taking the access.log BYTES = total bytes of<br>
> data sent (other than headers) and dividing by TIME = seconds that<br>
> the connection lasted.<br>
<br>
Generally this is the way to go. However with the limit of BYTES and<br>
TIME being big enough. See below.<br>
<br>
<br>
> This varies depending on many of the factors you mention,<br>
> particularly the amount of information in the signal. And for those<br>
> reasons, it can be misleading if measured in a very short window.<br>
<br>
Correct.<br>
<br>
<br>
> I just looked at one of my logs and saw a huge spread for the 1<br>
> second duration... Don't know how to post pictures though...<br>
<br>
<br>
That is right. There are a number of factors here that need to be kept<br>
in mind for very short connections:<br>
<br>
First of all Icecast sends the burst (of wich we know the estimated<br>
size from the configuration, but again: exact size is unknown to us as<br>
it depends on the signal at given point in time):<br>
<br>
The burst is sent as fast as possible. The time it takes depend on the<br>
connection's performance (speed, delay, and jitter), as well as the<br>
state of Icecast and the operating system and their configuration.<br>
<br>
E.g. if you have a burst of 128kByte (= 1024kBit = 1MBit), a connection<br>
speed of 10MBit/s, a delay of 64ms, an jitter of +/- 10ms, and a TCP<br>
Window of 64kByte a initial estimation would be:<br>
<br>
TIME_Burst = (1MBit / 10Mit/s) + 3*64ms +/- 4*10ms<br>
Connection speed delay for SYN-ACK, Jitter matching<br>
ACK, and final ACK delays + one for<br>
data jitter<br>
-> 100ms + 192ms +/- 40ms<br>
-> 292ms +/- 40ms (252ms .. 332ms)<br>
<br>
This ignores the internal operation of Icecast and the operating system<br>
(such as effects of multitasking).<br>
<br>
<br>
After the burst the actual data transmission begins. Let's call that<br>
TIME_Streaming, and BYTES_Streaming.<br>
<br>
After the streaming the client disconnects (we ignore "being kicked by<br>
the admin" here):<br>
<br>
Depending on whether the client sends a RST or times out (by stalling<br>
the connection; we also ignore the buffer state of the client as we are<br>
interested in the bitrate, not the playback time):<br>
<br>
TIME_Termination = 1*delay +/- 1*jitter + (0 .. ~Queue_Size/~bitrate)<br>
BYTES_Termination = 0 .. ~Queue_Size<br>
<br>
(Please note that we are only using approximations here for the reasons<br>
already discussed.)<br>
<br>
Let us use the Example of Queue_Size = 512kByte (= 4096<br>
kBit), bitrate = 112kbit/s:<br>
<br>
TIME_Termination = 64ms +/- 10ms + (0 .. (~ 4096 kBit / 112kbit/s)<br>
-> <br>
64ms +/- 10ms + (0 .. ~36000ms)<br>
-> 54ms .. ~36074ms<br>
BYTES<br>
_Termination = 0 .. ~Queue_Size<br>
-> 0 .. 512kByte<br>
<br>
<br>
The access.log now logs the sum:<br>
TIME = TIME_Burst + TIME_Streaming + TIME_Termination +/- TIME_e,<br>
BYTES = BYTES_Burst + BYTES_Streaming + BYTES_Termination +/- BYTES_e.<br>
<br>
However the TIME in access.log has a limited resolution of 1 second.<br>
Which means that it can not be better than +/- 0.5s as it can not<br>
provide "steps in between". In Addition there is clock-<br>
desynchronisation from the operating system (basically the clocks from<br>
all the cores tick a little bit differently) adding another +/-0.5s,<br>
and finally there are rounding errors adding another +/-0.5s.<br>
<br>
So TIME_e = 3*+/-0.5s = +/-1.5s.<br>
However in reality it is normally close to: ~TIME_e = -1s .. 0.<br>
<br>
BYTES_e is basically 0 as this is just a count. Most real world effects<br>
are already covered above.<br>
<br>
Let us put our example together:<br>
<br>
TIME = (292ms +/- 40ms) + TIME_Streaming + 54ms .. ~36074ms +/- 1500ms,<br>
TIME_Burst TIME_Termination TIME_e,<br>
<br>
-> 346ms + TIME_Streaming + (-1486ms .. 37614 ms)<br>
<br>
<br>
BYTES = 128kByte + BYTES_Streaming + (0 .. 512kByte) +/- 0<br>
BYTES_Burst BYTES_Termination BYTES_e<br>
<br>
<br>
BITRATE = BYTES / TIME<br>
<br>
128kByte + BYTES_Streaming + (0 .. 512kByte) +/- 0<br>
= --------------------------------------------------<br>
346ms + TIME_Streaming + (-1486ms .. 37614 ms)<br>
<br>
<br>
Let us calculate this for a TIME_Streaming = 5s and for<br>
TIME_Streaming = 500s with a bitrate of 112kBit/s (as used above):<br>
<br>
<br>
128kByte + (5s * 112kBit/s) + (0 .. 512kByte) +/- 0<br>
BITRATE_5s = ---------------------------------------------------<br>
346ms + 5s + (-1486ms .. 37614 ms)<br>
<br>
202752 Byte + (0 .. 524288 Byte)<br>
= --------------------------------<br>
5346 ms + (-1486ms .. 37614 ms)<br>
<br>
-> 36.87kBit/s .. 1471.50kBit/s<br>
-> (36.87kBit/s .. 1471.50kBit/s)/2 = 754.19kBit/s<br>
Expected: 112kBit/s<br>
<br>
and:<br>
<br>
128kByte + (500s * 112kBit/s) + (0 .. 512kByte) +/- 0<br>
BITRATE_500s = -----------------------------------------------------<br>
346ms + 500s + (-1486ms .. 37614 ms)<br>
<br>
7299072 Byte + (0 .. 524288 Byte)<br>
= ---------------------------------<br>
500346 ms + (-1486ms .. 37614 ms)<br>
<br>
-> 106.00kbit/s .. 122.51 kBit/s<br>
-> (106.00kbit/s .. 122.51 kBit/s)/2 = 114.26kBit/s<br>
Expected: 112kBit/s<br>
<br>
<br>
Conclusion:<br>
<br>
As clearly can be seen the shorter the amount of time for which is<br>
measured the bigger the likely error is. This depends on both effects<br>
of the real system as well as that calculations in form of<br>
lim t -> 0 x/t tend to amplify nose.<br>
<br>
The values from above can also be enhanced by estimating how much of<br>
the burst has been sent to a given client and subtracting that on both<br>
sides of the fraction. However this also comes with the downside of<br>
being only a estimation and therefore is mostly of statistical value.<br>
However it is expected to work well for connections with a connection<br>
time still relatively low so that the burst has an important impact but<br>
also long enough that other effects can be ignored (or otherwise<br>
compensated).<br>
<br>
<br>
If anyone finds errors in my calculations, feel free to point them out<br>
(on or off list).<br>
<br>
<br>
As always, in the hope that this was useful and did not cause more<br>
confusion.<br>
<br>
<br>
With best regards,<br>
<br>
> On Tue, Dec 21, 2021 at 5:40 PM Philipp Schafft <<br>
> <a href="mailto:phschafft@de.loewenfelsen.net" target="_blank">phschafft@de.loewenfelsen.net</a>> wrote:<br>
> > Good evening,<br>
> > <br>
> > On Tue, 2021-12-21 at 14:33 -0800, Milton Huang wrote:<br>
> > > Philipp,<br>
> > > <br>
> > > Thank you for your detailed and enlightening answers.<br>
> > <br>
> > you're very welcome.<br>
> > <br>
> > <br>
> > > Just to make sure I understand how 'bitrate' works here. Assuming<br>
> > we<br>
> > > are just working with audio to keep it simple. Let's also ignore<br>
> > > metadata,<br>
> > <br>
> > Ok. Just a word of warning to the general audience: This is a very<br>
> > limited view on the problem and might not apply to real deployments<br>
> > or<br>
> > fail to be valid at random.<br>
> > <br>
> > <br>
> > > as my use case is that I am generating an electronic music stream<br>
> > > from ffmpeg and broadcasting that using Icecast, so I can<br>
> > > control/calculate that.<br>
> > <br>
> > <br>
> > > I presume that if we specify `audio_bitrate` for a particular<br>
> > mount,<br>
> > > that is a 'target' rate that Icecast will attempt to transmit<br>
> > audio<br>
> > > data at.<br>
> > <br>
> > No, see below.<br>
> > <br>
> > <br>
> > > You mentioned that the actual rate can vary. Am I right to assume<br>
> > > this is because it is dependent on the source (ffmpeg) rate for<br>
> > > filling the 'queue', which in turn is used to fill each buffer of<br>
> > > each connected client?<br>
> > <br>
> > No. The bitrate depends on the codec, the encoder parameters, and<br>
> > the<br>
> > audio. E.g. bitrate can drop to virtually (or actually) zero for<br>
> > moments of silence as there just is no entropy to encode.<br>
> > <br>
> > <br>
> > > If so, wouldn't it generally be best for the mount<br>
> > `audio_bitrate` to<br>
> > > be set to the same bitrate that the source is generating/sending<br>
> > at?<br>
> > <br>
> > It is generally recommended to set as few options on the Icecast<br>
> > side<br>
> > as possible. Also see below.<br>
> > <br>
> > <br>
> > > I have the option of generating and compressing my mp3 stream at<br>
> > > either a fixed bitrate or to use a LAME VBR setting, and from<br>
> > what I<br>
> > > am now understanding it seems that using a fixed bitrate could be<br>
> > > better to avoid leading or lagging the queue buffer.<br>
> > <br>
> > Generally encoding with fixed bitrate tends to harm the quality of<br>
> > the<br>
> > signal. How much error is introduced depends again on the codec,<br>
> > the<br>
> > encoder parameters, and the signal.<br>
> > <br>
> > A quality based encoding is normally best as this results in the<br>
> > encoder trying to keep the signal on the same quality level.<br>
> > Bitrate is<br>
> > approximately a function of entropy*quality. The more constant you<br>
> > try<br>
> > to keep it the more quality of the signal will change with changes<br>
> > in<br>
> > entropy. Change of the amount of entropy per time in a signal is a<br>
> > very<br>
> > common event. E.g. you have a radical change between voice and<br>
> > music,<br>
> > but also between different kinds of music/compositions (fade-<br>
> > in/intro/fade-out/outro, instrumental-only, instruments+voice,<br>
> > simpler/more complex parts of a track, ...).<br>
> > <br>
> > With flexible transports such as IP based networks I see little use<br>
> > of<br>
> > constant or semi-constant bitrate modes. The main applications that<br>
> > comes to mind are fixed bitrate transports channels such as radio<br>
> > channels (e.g. GSM slots).<br>
> > <br>
> > <br>
> > Back to Icecast:<br>
> > Ignoring bursts and format specific headers, as well as syncing for<br>
> > a<br>
> > moment (which is exactly what happens once a listener is fully<br>
> > connected) Icecast sends data out as soon as it receives it.<br>
> > Icecast does not implement any bitrate control. There are no delays<br>
> > (which would be the only way for Icecast to do this, as it can not<br>
> > send<br>
> > data before it received it). All bitrate related options are<br>
> > cosmetic<br>
> > only (e.g. for user friendly display in directory services).<br>
> > <br>
> > Please note that this holds true for all official Icecast versions.<br>
> > Forks or custom versions may differ here.<br>
<br>
<br>
-- <br>
Philipp Schafft (CEO/Geschäftsführer) <br>
Telephon: +49.3535 490 17 92<br>
<br>
Löwenfelsen UG (haftungsbeschränkt) Registration number:<br>
Bickinger Straße 21 HRB 12308 CB<br>
04916 Herzberg (Elster) VATIN/USt-ID:<br>
Germany DE305133015<br>
_______________________________________________<br>
Icecast mailing list<br>
<a href="mailto:Icecast@xiph.org" target="_blank">Icecast@xiph.org</a><br>
<a href="http://lists.xiph.org/mailman/listinfo/icecast" rel="noreferrer" target="_blank">http://lists.xiph.org/mailman/listinfo/icecast</a><br>
</blockquote></div>