[flac-dev] Two questions

[Brian Willoughby]

On Aug 2, 2021, at 19:19, Federico Miyara <fmiyara at fceia.unr.edu.ar> wrote:
> Brian,
> 
> Thanks for your reply.
> 
>> You should not use the RIFF information. It is not part of the FLAC specification. It is an optional enhancement to store information that is only about the input WAV files. It will strictly be missing from FLAC files that are recorded live, converted from AIFF, and will even be missing when WAV is the source if the option is not added.
> 
> I'm aware of that. It happens that my flac files (there are many of them) have been converted from wav files and contain riff cues and associated text that have been annotated, many years ago, according to a research protocol. I need to retrieve that information automatically. The encoding has been done preserving foreign metadata and tests have shown the information is correctly kept. I have a script that can retrieve the info from the wav file, so I decode a dummy wav with very few samples and the whole metadata stuff, but it would be better to retrieve that information directly from the file.

I recommend writing your own utility based on the FLAC library, in C, with the features you want. I do not recall any feature in the flac command line utility that would allow this. Your workaround is a reasonable attempt, but it seems to have too many undefined side-effects.

> I've read the FLAC format and cannot find any mention to where are the foreign metadata included in the stream. Is it possible that it isn't actually documented yet?

The --keep-foreign-metadata feature was added to the command-line application after the FLAC format was finalized. The metadata ends up in an APPLICATION block, which is usually skipped by the FLAC library decoder. These are intended for third-party applications, and thus it's typically impossible to document them. Normally, a third-party software developer would add their own proprietary block to the FLAC file, and all other applications would just skip over it (because all blocks have a universal name and length at the start).

This foreign metadata feature is a special case, where the command-line flac utility uses 'RIFF', 'riff' or 'aiff' as "application" names, when actually it's the external file format and not the application that's being referred to.

https://xiph.org/flac/format.html#def_APPLICATION

and

https://xiph.org/flac/id.html

The only documentation of the APPLICATION block format is probably the source code for the flac command line utility. I did not design this, but I remember suggesting it a few times. Basically, the entire WAV or AIFF contents are in the block, verbatim, except for the chunk that would contain the audio. Since the FLAC data outside the APPLICATION block already contains the audio, that chunk is empty in the APPLICATION block.

By the way, one of the challenges of making a completely lossless WAV or AIFF compressor is that there is no predefined order for the various chunks in those files. The audio data chunk can appear before or after various other optional chunks. The solution for FLAC was to have that empty chunk inside the APPLICATION block. For WAV, the audio chunk is named 'data' and for AIFF the audio chunk is named 'SSND'. All other chunks are copied verbatim, but these audio chunks only have a name and size with no further bytes. It's basically a marker. I'm pretty sure that's how it was implemented, but you can check the flac command line source to confirm.


>> Are you seeing 3 bytes for 1 sample? ... or are you seeing 3 samples? Also, I recall that the FLAC library returns 32-bit numbers, so you have to convert these to 16-bit or 24-bit samples.
> 
> I think it returns exactly the sample type the original file contained, otherwise I guess it wouldn't be a lossless compressor.

There are two level to the FLAC source code. At the lowest level is the FLAC library, which deals only with the FLAC stream, either seekable or restricted to streaming only. The FLAC library does not understand WAV or AIFF or anything besides FLAC. The high level code is separate, and it's the flac command line.

So, yes, the flac command line returns the same sample type as the original file.

However, if you use the FLAC library directly, you'll note that samples are always 32-bit fixed-point integers. I've written my own object-oriented framework to convert FLAC to WAV, AIFF, CAF, and other formats. In this code base, I had to deal with the 32-bit integers. My apologies for confusing the FLAC library with the output files from the flac command line.

If you're going to use the command line, I'd recommend getting or writing some utilities that can analyze a WAV (or AIFF) file directly. Seems like some of the GUI applications out there can do unexpected things (for a long time, certain GUI apps would show MP3 song metadata in the audio samples!)

> However, I made a more careful test and with skip=0 until=1 and get 2 samples instead of 3. 

I have used these options for very long recordings where the FLAC file was up to 4.0 GB in size, but the output WAV or AIFF would exceed 4.0 GB. I had to use --until and --skip to chop up the recording and then piece them together later in the DAW. I never had trouble with repeated or missing samples, but perhaps there are bugs related to asking for exactly zero or one sample.

I'm usually paranoid about missing or duplicating samples, so I don't think there's a bug with very large values for skip or until. I'm fairly certain that I depended on documentation such that until would not include the sample, but skip would. i.e. I probably calculated a split point and used something like the following:

flac -d --force-aiff --until=N foo.flac
mv foo.aiff foo1.aiff
flac -d --force-aiff --skip=N foo.flac
mv foo.aiff foo2.aiff

... and then used Logic Studio Pro to splice the two files together again.

If you want to contribute to FLAC, you might try a few variations with skip != 0 and until much greater than 1. Then you can check the size of multiple test cases and see if there is a pattern. Maybe it only breaks with really small numbers, or when one value is zero.

If you find a pattern that indicates a bug, even just with the documentation, then please report it. Maybe you can fix it, or maybe someone else can.

Brian Willoughby