Qwertyuiopia

A brief overview of where and how things are stored

In the previous article in this series we have explored why having a digital music library with a consistent structure could be something you like. In this one we examine some technical aspects of storing digital music, namely a few of the file formats it can be stored in, how meta-information is stored and some basics on text encodings and file systems.

This should be able to help you decide which formats and options are best suited for you, as well as avoid a few common pitfalls when deciding on your tagger’s configuration.

Digital audio formats

Audio formats (or audio codecs) can be divided into two big categories. Those are Lossy and Lossless, depending on wether the codec discards audio information to reduce the filesize or not.

Lossy formats are the most common way to store music, since they reduce the filesize significantly making it possible to store music in logical amounts of disk space. Lossy formats are called this way because in order to compress the sound they discard information, such as sounds too high or too low for the ear to hear. Because the information discarded by each encoding is different, reencoding a file from one lossy format to another is problematic, resulting in serious loss of quality. Some of the most common lossy formats are these:

• Mp3: Mpeg Layer 3 is the most commonly used audio format, primarily because it is supported by all sorts of digital audio players. Apart from this advantage, it is the oldest of all the compressed formats we use for digital music, and its quality and compression ratio are worse than all the others.
• Wma: Windows Media Audio is microsoft’s compressed audio codec. Wma is an improvement over the original mp3 encoding and widely supported by portable players. On the downside, the format can contain Digital Rights Managment, which is a copyright expiration date thingy that may cause the files to stop working. Furthermore, tagging it in non-windows systems is a bit quirky at best, and many people opt not to use wma files when they can get their music in another file format.
• Ogg: Ogg Vorbis is an open source audio codec, also one generation newer than mp3. It is better than wma in audio quality, but lacks a little bit in support on portable playere (so you might want to check if a player supports it before you buy one). Being open source is a very good thing for an audio codec, since this enables developers of all kinds of software and hardware to use it for free, thus making it certain that you can find a tagger that can manipulate it properly. Barring the case that you already have a portable player that doesn’t support it, this is definitely the format to use when encoding from uncompressed audio.

Lossless formats keep all the information of the original sound. Extreme audiophiles sometimes like to collect their music in this form, though the files tend to be large enough to make having such a collection quite costly. Since there are no differences in quality between lossless files and their original (digital) source, though, there are no problems with re-encoding from one format to another. Two lossless formats you may have come across are:

• WAV: Waveform audio is Microsoft and IBM’s format for storing music on PCs. It takes up lots of space and it’s primarily used in audio editing, where you don’t want compression slowing this up. If you are keeping albums in this format you should probably convert them to FLAC or (assuming they’re not your mastertapes :-p) one of the lossy formats.
• FLAC: The Free Lossless Audio Codec is the open source lossless format. In addition, it has compression that reduces the filesize without discarding any information of the original file. If you are extreme about your sound quality requirements it is probably the format you’ll be using, although most people would find that it takes too much extra space for that extra bit of quality.

Id3 and other tags

Apart from the audio data itself, digital music files also contain a section of tags with information about each file, which we call metadata. These tags have various fields that can contain information ranging from an artist’s name to pictures of the album’s cover.

Different audio formats keep metadata in different ways. Thus, to consider an audio format fully supported by your system, you need a player that can both play the file and read its tags, and a tagger capable of manipulating them correctly (a subject which we will examine in the next article).

Mp3 files use a tagging format called ID3. It’s been around for a pretty long time, so when you’re working with it it’s best to make sure that your tagger supports the last version (currently id3v2.4). Id3v1 tags are kept separately from id3v2, and any decent tagger probably has an option to strip them completely and use v2. Id3v1 doesn’t support internationalization nor very long fields (so you might end with a few song or album names cut short), so before you start tagging you should make sure you’re using at least v2.3 (and ideally 2.4).

Most other file formats can use Id3 tags, but also have their own metadata formats. FLAC and Ogg use a format called “Vorbis Comment”, and it’s generally bad practice to tag them with id3 as it can break the container structure, effectively losing your information. Wma uses a microsoft format common to wma and wmv, though its later versions also support id3 (Although I wouldn’t expect many people that encode in wma to have an up-to-date version of the codec).

Text encodings

Another thing to be careful of is the text encoding that is used to write to the tags.

Initially, text in computers was stored according to the ISO standard, using Code Pages that contained 256 characters: The latin alphabet in uppercase and lowercase, the special characters, and one other language, with each language having its specific codepage(s).

Currently the format used in text is Unicode, which has a repertoire of 10000 characters, eliminating the need to use seperate code pages for different languages.

Especially if you have (or want to be open to having) music titles in crazy languages such as Greek or Bulgarian or anything with lots of accents, you do not want to use an ISO- codepage when you can use Unicode (UTF-8 or UTF-16).

However, certain pieces of very old software or hardware may not support UTF, so you might want to check if you have any problems with your equipment before converting everything. If you’re only using a computer to listen to music it’s definitely what you should be using.

Filesystem limitations

A file system is the method a device uses for storing and retrieving data. For our purposes, we care to go over some of the properties of the most common filesystems used in hard disks and CD/DVDs. The features that mostly concern us are the maximum length of file names, the character sets that can be used in those filenames, and the characters that can’t be used in filenames.

Depending on the operating system you use, your hard drive partition(s) are -probably- using one of the filesystems below:

• NTFS: Microsoft’s New Technology File System is what you ‘ll mostly be using if your computer runs Windows XP, Vista, NT or any version after Windows 2000. The filesystem supports paths up to about 32,000 Unicode characters with each path component (directory or filename) up to 255 characters long, and can display any character except ‘\0‘ (NULL) and ‘/‘ (slash). The Win32 Subsystem also excludes the use of \ (backslash): (colon)* (asterisk) ? (Question mark) “ (quote)< (less than)> (greater than) and | (pipe).
• ext3: The third extended filesystem is the most common of the unix filesystems, which you are probably using if you re on Linux. It can display any character except ‘\0‘ (NULL) and ‘/‘ (slash) and have a maximum filename length of 255 bytes. 255 characters is also the Linux VFS limit. ReiserFS and XFS are other common unix filesystems, with effectively the same limitations in character names.
• FAT32: File Allocation Table is the previous Microsoft filesystem. Dos used FAT16, and computers with Windows 98 (and up to Windows ME) use FAT32. Generally, there is reason to use it only on old computers that have to run Win 98. Its latest version can support 255 character Unicode filenames, however FAT partitions over 64 GB (even though theoretically possible) will not work well with Windows 98/ME. If you are using a computer too old to run any newer version of windows and still want to maintain a media library, you should probably upgrade to linux.

Also, CDs and DVDs can use two different standards when being written to. There is the newer Universal Disk Format, and the older ISO 9669. The ISO standard has a lot more restrictions than hard disk filesystems (i.e. maximum filenames of about 180 characters), and should probably be avoided.

When choosing how your library’s filename structure will look like (a subject which we’ll look into in a later article), you’ll probably want to go for compatibility, conforming with every (common) filesystem’s limitations. So, even if you are using a unix filesystem it’s better not to use the characters that windows doesn’t support, since it might make copying your library (or part of it) to windows systems rather messy. Your tagger probably has an option to do that for you.

And that’s with the basic stuff. In the next article (third in the series) we talk about taggers.

This entry was posted on Thursday, December 6th, 2007 at 12:41 and is filed under Tutorials. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.