Alpha matting & premultiplication

Anyone vaguely familiar with digital image editing is likely to know what an "alpha channel" is. Although the term can be used to describe any bitmap channel other than the standard colour ones (R, G and B, or C, M, Y and K), it usually refers to the opacity channel. I used the term opacity and not transparency because, typically, the higher the value of an alpha channel sample, the more opaque that pixel is (some file formats work the other way around, though, so calling it the transparency channel is fine). The alpha channel is typically represented by the letter A (ex., "RGBA").

But the term "alpha matting" is probably less familiar, and not very intuitive. In the context of digital image data formats, alpha matting does not refer to the use of alpha channels to create a "mask" or "matte"; it refers to a form of encoding where the value of the alpha channel influences the values of the colour channels. This encoding method is also known (in the specific case of a black matte) as "premultiplied alpha" (although it is actually the other channels that are premultiplied by the alpha).

Essentially, alpha matting mixes the colour of each pixel with a predefined "background colour" in a proportion defined by the alpha (opacity) channel. The more transparent the pixel, the higher the amount of the background colour. If the pixel is 0% opaque (100% transparent), its colour becomes identical to the background's.

Although any colour can be use for the matte, most software defaults to one of three main types of opacity encoding: straight alpha (where the colour and opacity values don't affect each other), black matte (where the colour values are premultiplied by the opacity value, "mixing" them with black) and white matte (where the values are mixed with white, in a sort of "reverse premultiplication").

The following image illustrates the data that is actually stored in the RGB and A channels using each type of encoding. The alpha channel is represented as a grayscale image where white means opaque and black means transparent. The text is 100% opaque, the blue rectangle is 25% opaque, and the pixels in the antialiased edges have many different opacity levels:

As you can see, there is no difference in the alpha channel itself; the difference is only in the RGB channels.

The advantages of alpha matting are as follows:

1. If a program loads a bitmap with premultiplied alpha and sends only the RGB values to the screen (ignoring the alpha values), the image will look right.

2. Er... that's about it.

That's pretty much the only situation in which alpha premultiplication has any advantage: some vague compatibility with software that ignores the alpha channel. Which, as you can imagine, isn't really a common problem with modern compositing software. Instead, alpha matting creates all sorts of compatibility problems and quality issues. I'll start with the latter.

Let's say we have two pixels, whose 8-bit RGB values are [ 255 , 128 , 10 ] and [ 253 , 129 , 14 ], respectively. In our paint (or photo editing) program, we give them an opacity value of 25 (corresponding to 10% opacity, on an 8-bit scale, that goes from 0 to 255). If our program works with premultiplied alpha, the actual RGBA values saved will be [ 25 , 13 , 1 , 25 ] for the first pixel and [ 25 , 13 , 1 , 25 ] for the second pixel. Noticed anything? Two pixels that started out with slightly different colours are now exactly identical. We have just lost the ability to distinguish between them based on colour (ex., with a "magic wand" tool). If we later decide to increase the opacity of these pixels back to 100% (255), our software will change both to [ 250 , 130 , 10 , 255 ]. Noticed anything else? The RGB values don't actually match either of the original pixels (and are still identical to each other, of course).

Let's see what would have happened if we had used "straight" alpha, with no premultiplication or matting. The original pixel RGB values are [ 255 , 128 , 10 ] and [ 253 , 129 , 14 ], respectively. When we add the alpha channel, they are stored (to memory, or to a file on the disk) as [ 255 , 128 , 10 , 25 ] and [ 253 , 129 , 14 , 25 ]. Still perfectly distinguishable. And if we later decide to increase the opacity value to 255, we get [ 255 , 128 , 10 , 255 ] and [ 253 , 129 , 14 , 255 ]. That means less processing (only the alpha value needs to be changed) and no loss of information or quality.

Of course, normal computer monitors can't display transparent pixels; the software (or the graphics processor) must still calculate the resulting colour of each pixel before putting it on the screen. And since most graphics cards work with 8 bits per channel, the data sent to the display will be exactly like the premultiplied example above: [ 25 , 13 , 1 ] for the first pixel and [ 25 , 13 , 1 ] for the second pixel. In other words, the pixels on the screen will still be identical. Using straight alpha won't magically give us more intermediate tones on the monitor. But it will preserve all the information about the original colours, and it will let the software distinguish (internally) between pixels based on their original colour, even if their "composited" (that is, final display) colour is identical.

This means that, if a single "matting" (or premultiplication) is done at the end of image processing, there will be no noticeable quality loss. Buf if the image is premultiplied one or more times at intermediate stages of the production process (ex., every time the file is saved), that will lead to a loss of information, which in turn will lead to a progressive loss of accuracy (affecting visual quality) in subsequent processing steps.

Compatibility problems arise when a file saved with one type of alpha channel (ex., white matte) is loaded into and used by a program that expects a different type of encoding (ex., black matte). You don't get an error message; instead the program will load the data, assume it's in the format it understands, and treat it as such. The result is dark (or discoloured) fringes around object outlines, and strange colour shifts in areas with intermediate opacity levels.

Let's look at what happens when there is a mismatch between the encoding of the source file and the way the software processes the file. Here I will use only the most common types of matte (things get even worse when coloured mattes are used):

Note that the issue here isn't which one looks "nicer". Some people might find the outline in the black-as-white sample aesthetically pleasing. And some people might think the shifted colours in the white-as-black have a really cool "eighties' video clip" look. The point is that, given the original image (with orange-yellow text, a light blue rectangle, and no outlines), you only get the correct result if the processing mode matches the way the original image was encoded.

For most file formats there is no direct way to determine the type of matting used by a given image (i.e., that information is not written to the file headers or anywhere else), so it's frequently up to the user to pick the right option. Since the edge and colour shift problems aren't always obvious in the small preview windows of video editing software (and many people seem to think an external video monitor is unnecessary...), it's not uncommon to render a complete project only to find, at the end, that some title or logo used the wrong kind of matte, and seems to have been generated by a VHS mixer from 1980. Occasional incorrect matting artifacts can even be found in some "big budget" TV series and ads.

From what I've written above, I hope it's pretty clear that you should, whenever possible, work with straight (unmatted, non-premultiplied) alpha. Unfortunately, "whenever possible" doesn't necessarily resolve to "always".

So how can you ensure that you pick the right kind of matte? Well, as a user, there are basically two things that you can do. First, whenever you load an image with an alpha channel into your editing or compositing program, try all types of alpha channel import options, render at least one frame at full size, and compare the results. Upon close examination, the one with the "right" type of matte setting should look significantly better than the others. The other thing is remember what type of matte each program uses when saving its files. Some programs give you an option, others won't. For example, Autodesk's 3dsmax has a checkbox in some file export dialogs (ex., TGA) to enable alpha channel premultiplication. If the checkbox is cleared, the frames will be saved with straight alpha. If the box is checked, the colours will be mixed with the background colour (which can be black or white or any other colour). In fact, it can even be a bitmap, which means each pixel in your rendered frames will be mixed with a different colour (good luck superimposing those frames on a new background!). Due to an error in the way the matte is calculated, however, premultiplication only works correctly with a black background, so this ability to select the background colour and enable premultiplication separately is useless (stick to black). Also, for most file formats, 3dsmax can only correctly interpret images (to use as textures, for example) if these have straight alpha or a black matte. It also cannot correctly import interlaced footage, but that's another issue.

Adobe Photoshop always saves the files with a white alpha matte (or at least it did up to "Photoshop CS", it's not clear if that has changed in the latest versions). Until recently, it could only correctly load files with straight alpha (unless they were in its own PSD format, in which case it correctly interpreted the white matte). New "CS" versions have the option to "remove white matte" or "remove black matte" from individual layers. It still cannot deal with any other matte colour.

If you're trying to transfer files between a program that can only save files with one type of alpha encoding (ex., Photoshop) and a program that cannot handle that type of encoding properly (ex., 3dsmax), you are up the proverbial creek without the proverbial paddle. The most practical solution (after you've fired a few angry calls or e-mails at the software developers in question) is to convert all the files using a program that can understand both formats (or at least import one and export the other).

This section is aimed mainly at software developers, but regular users should read it too, because it shows how simple it is to deal with these issues from a programming point of view.

It might sound a bit self-righteous to write an article telling Photoshop's or 3dsmax's programmers how to do their job but... well, someone has to, because it's pretty obvious they aren't going to get it right by themselves (both are up to version 9, and are still lacking basic alpha channel conversion abilities).

First, here are the formulas to convert between matted and non-matted alpha. They can deal with any matte colour and are easy to implement for any colour depth and in any language. The variables used in the formulas are:

Cm = Component (matted)
Cu = Component (unmatted)
Cb = Component (background / matte colour)
a = Alpha
amax = Maximum possible alpha value (ex., 1.0)

All values are assumed to vary between 0.0 and 1.0 (for 8-bit files this would be 0-255 and for 16-bit files it would be 0-65535; remember to divide by the scale if you use anything other than 0.0-1.0).

To convert from unmatted alpha to matted alpha:

Cm = ( Cu ∙ a ) + [ Cb ∙ ( amax - a ) ]

To convert from matted alpha to straight (unmatted) alpha:

If a = 0, then Cu = Cm
Else, Cu = [ Cm - Cb ∙ ( amax - a ) ] / a

The formulas should be applied once to each colour component (ex.: R, G and B) of each pixel.

Those formulas are enough to add support for alpha channel conversion (with any matte colour) to any application. The software can then either cache a "working format" version of the image (generally meaning a straight-alpha or black-matte version) or it can perform the conversion whenever it needs to read a pixel's colour channel values. There is really no excuse for not including this basic ability in a professional-level application.

The second issue is how to determine the kind of alpha channel used by a file that the user has just imported. One solution is to simply ask the user, but that's a pretty poor solution; users have more important things to worry about. So let's look at how software can determine the type of alpha channel and, if it turns out to be a matted alpha channel, how it can determine the background colour.

First we look for a pixel that is 100% transparent (0% opaque). If one exists, store its RGB values. Look for another pixel that is 100% transparent. Compare its RGB values with the ones previously stored. Repeat until the end of the file. If all pixels with 0% opacity have the same RGB values, it's almost guaranteed that that colour is the one used for the matte (if indeed there is a matte). If the colour varies, there's a good chance that we're looking at a straight-alpha file, or that the file uses some sort of really weird encoding (like the background bitmap example given above).

Now, even if all 100% transparent pixels have exactly the same colour, the file might still use "straight" alpha, so it's time to take the "background" RGB values we found above, plug them into the matted / unmatted conversion formulas and see if they make sense.

No pixel can have less of the matte colour in its RGB values than its alpha value would allow. For example, a colour channel in a pixel that is only 10% opaque cannot deviate more than 10% from the same channel of the matte colour.

If this condition isn't met by every pixel in the file, then there is no matte. If it is met by every pixel in the file, there's a very good chance that there is a matte, using the background colour we determined earlier.

There are other ways to make these decisions, some of which are simpler, faster, and will also get it right most of the time (for example, the program might analyse only a few pixels, rather than every pixel in the image).

Software should perform these tests when a file is first imported, and use the results to pre-fill the source format (or "footage interpretation") settings. This means that, in 99% of cases, the user will only need to click "ok".

If the tests are inconclusive (ex., because there are no transparent pixels in the file), then the software should ask the user to make the choice.

In any case, the user should always be given the option to toggle the type of alpha channel (between straight and matted), to pick the matte colour (manually or from the image) or to have the software "guess" the alpha channel type and matte colour again. These options should be available at any moment; not just when the file is first imported. If the software uses working proxies, a change of source format settings should cause the file to be re-imported.

Adobe Photoshop and Autodesk 3dsmax are, unfortunately, not the only programs with less-than-perfect (or indeed, given their price, less-than-acceptable) handling of alpha mattes. In fact, few programs get this essential aspect of compositing right.

Early versions of Premiere Pro had bugs that made it impossible to composite layers saved with a white matte (such as the ones present in Adobe's own PSD format). Versions after 2.0 have a "Remove Matte" effect, which works like the one found in Photoshop CS. This effect is not located in a very logical place, though (it's under "Keying", but the alpha matte type is a property of the source file itself, not a form of keying).

In older versions of Premiere (version 6.5 and below) the alpha matte type was picked in the transparency compositing mode (which makes somewhat more sense), but the options were still limited to white matte, black matte and straight alpha, and there was no automated "guessing" of encoding type.

Sony Vegas and 3dsmax give users only the option between straight and premultiplied alpha (meaning a black alpha matte).

Eyeon Fusion seems to work internally with a premultiplied (black matte) format, and only gives the option to "postmultiply by alpha" (which does the opposite of "remove black matte") in its file import panel. It does not let the user select a white or coloured matte.

As for Adobe After Effects... well, after mentioning that both Adobe Photoshop and Adobe Premiere Pro have significant limitations, you might expect the same to apply to After Effects. But, as it turns out, After Effects probably has the best alpha matte handling of any software package. Not only can it handle straight alpha, white mattes and black mattes, but it can also handle coloured mattes (any colour), it can try to guess the matte type and colour or let the user pick it manually, it allows all these settings to be changed at any moment and it stores them in a logical place (the "Interpret Footage" property sheet).

Thank you for restoring my faith in the human species, After Effects crew. Maybe you could give some pointers to the Photoshop and Premiere teams.

Adobe isn't alone in having poor internal communication. Despite 3dsmax's issues, Autodesk Combustion also gets it right. Controls very similar to After Effects' can be found under "Footage Controls" (there are some interface issues, though, as Combustion won't let you pick a matte colour directly from the image).

It's hard to explain why programs developed by the same company deal with these issues in such different ways. It's as if, instead of communicating and sharing the best solutions, the development teams keep trying to reinvent the wheel (and often ending up with something square).

I hope this article helped to shed some light on a somewhat obscure but critical aspect of digital compositing. I suspect the deficiencies of many software packages in this area are due to the fact that the people writing the software don't actually use it, and the people using it don't know enough about software architecture to figure out what, exactly, their editing or animation program is doing wrong.

According to Murphy's Paradox, "the optimist believes we live in the best of all possible worlds; the pessimist fears that might be true". In that vein, I will end this article with the following (true) story:

I was once a beta-tester for a company that developed video editing software. They had some serious issues with the correct interpretation of alpha mattes (their software assumed that all files used straight alpha). Although it wasn't part of my job, I wrote some code for them (to convert any matte type to straight alpha). They thanked me, tested the code, confirmed it did what it was supposed to do, and, two months later, released a new version of their software... which still could only correctly handle straight alpha.