Video Processing and Colorspaces (why is My Lossless Lossy?)

RGB Colorspace - The Easy Stuff
Not understanding video colorspaces has caused me to lose days worth of effort because all of my video processing was for naught. We'll start from the screen format of red, green, and blue (RGB) and work into the other ones. Video monitors and televisions use red, green, and blue mixed together to create all of the colors that they display. Each color is called a channel. So RGB has three color channels. The easiest way that this is represented is that in each pixel, the red, green, and blue are each represented by one byte. In this format, each byte is a numerical representation of the "brightness" of that color. If all the bytes for a pixel are zero, then the pixel is black. If they are all at their maximum value, the pixel is white. If red and blue are non-zero but green is zero, you have purple and so on. Since each byte is 8 bits, each color has 8 bits in one pixel. This format would be called RGB24. A related format called RGB32 is the same 8 bits per pixel, but discards the last byte as extra data. It is easier for computers to deal with 4 bytes at a time than 3, so it is useful to have extra data that is discarded. RGBA32 is a slightly different format where instead of discarding the last byte, the last byte is used for a transparency (or "alpha") channel (remember a channel is just a way to talk about colors individually), that specifies how transparent the pixel is. This is useful when laying one image on top of another. Take a purple pixel with an alpha value that makes it halfway transparent so when this pixel is "overlaid" on another image, the purple will only partially mix with the pixel underneath, and the pixel underneath will still be partially visible. This technique is used for overlaying images such as for displaying television channel logos over a television show.

YUV Colorspace - What is That?
Ok, now to the complicated part. The problem is that I always understood RGB24, etc, and I thought that all video codecs and outputs used this style format. This caused me a lot of problems with video processing as I found that many video formats and codecs use different colorspaces that I had never heard of. Every time I processed a video, I ended up converting it to a different colorspace and losing image data or causing inconsistencies or extra data. This was very frustrating as I didn't understand what was happening or what was causing it to happen. Another common video format is called YUV. This is COMPLETELY DIFFERENT than RGB. Somebody much smarter than I am learned a better way to compress and display video information. The YUV format uses Y as a brightness channel, and then U and V as color channels. The U channel is is a color channel that is a bluish-purplish-greenish and the V channel is a greenish-reddish. This is how analog televisions worked. Black and white televisions only had a Y (brightness) channel. When color television was developed, U and V channels were added as additional signals so that black and white televisions and color televisions could use the same broadcast signals. Black and white televisions discarded the U and V channels, and only displayed the Y channel. This is why on color TV's, when sometimes the signal was poor, the channel would come in but the color would go in and out. YUV is now a more popular format for storing video and images than RGB is.

YUV and Compression and Visual Tricks
Developers of video and formats and colorspaces have learned that the human eye is more sensitive to changes in brightness than it is to changes in color. So when developers were trying to figure out new ways to compress video, they learned that they could throw away 75% of the color information (U and V channels) in a picture or video, while keeping the original brightness (Y channel) with little perceived quality loss in the picture or video. This was great, they saved a lot of space, and still had good-looking media. But when processing and re-processing video, these conversions can be a problem. Let's take the example of a recorded screencast of some task performed on a computer. Most likely, this is recorded in RGB24 or RGB32 format, since that is what a computer's display is using. By just encoding and re-processing willy-nilly with any old video codec, the video could be inadvertently converted to a YV12 colorspace (YV12 is one particular YUV format) and lose 75% of the original color information! Now when making the final output video, this may be the desired result. But during the re-processing stage, this is not a good idea.

Put Colorspaces to Use
I use the Internet extensively when determining what colorspaces to use. The first step before reprocessing any video is to determine the original colorspace by looking up the codec. The individual formats of the colorspaces do matter beyond whether the format is RGB or YUV. The online reference http://www.fourcc.org has details on many of these formats. It describes how much information is kept and what may be discarded. When it says a "2x2 subsampled plane", it means that color channel compresses 4 pixels worth of information (a square 2x2 block) into a single pixel. For the purposes in processing video, it is probably OK to treat "color channels" and "color planes" the same so don't worry about the terminology. Some formats may throw away only half of the color information, some may throw away 3/4 of it. To truly keep video in lossless format, make sure that the target colorspace in the target codec is the same as the colorspace used in the codec from the source video. Some lossless formats are only available in one or the other colorspace, and this causes losses as the conversion is performed back and forth.

Video Display and Outputting
Watch out that video playback software does not mislead. Sometimes, even though the display uses RGB to display pictures and video, the video playback software does not. Some video playback software (especially on a linux-type system), uses YUV as an intermediate format before it gets to the display and is converted back to RGB. So be careful that what you see is not always what you get.