dvd-hq.info Forum Index dvd-hq.info
DVD & video forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Major colour banding on large areas of subtle tone

 
Post new topic   Reply to topic    dvd-hq.info Forum Index -> Compression
View previous topic :: View next topic  
Author Message
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Wed 30 Apr 2008, 4:56    Post subject: Major colour banding on large areas of subtle tone Reply with quote

I am a visual artist who among other things creates abstract digital animations inspired by clouds. The work consists of very slow moving subtle abstract shapes that are constantly morphing. I was using MainConcepts mpeg2-dvd (Adobe Premier Pro 2) but found it very poor in that it washed out my images (I think it only allows CCIR601 output) DVDHQ.info and TMPGEnc have been a god send, now I have excellent contrast and saturation. However I am still experiencing major colour banding in large areas of subtle tone (see image below, actual size)

This banding is very pronounced when the dvd is viewed on a 720i lcd tv. I don't know if I am expecting too much of mpeg2. I would be eternally grateful if anyone could offer advice as to reducing colour banding. My animations are about 8 mins in length so size is not an issue and render times don't bother me.

Here are the settings I used in TMPGEnc 2.5

-PAL 16:9
- CBR, 8 Mb
- DC Component - 10bit
- Motion search high quality
- GOP, I-1, P-4, B-2
- Detect scene change
- Quantize Matrix, CG/Animation, Output basic YCbCr


_________________
Scott A Taylor
visual artist

www.scotttaylor.info


Last edited by Scott A Taylor on Wed 7 May 2008, 1:59; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Fri 2 May 2008, 6:13    Post subject: Reply with quote

Why are you using the CG matrix? The "Default" TMPGEnc matrix should be significantly better at preserving low frequencies (i.e., smooth colour transitions). The "CG" matrix is mainly for drawings and animations with hard edges (think anime).

A quantize matrix is something like this (only 8x8):

X X X X
X X X X
X X X X
X X X X

Where lower numbers represent more faithful preservation of frequencies and higher numbers represent higher quanization (less steps). The numbers near the top left corner apply to lower frequencies (smooth variations), the numbers near the bottom right corner apply to higher frequencies (hard edges, small details and noise).

There's a (simplified) explanation of how it works in my article about lossy compression.

So what you need for this kind of animation is a matrix where the numbers grow from the top left to the bottom right corner. Of the standard TMPGEnc matrices, "Default" is the best one. You can try tweaking it some more (ex., setting the topmost-leftmost 9 coefficients of both matrices to 8), but it'll probably give you good results as it is.

Also, you can probably use a longer GOP (ex., I=1, P=7, B=1). Even if there are any sudden changes, scene detection will take care of them.

And I suspect the "high quality" motion search is a bit of a waste (motion estimation should produce virtually identical results in less time).

Also, note that some decoders are better than others, and many LCD TVs use 6-bit panels (in other words, at best you'll get 64 intermediate tones, not 256, like in a proper 8-bit panel).

To judge what's really on the DVD, play it back using a good player (of the mainstream brands, Pioneer is probably the best, in my experience), connected to a CRT TV or to an LCD panel that you are sure is 8-bit or better (preferably an IPS or VA panel, not a TN panel, which is the most common - and usually cheaper - type).

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Fri 2 May 2008, 8:35    Post subject: Reply with quote

Great, thanks.

I'll admit to not trying the default matrix setting because of its title, 'default' The CG setting gave better results than the Mpeg standard.

Because every section of the animation is constantly morphing I thought less intermediate frames i.e. P and B frames, would result in a higher quality result. In saying that I did try I frames only but didn't see a noticeable improvement.

Render times don't bother me hence the high quality motion search.

I'll give your suggestions a go and get back to you as to the results, I might even try manually configuring the matrix.

Thanks again.

Cheers.
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Fri 2 May 2008, 19:19    Post subject: Reply with quote

It's odd that the "CG" matrix worked better than the "MPEG Standard". It should be significantly better at preserving hard edges, but worse at smooth colour transitions. Then again, the "MPEG Standard" matrix only has a frequency bias for the I-pictures (it preserves all frequencies equally to the P- and B-picture residuals), so for complex motion it won't be very different from the CG. The "Default" matrix should be noticeably better at dealing with the "smooth gradient" + "random motion" combination, as long as there are no small details (like thin, high-contrast lines).

Using I-pictures only will generally require a higher bitrate (over 20 Mb/s) to achieve good quality (depends on the detail in the original images and the matrices, too). At 8 Mb/s, using I-pictures only can make motion slightly more regular, but that comes at the cost of more (JPEG-style) "blockiness" per frame.

The scene change detection option should insert I-pictures if and when it determines that one would achieve better quality than a P-picture, so a longer GOP is generally the way to go. If the "morphing" is fast, then go with P=14 and B=0, instead of P=7 and B=1.

B-pictures will save some space (which will improve the P and I, indirectly) but, if there are a lot of changes, they will make the motion search less efficient, which in turn will make quality worse. By using P-pictures only (plus the starting I, of course), motion search should be able to get better results, which then requires less bits to "clean up". You'll have to experiment a bit to determine the best settings, but P=7 B=1 should be close to ideal.

But, most of all, make sure the banding isn't being caused by the player or LCD panel. LCDs have great geometry, but 99% of the models out there have far worse colour reproduction than even an average CRT.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Mon 5 May 2008, 7:03    Post subject: Better Reply with quote

I have tried several combinations of what you suggested and have settled on increasing the bitrate to 8.9Mb, GOP 1xI, 2xP and using the CG Matrix changing the top 9 leftmost values to 8 in both fields (still found CG slightly better than the default). I do get a bit of blockiness with the low amount of P frames but I find it breaks up the colour banding a bit, of which there is still a fair amount. The smoothness of the motion is perfect. I think I am expecting too much of dvd so may rethink my animations to compensate at least until I can upgrade to HD DVD or Bluray.

Something I was wondering was the fact I use TMPGEnc to encode the bmp sequence of frames directly and each frame (all 12000) are 1.43mb in size. Would shrinking the frame size improve encoding quality since less compression is needed? If so can you recommend an image format? jpeg and png seem to degrade the quality a bit too much. I did try creating an uncompressed avi with Premier but found it decreased the quality slightly.

Thanks again Rui for your prompt professional advice.
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 7 May 2008, 0:58    Post subject: Reply with quote

The size (in bytes) of the original files has no influence on quality. Regardless of their format, the frames always need to be decoded (in other words, converted into uncompressed bitmaps) before any program can do anything to them (such as create a video file from them or apply filters, etc.).

Make sure the original frames are the correct dimensions (in pixels), though (720x576 or 704x576, assuming you're using PAL), otherwise TMPGEnc has to resize every single frame, which takes a long time and can cause some loss of quality.

PNG is a lossless format, so it won't make any change to the quality (pixels keep exactly the same values). The uncompressed AVI should also be, pixel-for-pixel, identical to the original frames (assuming there wasn't any further processing). The same goes (for example) for an AVI in a format like HuffYUV, which is also lossless.

JPEG, M-JPEG, DV or MPEG are lossy formats (will alter the image).

But there's nothing to be gained by converting to another format, so don't bother. Just make sure the resolution is the right one. And try the result in more than one player / TV.

If you want to store a high-quality version in as little space as possible, the best compression you can get without quality loss is probably with the HuffYUV AVI codec. If you don't mind a small loss, XviD, x264 and DV are probably your best options in terms of quality and "future proofing".

When you say using shorter GOPs "breaks up" the banding, do you mean banding tends to increase through a GOP (i.e., starts smooth, then gets more quantised)? If so, I suspect that's a rounding off error in TMPGEnc. I remember seeing that problem a long time ago, then it seemed to get fixed in a newer version. Maybe it still happens with some types of images. You can try to work around it like this:

Instead of CBR, select CQ (Constant quality). In the "setting", set the "quality" to 100, set the maximum bitrate as high as possible (9800 minus the audio), set the minimum to 2000 or higher, and set both "spoilage" values to -100. Try this with a long GOP (ex., I=1, P=14) and see if the problem disappears.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Wed 7 May 2008, 1:58    Post subject: Reply with quote

Yeah I am using PAL and the original frames are 720 x 576. I realise PNG is supposed to be lossless but when converting from bmp to png via Photoshop CS3 there is a slight increase in colour banding as there was with Premier's uncompressed avi format. I am not sure what is happening because as you say there shouldn't be quality loss. Maybe I have colour space conflicts. Anyway I'll stick to bmp if file size doesn't effect encoding. By the way bmp is pretty good for storage as zipping reduces bmp file size to a third of the original. I do prefer to keep the individual frames as opposed to a video for the master.

As you mentioned in a previous post shorter GOP's can create jpeg like blockiness, which it did for me. It is this subtle blockiness I find that breaks up the banding a little ie the bands don't have such a sharp edge.

I am happy with this result, so much better than my attempts before receiving your advice (most improvement is with the motion)

Thank you Rui for all your time
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 7 May 2008, 19:24    Post subject: Reply with quote

Scott A Taylor wrote:
I realise PNG is supposed to be lossless but when converting from bmp to png via Photoshop CS3 there is a slight increase in colour banding as there was with Premier's uncompressed avi format.


Photoshop up to CS2 did have a problem with PNG files, but only when they included transparency (it was related to alpha matting, one of my pet hates in the CG world). But CS3 seems to have fixed that. Your problem might be related to colour management.

Quote:

Maybe I have colour space conflicts.


Ah, yes, if you have colour management enabled in Photoshop, you can have issues. Unless you're doing something for print it's generally best to keep colour management turned off (you can still use a profile for your monitor, in Windows' display properties, that won't affect what's actually saved in the files, but Photoshop's built-in colour management will).

Colour management, and colour space covnersion for the screen only pays off if you're working with more than 8 bits per channel. At 8 bpc it can cause noticeable banding.

Quote:

Anyway I'll stick to bmp if file size doesn't effect encoding. By the way bmp is pretty good for storage as zipping reduces bmp file size to a third of the original.


Well, that shows how inefficient the BMP format is. Smile You'll get a bigger reduction (meaning smaller file size) by using PNG, I expect (PNG uses a smarter compression algorithm, optimised for graphics; ZIP is more generic). Using RAR instead of ZIP will probably also give you slightly better compression.

Quote:

I do prefer to keep the individual frames as opposed to a video for the master.


Yes, it makes corrections to individual frames much easier. By the way, most animators use TGA. The size will be approximately the same as BMP, but virtually every program can load TGAs, and a few can't load BMPs. TGA can use RLE compression (not as efficient as PNG or ZIP) directly, which is also lossless. And you can apply ZIP / RAR / etc. (called LZ-style compression) on top of RLE. PNG will still give you the smallest file size, for most types of images. There's a comparison here:

http://www.dvd-hq.info/data_compression_1.php#Comparison

Quote:

As you mentioned in a previous post shorter GOP's can create jpeg like blockiness, which it did for me.


At low bitrates, yes.

Quote:

It is this subtle blockiness I find that breaks up the banding a little ie the bands don't have such a sharp edge.


But you get a bit of a "grid overlay", caused by the fact that JPEG compression operates on square (8x8 or 16x16 pixel) blocks. I guess it's a matter of picking between two evils.

But do try CQ (or VBR) with the spoilage values set to -100, and a long GOP. Theoretically that means TMPGEnc will try to use the residual part of the P-pictures (that's the data added after the motion vectors, to correct mistakes) to improve the quality, in relation to the I-pictures. That might get rid of the banding, if indeed it's being caused by a rounding error.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Fri 9 May 2008, 7:56    Post subject: Reply with quote

Turns out Photoshop colour space was the issue with my png’s. Thanks.
When I said zip I meant it as a generic term sorry, I do use rar. I am aware of Targa being the animator’s choice but Window’s picture and fax viewer doesn’t show them. I like to be able to quickly see an image without opening Photoshop or the like. Bmp isn’t much bigger as you say. I will stick with bmp especially considering the little colour space issue I had, I have never had an issue with bmp creating or viewing.
I have tried your suggestion and have posted frames below from the original footage, Premier’s default mpeg2-dvd codec (MainConcept) and 2 different settings using TMPGEnc.



This is the bmp output by Premier of the final edit



Mainconcept - 9.8 CBR, GOP M-3 N-12



TMPGEnc – 9.8 CBR, MP@ML, Highest Motion, I-1 P-2, Detect Scene, CG Matrix (top left 9 changed to 8 in both), Basic YCbCr



TMPGEnc – 9.8 CQ (Quality 100, Max 9800, Min 6000, Spoilage both -100) , MP@ML, Motion Estimate, I-1 P-14, Detect Scene, Default Matrix (top left 9 changed to 8 in both), Basic YCbCr

I am happiest with the first TMPGE frame. Notice the slightly better contrast and saturation which seems to be because of the CG Matrix.

I think we have tried almost everything.
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Sun 11 May 2008, 20:36    Post subject: Reply with quote

I've used your first frame to make a few tests, and I found that there is no change in the banding even when creating a HP@HL 11-bit MPEG-2 file, which is odd. At 11 bits per DC coefficient there should be enough margin to preserve pretty much any 8-bit RGB gradient.

Other tests with "strange" matrices suggest that there is some limit being hit, beyond wich the extra precision is not used either for encoding or (I'm starting to suspect) for decoding. In other words, MPEG-2 decoders simply don't bother to go beyond a certain number of intermediate steps (ex., maybe they always use 8 bits, because using 10 would effectively double the decoding time, due to the way most CPUs work).

If I have a chance, I'll try a few less common MPEG-2 decoders to see if there's any difference.

You can "break up" the banding by adding a bit of luminance noise to the original, but then it looks a bit dirty (and the banding is still there, just less regular).

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Tue 13 May 2008, 8:32    Post subject: Reply with quote

I truly appreciate all the time you are spending on my problem. Thanks.
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Tue 13 May 2008, 23:41    Post subject: Reply with quote

Without much in the way of useful results, though. Smile I've tried other decoders and the same thing happens, although the exact banding pattern varies slightly (which might indicate that the issue is indeed with decoders cutting some corners - but some cut different corners from others).

And the same thing happens with MPEG-4 ASP, by the way, even with quantisation set to 1 and no limit on the bitrate. I haven't tried MPEG-4 AVC yet, but if it behaves the same way, using BluRay / HD-DVD won't make any difference.

Lossless codecs (like HuffYUV) preserve all the shades, naturally.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Scott A Taylor



Joined: 29 Apr 2008
Posts: 7
Location: Australia

PostPosted: Wed 14 May 2008, 0:18    Post subject: Reply with quote

Looks like I will have to rethink my animations, in saying that I am happier with the results we ended up with as opposed to my original solo attempts.

Technical restrictions can push the creative process into new directions so I'll wait to see where this leads me.

Thanks once again Rui and congratulations on a fantastic website.

All the best.
_________________
Scott A Taylor
visual artist

www.scotttaylor.info
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 14 May 2008, 17:53    Post subject: Reply with quote

Do you make these for sale (meaning you need a format that's easy to distribute) or for exhibitions where you have more control over the format and playback equipment?

By the way, I had a similar problem (temporal, not spatial) ages ago with some animations that included very slow cross-fades (several minutes long). Regular video editing software couldn't make them smooth enough, you always noticed little "jumps" every few seconds. In the end I had to do the cross-fades in 3D software, that renders internally with floating-point (32 bits per channel) precision and then dithers down to 8-bit (24 per pixel). The result was that different pixels effectively faded at different times, and it looked smooth.

But that's a more extreme (and rarer) case than a smooth gradient so I'm a bit surprised that MPEG encoders (or, possibly, decoders) have this problem.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    dvd-hq.info Forum Index -> Compression All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group