HDTV Magazine
Welcome, Anonymous  •  Sign In  •  Register  •  Help


Product line up and pricing

  • DVD: HQV Benchmark Version 1.4 NTSC $20
  • DVD: HQV Benchmark Version 1.4a PAL $20 (not reviewed)
  • Blu-ray: HQV Benchmark Version 1.0 $20
  • HD DVD: HQV Benchmark Version 1.0 $10
  • Bundle 1: Blu-ray, DVD (NTSC) $30
  • Bundle 2: HD DVD, DVD (NTSC) $25
  • Bundle 3: Blu-ray, DVD (PAL) $30 (not reviewed)
  • Bundle 4: HD DVD, DVD (PAL) $25 (not reviewed)

Summary: A very useful testing regimen if you understand the limitations

Have you ever wondered why special features on DVD don't look as good as the movie you watched? Maybe you've wondered why some of your DVDs don't look as good as others? Or why Blu-ray Hollywood movies appear to have more detail than concerts or documentaries? HQV Benchmark has a testing regimen to help you figure all of that out!

HQV Benchmark is produced by Silicon Optix, a leading developer of video processing technology. If you are new to the terms video processing and scaling, then a great foundational start is our own Video Dictionary on HD Library, Scaler.

The HQV Benchmark series of discs have received a lot of press and a lot of players have failed to pass many of its tests. Let's take a look at what each of these tests are, what their purpose is and what it actually means to your overall experience.


Testing, Scoring, Education

The test material concentrates mostly on deinterlacing and scaling. Silicon Optix provides a downloadable PDF file of the test regimen which includes how to score each test along with detailed explanations. While the guide provides comparison images, the resolution is not high enough for many of the tests to assist you in fully appreciating what to look for.

HQV Benchmark DVD Testing and Scoring Guide
HQV Benchmark Blu-ray and HD DVD Testing and Scoring Guide

Many of the DVD test results and all of the Blu-ray and HD DVD test results rely on perception requiring proper calibration of the display for valid results and is noted by * in the title. Neither disc provides a full suite of calibration test patterns. Digital Video Essentials is recommended and available in Blu-ray and DVD. There is also an HD DVD and DVD combo available while supplies last. If your system has been ISF calibrated, you are ready for testing. If not, check the ISF Forum for a professional ISF calibrator in your area.


HQV Benchmark DVD

The test material is a mix of 4:3 and 16:9 original aspect ratio (OAR) content, which will require you to manually change the aspect because there are no flags to trigger the auto 4:3/16:9 aspect feature that some players support. If the auto aspect does not provide a manual feature then you are stuck with 4:3, which will not be correct for some of the 16:9 tests. The introduction and test material does not provide reference imaging quality for showing off the DVD format at its best.

Color Bars (4:3)
This is a resolution test that is part of a multiple test pattern. This is not only one of the more useful patterns on the disc, but contains other tests such as color decoding at different saturation levels and luminance (video) levels. I recommend testing in both 16:9 and 4:3 aspects. Above the middle is a resolution response test for both luminance and chroma. Going left to right they are numbered as 4, 3, 2 and 1 with 1 representing 720 pixels horizontally. The catch is that this is a 4:4:4 encoded pattern which means the chroma has the same response as luminance. The consumer DVD standard (and HDTV) uses 4:2:0 encoding which cuts the chroma response to half of luminance to conserve bandwidth and storage space. The chroma response in block 1 serves no purpose since it cannot be properly reproduced so disregard those results. This pattern is part and parcel of confirming proper calibration prior to testing.

Jaggie 1 (16:9)
A single bar is constantly rotated inside a circle testing the high detail b/w video or luminance portion of an image.

Jaggie 2 (16:9)
Three bars move back and forth in a narrow arc, covering 50-20 degrees, inside a circle testing the high detail b/w video or luminance portion of an image.

Flag (4:3)
A final jaggie test of both color and luminance using the common and notoriously difficult American flag waving in the wind.

Detail (16:9)
The disc narrative and guide both stress that this material should "exhibit fine detail resulting in a crisp realistic image" and other similar statements. This material will never have the response one would expect or can get viewing properly captured and mastered DVD video. Page 10 of the guide compares two images that hardly look different for a test score high of 10 and 0, and in this case resembles reasonable expectations for this test.

Noise (4:3)
If the TV or player does not have a noise reduction (NR) feature, skip this test; it is not about players. MPEG NR targets compression noise, is processed differently and should not be used for this test. A series of 12 still images are provided and most target the blue color channel. Two of those images never showed any noise and one was marginal. These images are great examples of a noisy analog cable service or a satellite / cable set top box delivered via channel 3 or 4 to your TV. This test is all about the NTSC broadcast television system, analog cable, analog RF tuners and the RF noise that can easily come from them. While having little to do with players, they are useful for DVD recorders and broadcast NTSC. For high fidelity with DVD, NR on your DVD player should be turned off. For recorders this feature may make some or all of your noisy channels more palatable but in most cases the setting will also apply to DVDs, in which case it should be turned off.

One point missing from the guide; the first step is to view the content with NR turned off and that includes the TV if available. Look through or ignore the noise and recognize the detail that is present. Now turn on NR on the player and see how much noise is removed along with any loss in detail. You can also reverse the test, turn the NR off on the player and turn it on for your TV if available. Some NR circuits offer different range levels in which case test all of them and determine which setting provides the best balance of detail versus noise suppression.

Motion Adaptive Noise (16:9 and 4:3)
A 16:9 image of a roller coaster and a 4:3 image of a boat going down a river are provided. While an NR circuit can successfully navigate the prior noise test of still images, the addition of motion will show any artifacts created by the process and may help identify what kind of NR the video processor is using. Follow the same procedure for testing as previously described, following the guide for evaluation. The roller coaster is also a convenient test for LCD pixel speed provided you turn off NR on the player and display.

Film Detail (4:3)
In the guide, this test is called 3:2 Detection, which is a far better description of the test result. A familiar movie scene is provided of an F-1 car passing by empty bleachers testing the standard 3:2 cadence required with 24fps (frames per second) content.

Assorted Cadences (16:9)
A test clip is provided in 8 different cadences. This is the most brutal part of the testing regimen that few players or displays will pass. External scalers should pass most if not all of the following tests:

2-2 30fps film
2-2-2-4 DVCAM
2-3-3-2 DVCAM
3-2-3-2-2 VARI SPEED Broadcast
5-5 Anime
6-4 Anime
8-7 Anime
3-2 24fps film

Mixed 3:2 with titles (4:3)
4:3 images at 24fps are provided with the same kind of 30fps-based titles you would get from your broadcaster notifying you of weather alerts or emergencies. This is also delivered in the form of end credits at the end of a TV show. This applies far more to broadcast TV and your display rather than players.


HQV Benchmark DVD on Your Player

Just because a player fails some or all of these tests does not mean it will generate the same errors when playing a Hollywood movie on your player, nor would passing some of these tests qualify as high fidelity performance. None of the tests even relate to how the vast majority of movies are captured, processed and mastered for DVD along with how your player is designed to reproduce them. Missing from this disc is the same test material processed and mastered just like Hollywood does. Without such a reference point the person doing the evaluation may have unrealistic expectations of how well the test material should perform.

While it can be argued on the surface that some of these tests should apply to a player, as a reviewer I find myself in a catch 22. Test material from Avia, Video Essentials, Sound and Vision, Digital Video Essentials and nearly all popular movies look decent to fantastic although that same player fails all or some of the HQV Benchmark material. How can that be? And as reviewer, how do I report a passing or failing grade?

The key is understanding why an inexpensive DVD player can get decent results with no-name video processing. This is achieved during mastering by including progressive flags in the data directly from the mastering studio telling the video processor in the player how to take the interlaced fields and put them together for a proper 480p presentation. This is an extremely intelligent way to deliver a high fidelity performance envelope on the cheap! With a native 480p 16:9 display, typically CRT only, and a properly designed 480p player, you are in videophile nirvana due to this free ride but this is an HDTV world and most displays these days require the 480p free ride gets scaled to one of the HD scan rates, 720p, 1080i or 1080p. Proper deinterlacing is the crux and scaling is far easier so this free ride can provide decent to high quality performance beyond 480p depending on the design goals! There is a catch; no flags, no free ride and with incorrect flags, the free ride could turn bumpy with errors/artifacts.

The HQV Benchmark DVD technical twist is that the material is encoded as raw 480i, no progressive flags for dumb scaling, leaving the player entirely on its own to figure out how to deinterlace the content. The bottom line is that the vast majority of players are going to fail many of the tests that relate to DVD content. While the movie is bound to have these progressive flags, that may not be the case for special features. This has improved over the years, but for a movie buff and/or DVD collector much of a library is going to contain such content. On top of that, many a collection will have 4:3 letterboxed releases along with the oddball cadences that come with low volume or low budget productions, cult classics, anime and TV shows on DVD. Some folks desire a player or external scaler that can get the most out of such content. HQV Benchmark is exactly what the doctor ordered for reviewers and videophiles alike who are looking for a simple straight forward battery of tests to quickly determine performance with such content. The only test missing is for 4:3 letterboxed material.


HQV Benchmark Blu-ray / HD DVD

The introductory scenes of 16:9 video content are quite short, although in the other chapters about testing there is material of greater length. This content is worthy of overall image quality evaluation, but it does appear slightly soft in detail and flat in dynamic range compared to other reference material. Both discs provide the test materials in three different play formats; Play Loop, Play All Tests (manual advance) and Select Individual Tests. The first two automated play formats unfortunately skip one or more tests (noted in the test pattern breakdown) so if you want to view them all then choose Select Individual Tests.

HD Color Bars
This pattern is only available under Select Individual Tests and appears to be derived from a non HD source and scaled to 1080i based on the color pixel errors at the edges where two colors meet. Useful for checking / confirming luminance and color levels prior to testing.

HD Noise
Provides two images, a flower and sailing boat, which have motion components as well as noise. The Blu-ray version skips the sailing boat when choosing Play Loop. You may be hard pressed to even use this test since a noise reduction feature is not common for HD disc players or displays when viewing HD scan rates. Please check the DVD version of this test for additional details.

Video Resolution Loss
A SMPTE RP-133 1920X1080 1:1 pixel map of luminance, b/w video, has a 360 degree constantly rotating bar. The results as described in the guide require a 1920x1080 pixel matrix to test the internal scaler along with proper aspect setting of the display for 1:1 pixel mapping best confirmed prior to testing with the same pattern from Digital Video Essentials. This pattern has a black and white video level error and pixel mapping error. Black is about 10% above peak black and white is about 10% below peak white; this is not critical to the test. The pixel map is correct for the vertical plane, 1080 pixels, but incorrect for the horizontal plane, 1920; this is not critical for this test of 1080 vertical resolution but the 1920 burst for horizontal resolution will not show up appearing as a gray box. As a 1:1 pixel map pattern, the horizontal response should be ignored. This test applies to other pixel matrixes and 1080i content but you will likely have problems with the vertical 1080 burst and the response may vary in the five different areas of the image. The rotating bar will tell you if the 3:2 cadence is being properly detected and should show smooth motion as it spins.

Diagonal Filtering Jaggies Test
Provides the same two tests as the DVD. The first is the single bar constantly rotating and the second is where three bars move back and forth in a narrow arc. The Guide calls this Video Reconstruction Tests, does not document the second test and both disc versions require you choose Select Individual Tests for the second jaggie test to be viewed.

Film Resolution Loss
A SMPTE RP-133 1920X1080 1:1 pixel mapped test pattern is panned back and forth along with a horizontal pan of a football stadium. The Blu-ray version skips the stadium when choosing Play Loop. The test uses the same pattern as Video Resolution Loss containing the same errors and has the same requirements for testing. Unlike the previous test, this one can show horizontal response errors due to motion. As before, the 1920 box will be gray. Included in this pattern are video level boxes with percentage of modulation and you can see that the one pixel space between the two digits is missing implying that frequency response has been slightly extended. This means that the 1920, 960 and 480 boxes actually have slightly higher pixel counts which is why it does not pixel map and probably why the 1920 response is missing. This test still applies but due to this error it clearly is not the ultimate test it could be and who knows if a scaler might perform better with a true pixel map expecting 1920x1080. That said, the guide and disc only discuss errors in the vertical response.

The stadium test follows but lacks the snap of detail expected from quality capturing and 1:1 pixel mapping. It only pans right to left and one display that was tested with the SMPTE pattern had less horizontal response artifacts in that direction over the other. It's a shame the stadium was not panned in both directions like the SMPTE pattern. When the 50 yard line hits about the center of the screen there is a subtle pixel response change and a change of light output level in some areas of the scene. Two displays failed the SMPTE pattern test in clearly different ways for either the horizontal or vertical planes. I found this test one of the most difficult to analyze.

With either display the horizontal plane detail would drop in and out at what appeared to be the same points. The display that passed the prior test for vertical response simply had more detail while in motion and at the final still point of a few seconds. While the other display faired far better with the prior test of horizontal response, both appeared to respond equally. With a casual viewing you may be hard pressed to detect any appreciable difference in the stadium pan. If you concentrate on the horizontal response you will miss the point of this test. Concentrate on the vertical response or horizontal lines in the bleachers, light towers, field and elsewhere in the image. Those are the trees you are looking for in this forest and the horizontal response of vertical lines seems best left ignored.


HQV Benchmark Blu-ray / HD DVD on Your Player

As with the DVD version, missing from these discs is the same test material processed and mastered just like Hollywood does. A 1080p24 reference point with a proper 1080p24 display would show the viewer what correct performance looks like assisting the 1080i evaluation. It would be interesting to see how a player handles converting a Hollywood 1080p24 version of these specific test materials to the other scan rates.

Just like DVD, we have the same dilemma of how a player can fail these tests yet show not one sign of trouble with actual movies or calibration and test discs like Digital Video Essentials. The answer is the same; HD disc movies, for the most part, are not mastered as 1080i, they are mastered as 1080p24 and the player is designed to work with that when deriving other output scan rates.

Like DVD, special features still receive little to no attention and can vary from 480p all the way through 1080p24. I have even seen 480p 4:3 letterboxed content. Even though Blu-ray is an HD 16:9 destined format you just don't know what you are going to get. It doesn't end there; feature material that was captured as native 1080i is mastered as is. Most of those titles are concert, documentary features and television programming along with some indie movies with the common thread being that they were captured with HD 1080i based cameras. If the content is original 24 frame film then telecine mastering is performed at 1080p24 and that represents the majority of the Hollywood catalog.

Testing HD disc players is not nearly as straight forward as it was for DVD. Getting acceptable imaging out of lower resolutions (such as standard definition) is far more difficult. With DVD content you can run into some oddball content that will create very noticeable artifacts far more easily detected, even at far viewing distances. The nine cadence tests for DVD are a prime example and are not part of the HD version. With that in mind, the most critical HQV test for any HD disc viewer is proper cadence detection of the rotating bar of the Video Resolution Loss test which must pass with smooth motion and should be tested at 720p, 1080i and 1080p depending on what HD scan rates your display will accept. You may find that only 1080i provides the correct response. As for the other tests of 1080 horizontal lines testing vertical response, failure hardly means your image will stink but it does mean that 1080i content will have a loss in horizontal line detail. More than likely the failure will be due to vertical filtering which requires a deeper understanding of burst testing, what is going on and how it affects specific elements of an image. I refer you to HD Waveform - Vertical and Horizontal Filtering for detailed information.

For those with a display that only accepts a 1080p60 or 1080i output you can test the player's ability to convert Hollywood 1080p24 to either of those scan rates by using the SMTPE RP-133 from Digital Video Essentials for both Blu-ray and HD DVD. In this case the pattern is properly pixel mapped in both horizontal and vertical planes but it is a still pattern and cannot test for cadence or loss of resolution due to motion. For either scan rate setting you are looking for single pixel lines in the vertical 1080 box response and horizontal 1920 box response.


Vertical Filtering - Finding the Right Recipe

If you want every last shred of detail with 1080i content then vertical filtering is unacceptable. The easiest route to overcoming that is no recipe at all, which requires a player to properly deinterlace 1080i and convert it to 1080p60 or 1080p24 for a proper response along with advanced features that would allow you to output native 1080p24 content untouched. Most 1080p displays these days provide 1:1 pixel mapping at 1080p60 and many include 1080p24 as well. More importantly, the player must do this automatically providing a convenient hands-off and worry free approach for the user. While such a player would provide the ultimate keep it simple solution for any viewer, finding one that has been designed much less reviewed for this attribute may be far more difficult. For current Blu-ray players the right recipe of display and player can achieve the exact same results and external scaling provides yet another solution.

The discs were tested with a Sony PS3 and Toshiba HD-A35. The PS3 will not do a thing with native 1080i content except pass it along as is; testing 1080i conversion of the player to 1080p was impossible. With a full featured 1080p display the PS3 defaults to native output of the source. I did find it curious that the HQV Blu-ray menus are native 1080p24. The Toshiba on the other hand does not support native output and follows what you have set it for; 720p60, 1080p60 or 1080p24. Set for 720p or 1080p60 the Toshiba failed to pass the vertical 1080 box. Set for 1080p24 the Toshiba completely wiped out on cadence with every one of the tests creating a strobing effect at all times. Both players passed the Digital Video Essentials SMPTE RP-133 test at 1080i and 1080p60. A Panasonic PTAE-1000 and BenQ W10000 front projector were also tested. The Panasonic passed with flying colors for the most part while the BenQ failed due to vertical filtering.

The combination of the Sony PS3 and the Panasonic PTAE-1000 provides optimal results. Since the Sony defaults to native output of disc content and the Panasonic passes 1080i testing along with native 1080p60 (1080p 30 frame source) or 1080p24 support you are getting full automation for the best results. Either the internal scaler of the display is deinterlacing and scaling or is getting a direct feed bypassing it providing a 1:1 pixel map instead.

The Toshiba HD DVD player on the other hand does not have a native mode and simply does what you tell it to. If you know the HD DVD disc has native 1080i content then even with a display like the Panasonic you will have to change the output scan rate to match the disc source; same goes with the Toshiba feeding an external scaler. While lacking automation you can get optimal results manually.

The BenQ is a stellar performer and nearly a reference for video standards when pixel mapped at 1080p60 or 1080p24 yet 1080i is its Achilles performance Heal. Neither player tested provides a direct solution alone. One solution requires an external scaler with advanced features that will de-interlace and scale 1080i while allowing a 1080p60, 1080p30 or 1080p24 bypass. Mated with the PS3 you would achieve simple automation.

Untouched native 1080p24 content from player to a 1080p24 display is quite easy to acquire and will satisfy most users. In the end it all comes down to you, your system, how you use it (viewing distance) and the importance you place on some or all of the content you are viewing. Ultimately HQV Benchmark is limited in its ability to answer all of these questions if you want the most out of every bit of content you might be feeding your system since it is limited to 1080i testing only.


HQV Benchmark on Your Display

A properly designed 480i DVD player can be used to evaluate the internal scaler of a display or external scaler using the analog component inputs. Digital video, HDMI/DVI, is typically limited to 480p but if your display accepts 480i and your player can provide it without artifacts then that would be a valid test. Simply set the output of the player to 480i. Testing S-video and composite video connections is a bit more dicey because that requires proper down conversion so while likely not the ultimate reference test signal from your player there are still things that can be learned. For cable and satellite boxes using the analog component input of your display, these tests have direct value if you are setting the box to native so NTSC is output as 480i. Since you can't test the box it won't help you determine if the box or your display is doing a better job. Unfortunately, what this disc can't test is your NTSC TV tuner. Keep in mind during your testing how some displays apply, lock or limit different video processing features based on the input type.

A properly designed Blu-ray or HD DVD player can be used to evaluate how the internal scaler of a display or external scaler handles 1080i content using the analog component inputs or HDMI/DVI (with the player set for 1080i output).

Conclusion

While the some of the tests represent a small portion of the available catalog, the DVD version helps those seeking videophile nirvana with any and all DVD content on the planet via a player or external scaler. All of the tests are useful for determining how your display or external scaler handles broadcast NTSC video. For player evaluation, it should be considered a secondary test to the primary test of other discs that do follow Hollywood mastering representing the vast majority of what you would rent or purchase. The Blu-ray and HD DVD version provides a battery of tests to evaluate how your display or an external scaler handles HDTV 1080i content from Blu-ray disc or broadcast HDTV. If Blu-ray Hollywood movies are your main concern then just like DVD it should be considered a secondary test to the primary test of other discs that provide a native 1080p24 response representing the majority of Hollywood features. Either version of HQV Benchmark is unique in what it brings to the videophile table for display and player evaluation. With those limitations understood, the HQV Benchmark series is a unique evaluation tool that deserves a place in the videophiles toolbox.

Test Results of Past Products

Panasonic PT-AE1000U LCD Front Projector
BenQ W10000 DLP Front Projector
Toshiba HD-A3, HD-A30, HD-A35 HD DVD and SD DVD players
OPPO DV-981HD Upconverting SD DVD Player

Posted by Richard Fisher, June 19, 2008 12:24 PM

More from Richard Fisher

» - Currently Reading

More in Category: HDTV Accessories

About Richard Fisher

Richard Fisher is the President of Mastertech Repair Corporation, serving north east Atlanta, Georgia, and has been servicing, calibrating and reviewing audio video products since 1981. Tech Services USA, a division of Mastertech, creates sites, communities and libraries for consumers and professionals to share their technology knowledge and learn from each other. These include The ISF Forum and HD Library. HDTV Magazine exclusively publishes HD Library and Forum for Tech Services USA.

Richard is ISF and HAA certified providing calibration and A/V reproduction engineering services. Richard is a technical consultant and also provides performance ISF and HAA home theater systems and calibration via Custom HT. Mastertech Repair Corporation is a factory authorized service center for Hitachi, Mitsubishi and Toshiba and a member of the National Electronics Servicing Dealers Association, NESDA, and the Georgia Electronics Servicing Dealers Association, GESDA.