Video quality

The terms model and metric are often used interchangeably in the field. However a metric has certain mathematical properties, which, by strict definition, do not apply to all video quality models.

The term “objective” relates to the fact that, in general, quality models are based on criteria that can be measured objectively – that is, free from human interpretation. They can be automatically evaluated by a computer program. Unlike a panel of human observers, an objective model should always deterministically output the same quality score for a given set of input parameters.

Objective quality models are sometimes also referred to as instrumental (quality) models,[1][2] in order to emphasize their application as measurement instruments. Some authors suggest that the term “objective” is misleading, as it “implies that instrumental measurements bear objectivity, which they only do in case that they can be generalized.”[3]

Classification of objective video quality models into Full-Reference, Reduced-Reference and No-Reference.

No-reference image and video quality assessment methods.

Objective models can be classified by the amount of information available about the original signal, the received signal, or whether there is a signal present at all:[4]

• Full Reference Methods (FR): FR models compute the quality difference by comparing the original video signal against the received video signal. Typically, every pixel from the source is compared against the corresponding pixel at the received video, with no knowledge about the encoding or transmission process in between. More elaborate algorithms may choose to combine the pixel-based estimation with other approaches such as described below. FR models are usually the most accurate at the expense of higher computational effort. As they require availability of the original video before transmission or coding, they cannot be used in all situations (e.g., where the quality is measured from a client device).
• Reduced Reference Methods (RR): RR models extract some features of both videos and compare them to give a quality score. They are used when all the original video is not available, or when it would be practically impossible to do so, e.g. in a transmission with a limited bandwidth. This makes them more efficient than FR models at the expense of lower accuracy.
• No-Reference Methods (NR): NR models try to assess the quality of a distorted video without any reference to the original signal. Due to the absence of an original signal, they may be less accurate than FR or RR approaches, but are more efficient to compute.
  • Pixel-Based Methods (NR-P): Pixel-based models use a decoded representation of the signal and analyze the quality based on the pixel information. Some of these evaluate specific degradation types only, such as blurring or other coding artifacts.
  • Parametric/Bitstream Methods (NR-B): These models make use of features extracted from the transmission container and/or video bitstream, e.g. MPEG-TS packet headers, motion vectors and quantization parameters. They do not have access to the original signal and require no decoding of the video, which makes them more efficient. In contrast to NR-P models, they have no access to the final decoded signal. However, the picture quality predictions they deliver are not very accurate.
  • Hybrid Methods (Hybrid NR-P-B): Hybrid models combine parameters extracted from the bitstream with a decoded video signal. They are therefore a mix between NR-P and NR-B models.

Some models that are used for video quality assessment (such as PSNR or SSIM) are simply image quality models, whose output is calculated for every frame of a video sequence. This quality measure of every frame can then be recorded and pooled over time to assess the quality of an entire video sequence. While this method is easy to implement, it does not factor in certain kinds of degradations that develop over time, such as the moving artifacts caused by packet loss and its concealment. A video quality model that considers the temporal aspects of quality degradations, like VQM or the MOVIE Index, may be able to produce more accurate predictions of human-perceived quality.

An overview of recent no-reference image quality models has been given in a journal paper by Shahid et al.[4] As mentioned above, these can be used for video applications as well. No-reference, pixel-based quality models designed specifically for video are however rare, with Video-BLIINDS[5] being one example. The Video Quality Experts Group has a dedicated working group on developing no-reference metrics (called NORM).

The most traditional ways of evaluating quality of digital video processing system (e.g. a video codec) are FR-based. Among the oldest FR metrics are signal-to-noise ratio (SNR) and peak signal-to-noise ratio (PSNR), which are calculated between every frame of the original and the degraded video signal. PSNR is the most widely used objective image quality metric, and the average PSNR over all frames can be considered a video quality metric. PSNR is also used often during video codec development in order to optimize encoders. However, PSNR values do not correlate well with perceived picture quality due to the complex, highly non-linear behavior of the human visual system.[6]

With the success of digital video, a large number of more precise FR metrics have been developed. These metrics are inherently more complex than PSNR, and need more computational effort to calculate predictions of video quality. Among those metrics specifically developed for video are VQM and the MOVIE Index.

Based on the results of benchmarks by the Video Quality Experts Group (VQEG) (some in the course of the Multimedia Test Phase (2007–2008) and the HDTV Test Phase I (2009–2011)), some RR/FR metrics have been standardized in ITU-T as:

• ITU-T Rec. J.147 (FR), 2002 (includes VQM)
• ITU-T Rec. J.246 (RR), 2008
• ITU-T Rec. J.247 (FR), 2008 (see PEVQ)
• ITU-T Rec. J.341 (FR), 2011 (see VQuad-HD)
• ITU-T Rec. J.342 (RR), 2011

The Structural Similarity (SSIM) FR image quality metric is also often used for estimating video quality. Visual Information Fidelity (VIF) – also an image quality metric – is a core element of the Netflix Video Multimethod Assessment Fusion (VMAF), a tool that combines existing metrics to predict video quality.

Full or reduced-reference metrics still require access to the original video bitstream before transmission, or at least part of it. In practice, an original stream may not always be available for comparison, for example when measuring the quality from the user side. In other situations, a network operator may want to measure the quality of video streams passing through their network, without fully decoding them. For a more efficient estimation of video quality in such cases, parametric/bitstream-based metrics have also been standardized:

• ITU-T Rec. P.1201, 2012
• ITU-T Rec. P.1202, 2012
• ITU-T Rec. P.1203.1, 2016

Few of these standards have found commercial applications, including PEVQ and VQuad-HD. SSIM is also part of a commercially available video quality toolset (SSIMWAVE). VMAF is used by Netflix to tune their encoding and streaming algorithms, and to quality-control all streamed content.[7][8] It is also being used by other technology companies like Bitmovin[9] and has been integrated into software such as FFmpeg.

Since objective video quality models are expected to predict results given by human observers, they are developed with the aid of subjective test results. During development of an objective model, its parameters should be trained so as to achieve the best correlation between the objectively predicted values and the subjective scores, often available as mean opinion scores (MOS).

The most widely used subjective test materials are in the public-domain and include still picture, motion picture, streaming video, high definition, 3-D (stereoscopic) and special-purposes picture quality related datasets.[10] These so-called databases are created by various research laboratories around the world. Some of them have become de facto standards, including several public-domain subjective picture quality databases created and maintained by the Laboratory for Image and Video Engineering (LIVE) as well the Tampere Image Database 2008. A collection of databases can be found in the QUALINET Databases repository. The Consumer Digital Video Library (CDVL) hosts freely available video test sequences for model development.

In theory, a model can be trained on a set of data in such a way that it produces perfectly matching scores on that dataset. However, such a model will be over-trained and will therefore not perform well on new datasets. It is therefore advised to validate models against new data and use the resulting performance as a real indicator of the model's prediction accuracy.

To measure the performance of a model, some frequently used metrics are the linear correlation coefficient, Spearman's rank correlation coefficient, and the root mean square error (RMSE). Other metrics are the kappa coefficient and the outliers ratio. ITU-T Rec. P.1401 gives an overview of statistical procedures to evaluate and compare objective models.

Objective video quality models can be used in various application areas. In video codec development, the performance of a codec is often evaluated in terms of PSNR or SSIM. For service providers, objective models can be used for monitoring a system. For example, an IPTV provider may choose to monitor their service quality by means of objective models, rather than asking users for their opinion, or waiting for customer complaints about bad video quality.

An objective model should only be used in the context that it was developed for. For example, a model that was developed using a particular video codec is not guaranteed to be accurate for another video codec. Similarly, a model trained on tests performed on a large TV screen should not be used for evaluating the quality of a video watched on a mobile phone.

When estimating quality of a video codec, all the mentioned objective methods may require repeating post-encoding tests in order to determine the encoding parameters that satisfy a required level of visual quality, making them time consuming, complex and impractical for implementation in real commercial applications. There is ongoing research into developing novel objective evaluation methods which enable prediction of the perceived quality level of the encoded video before the actual encoding is performed.[11]