Webinar: Visual quality assessment for decision making in standardization projects

Springer, EURASIP Journal on Image & Video Processing, and European Association for Biometrics present a free monthly author series on the first Thursday. We hope you can join us!
Like

Read the paper

SpringerLink
SpringerLink SpringerLink

Remote expert viewing, laboratory tests or objective metrics: which one(s) to trust? - EURASIP Journal on Image and Video Processing

We present a study on the validity of quality assessment in the context of the development of visual media coding schemes. The work is motivated by the need for reliable means for decision-taking in standardization efforts of MPEG and JVET, i.e., the adoption or rejection of coding tools during the development process of the coding standard. The study includes results considering three means: objective quality metrics, remote expert viewing, which is a method designed in the context of MPEG standardization, and formal laboratory visual evaluation. The focus of this work is on the comparison of pairs of coded video sequences, e.g., a proposed change and an anchor scheme at a given rate point. An aggregation of performance measurements across multiple rate points, such as the Bjøntegaard Delta rate, is out of the scope of this paper. The paper details the test setup for the subjective assessment methods and the objective quality metrics under consideration. The results of the three approaches are reviewed, analyzed, and compared with respect to their suitability for the decision-taking task. The study indicates that, subject to the chosen test content and test protocols, the results of remote expert viewing using a forced-choice scale can be considered more discriminatory than the results of naïve viewers in the laboratory tests. The results further that, in general, the well-established quality metrics, such as PSNR, SSIM, or MS-SSIM, exhibit a high rate of correct decision-making when their results are compared with both types of viewing tests. Among the learning-based metrics, VMAF and AVQT appear to be most robust. For the development process of a coding standard, the selection of the most suitable means must be guided by the context, where a small number of carefully selected objective metrics, in combination with viewing tests for unclear cases, appears recommendable.

Our next 1-hour webinar will take place Thursday, September 5 at 2:00pm CEST with esteemed author, Dr.-Ing. Mathias Wien.

RSVP here to join: https://cassyni.com/events/3gasWoMkxJY86L2SWn7Hdw?cb=functi

Title: Visual quality assessment for decision making in standardization projects

Abstract: In the context of the development of compression standards for visual media, typically, most decision making relies on the measurement with one or more objective quality metrics. In many cases, a small number of very simple metrics, such as the PSNR or the SSIM, are applied in decision making processes, e.g., in the context of adoption of coding tools in to a draft specification. THis applies to a variety of visual media under consideration, such as classical 2D video or various representations of immersive visual media like dynamic point clouds or meshes. Given the rise of learning-based coding tools and -apparently- competitive end-to-end learned coding schemes, as well as the increasing number of filtering blocks inside or outside of the coding loop of conventional coding schemes, the suitability of such metrics may be questioned. This is due to a potential lack of correlation with mean opinion scores acquired by subjective assessment, especially if specific artifacts, such as temporal consistency, are not well reflected by the metric. This problem can be even more significant for more advanced, potentially learning-based metrics, which may show unexpected behavior if being applied to compression artifacts which have not been known or seen by the time of training the corresponding metric.

Advisory Group ISO/IEC SC 29/AG 5 MPEG Visual Quality Assessment is tasked with evaluating and recommending metrics and testing procedures for the use in standardization projects inside the body of MPEG Working Groups developing compression standards for visual media. This webinar presents recent insights in the performance of metrics and subjective assessment methods for a variety of visual media types. The evaluation includes laboratory tests as well as remote and on-site expert viewing sessions which are frequently conducted during MPEG standardization meetings. The results and the performance of such subjective tests are assessed and used to benchmark objective metrics commonly used or considered for application in the development process. Furthermore and outlook is provided to the dataset of compressed video for study of quality metrics (CVQM) which is currently being developed in AG 5 and which includes reconstructed video sequences from a set of conventional and learning-based coding schemes.

Speaker Bio: Mathias Wien received the Diploma and Dr.-Ing. degrees from Rheinisch-Westfälische Technische Hochschule Aachen (RWTH Aachen University), Aachen, Germany, in 1997 and 2004, respectively. In 2018, he achieved the status of the habilitation, which makes him an independent scientist in the field of visual media communication. His research interests include image and video processing, immersive, space-frequency adaptive and scalable video compression, and visual quality assessment. Since 2020, Mathias serves as Convenor of ISO/IEC JTC1 SC29/AG5 “MPEG Visual Quality Assessment”. Mathias has been an active contributor to H.264/AVC, HEVC, and VVC. He has participated and contribute to ITU-T VCEG, ISO/IEC MPEG, the Joint Video Experts Team (JVET) and preceding joint teams of VCEG and ISO/IEC MPEG. Mathias has published more than 80 scientific articles and conference papers in the area of video coding and has co-authored several patents in this area. Mathias has further authored and co-authored more than 250 standardization documents. He has published the Springer textbook “High Efficiency Video Coding: Coding Tools and Specification”, which fully covers Version 1 of HEVC.

We look forward to seeing you there!

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Follow the Topic

Computer Imaging, Vision, Pattern Recognition and Graphics
Mathematics and Computing > Computer Science > Computer Imaging, Vision, Pattern Recognition and Graphics
Signal, Speech and Image Processing
Technology and Engineering > Electrical and Electronic Engineering > Signal, Speech and Image Processing
Image Processing
Technology and Engineering > Electrical and Electronic Engineering > Signal, Speech and Image Processing > Image Processing
Image Processing
Mathematics and Computing > Computer Science > Computer Imaging, Vision, Pattern Recognition and Graphics > Image Processing