CIMM and TVision Explore Potential for Age-Based Bias in Active Meter Audience Measurement Panels, Compared to Passive Meter Panels

Study calls for greater transparency into methods and modelling corrections

The Coalition for Innovative Media Measurement (CIMM) and TVision today released a new study, “Understanding The Issue Of Age-Based Bias In Active Audience Measurement Panels”.

The study used an artificial simulation* of an active panel to explore potential age-based bias in active panels, and to identify the kinds of non-compliance issues that can arise without active management and corrections.           

Panelists participating in the simulation were provided with a tablet and instructed to actively report viewing, while their viewing was also being passively monitored, using ACR and TVision’s Computer Vision. The simulation identified a range of potential errors that could emerge in active panels, requiring corrective solutions:

• When the television was on during the study, the simulation found that passive and active metering approaches agreed on viewer counts only around 56% of the time over the four-week period of the study.
• Up to 44% of TV on-time in the simulation was over- or under-reported by viewers.
• In the simulation, 18–34-year-old viewers were as much as 7X more likely to under-report viewing than over-report when measured through an active meter panel.
• Viewer fatigue was seen over time on active panels, with underreporting going from 23% in Week 0 to 36% in Week 4 and over-reporting going from 19% to 12% over the same time period. Error rates were much higher for older demographics, though over and under reporting by the group generally offset each other.

Potential Drivers and High Costs of Non-Compliance 

Non-compliance on an active panel could be the result of panelist errors or technical faults, but the data could also represent actual viewing behavior. Distinguishing between non-compliant data and real viewing data issues can be complex and costly, requiring field team visits and/or sophisticated modelling work to correct. With a large 40,000+ panel, this requires significant resourcing. Without ongoing active management, the study suggests that panelist response rates will erode over time, relatively rapidly in some cases. 

Even in the short-term, small errors may be inevitable. For example, an individual household might comply for a short period, then stop complying for a few sessions, and then begin complying again. Over time, non-compliance adds up and errors creep in. This could occur in the case where a TV is on, but no buttons have been pressed, making the viewing audience unidentified for a period of time. As a result, the measurement provider needs to make a decision about how best to assign this viewing, given the composition of the household – introducing more complexity and potential for error.

Yan Liu, CEO of TVision, said: “When we consider all the opportunities for potential error in an active panel, and the additional cost burdens associated with effectively managing and reducing that potential for error, we can see that active panels present a significant cost burden for the industry. In contrast, passive meters eliminate non-compliance issues and require less intrusive interactions between the panel operations teams and the panelists. This results in greater accuracy, lower panel churn rates, and lower operational costs.”

“Although measurement providers have developed sophisticated methodologies for addressing non-compliance on active panels,” said Jon Watts, Managing Director, CIMM. “The study helps to illustrate some of the issues that can arise, as vendors look to correct for non-compliance. The impact and effectiveness of these corrections over time may be difficult to determine, unless some kind of secondary measurement check is introduced on an ongoing basis to determine whether the corrections made are accurate.”

* The simulation was not intended to be an active representation of a real panel but was used to explore the behaviors of active and passive panelists, identifying points of divergence across different groups of panelists. 

About CIMM

The Coalition for Innovative Media Measurement (CIMM) is a non-partisan, pan-industry coalition focused on cultivating improvements, best practices and innovations in measurement and currency, new metrics and approaches to understanding the value of media, and data collaboration and enablement. CIMM’s members include leading networks, studios, streamers and programmers, MVPDs, TV OEMs and OS providers, major digital businesses, agencies, measurement and data providers, trade bodies and consultants. CIMM is a subsidiary of the Advertising Research Foundation and adheres to the ARF’s principles of scientific rigor, objectivity and evidence-based research. 

About TVision 

TVision provides second-by-second, person-level data about how people watch TV – who’s watching, what they’re watching, and how much attention they are paying to both linear and streaming TV. Advertisers, agencies, networks, streaming content providers, measurement companies, and data platforms use TVision data to make more informed media decisions, measure performance, produce content that engages audiences, and benchmark their results against competitors. TVision is headquartered in New York City, with offices in Boston and Tokyo.