State of the News Media Methodology
The State of the News Media fact sheets use a range of different methodologies to study the health of the U.S. news industry, including custom analysis of news audience behavior, secondary analysis of industry data and direct reporting to solicit information unavailable elsewhere.
State of the News Media industry data
The State of the News Media fact sheets consist of data originally generated by other individuals or organizations that Pew Research Center then collected and aggregated.
For the data aggregated from other researchers, Pew Research Center’s team took several steps. First, Center researchers tried to determine what data had been collected and by whom for the media sectors studied. In many cases, this included securing rights to data through license fees or other means, and often included paying for use of the data.
Next, we studied the data closely to determine where elements reinforced each other and where there were apparent contradictions or gaps. In doing so, Pew Research Center endeavored to determine the value and validity of each dataset. That, in many cases, involved going back to the sources that collected the research in the first place. Where data conflicted, the Center has included all relevant sources and tried to explain their differences, either in footnotes or in the narratives.
All sources are cited in footnotes or within the narrative and in all graphics in the report.
Analysis of comScore digital audience data
In order to provide as clear a sense as possible of the digital audience data of U.S. news organizations, researchers took several steps using digital audience measurement data from comScore, a cross-platform measurement company.
After the initial selection process, each comScore entry representing outlets or websites (“entity”) was individually vetted by researchers and unusual shifts in data over time were checked with comScore analysts. Using feedback from analysts, entities that registered increases or declines in unique visitors over the time period studied (October 2014-December 2016) that were likely due to inorganic changes in measurement or site structure were considered to not have trendable data and were removed from the analysis.
For all outlets, entities that are representative of an entire web domain (“total domain entities”) were used whenever possible. When domain entities were incomplete – i.e., when they did not represent traffic to an entire web domain – custom entities that represent the total domain were used when available. Multiple entities of the same URL that were the result of a structure change in the comScore database over time were considered comparable. Researchers consulted comScore analysts throughout the entity selection process.
The selection process for each fact sheet that included comScore data is detailed below.
Newspapers: Researchers analyzed the comScore data of the top 49 newspapers by average Sunday circulation for Q3 2015 and Q3 2016 according to the Alliance for Audited Media data, with the addition of The Wall Street Journal. Each was matched with its associated total-domain entities in comScore. The following 50 entities were used in 2016 (for the entities used in 2015, see the methodology in the 2016 State of the News Media):
Digital-native news outlets: Researchers assessed all domains from 11 comScore categories (Business/Finance, Directories/Resources, Entertainment, Games, Lifestyles, News/Information, Regional/Local, Services, Social Media, Sports, and Technology) with at least 10 million average monthly unique digital visitors in the fourth quarter of 2016. From that set of entities, they then selected digital-native news outlets using the following criteria:
- Must be “born on the web,” i.e. not the website of a legacy news brand (Note: may be owned by a legacy media company).
- It is a publisher of original content about news, defined as current events affecting public life (can include both original reporting and commentary/analysis). Sites are judged by an assessment of the material appearing on their home page. A review of top stories on the home page must render some evidence of original reporting, such as interviews, eyewitness accounts or referral to source documents, by a dedicated reporter/editorial staff. Sites are also judged as news publishers if they self-describe as an organization that produces news, either in the subject headers/navigation bar and/or in their “about” or advertising section through usage of terms like “news,” “journalism,” “covering,” or “informing.”
- It is not entirely focused on reviews, advice, recipes or unedited raw data.
- It is not primarily a user-generated or aggregated content platform (such as Medium, Reddit or Wikipedia). Branded content such as NBA.com was also excluded.
The following 36 entities were used for 2016:
For each website, minutes per visit and unique visitors for October-December of each year were acquired from the comScore Media Metrix database for Total Digital Population.
Comparisons year over year are between monthly averages of October-December data in each year.
For sites that didn’t meet the reporting threshold for one month out of a quarter, we averaged across the two months for which we had data.
Analysis of Nielsen Media Research data
The analysis of viewership trends for cable TV is based on Nielsen Media Research data.
Nielsen provides audience data for the three major cable channels: CNN, Fox and MSNBC. This includes prime-time and daytime viewership. For network TV, Nielsen provides audience data for news programming broadcasted by the three networks – ABC, CBS and NBC – in the morning and evening, as well as audience data for the Sunday morning talk shows and TV news magazines.
Beginning in 2015, Nielsen viewership data was presented in weighted form, made available to Pew Research Center by Nielsen Media Research for all years starting with 2007. Weighted data better account for the number of programming hours in a given broadcast month, though they do not differ substantially from unweighted data.
Methodologies have evolved from 2007 to 2016, and include Local People Meter, Set-Meter with Diaries, Diary Only, Set-Meter with Viewer Assignment, and Code Reader with Viewer Assignment.
Most recently, in January of 2016, local measurement in 45 mid-sized markets (currently called Set-Meter and Code Reader markets), changed from diary measurement for persons, to full electronic measurement with the use of Nielsen’s proprietary Viewer Assignment technique for demographics.
The analysis of newspaper newsroom employment is based on a technique developed by Alex T. Williams in collaboration with Pew Research Center.
Digital-native news outlet audit
Researchers studied several outreach avenues that digital-native news outlets could take to engage with their audiences. For mobile apps, researchers searched the Google Play and iOS App Store for each site. For newsletters, researchers searched each site for a sign-up form. For Apple News, researchers searched the Apple News app for official channels. For podcasts, researchers searched the iTunes podcast store for podcasts from the outlet and performed a search on each site. An outlet was determined to support comments if at least one of the first five stories on its homepage supported comments at analysis time. For social media outreach, researchers searched for official pages, accounts or channels on each platform, as well as on the outlet’s primary website.
In addition to those who compiled the fact sheets, assistance in data analysis was provided by Jalen Alexander, Allison Becker, Emily Bello-Pardo, Jake Bleich, Erin Farrell, Kristine Lu, William Tedrick, Spencer Ward and Robert Weed. Margaret Porteus and Shannon Greenwood also provided graphic and web producing support.