Methodology

Examining the news agenda of 48 different outlets in five media sectors, including newspapers, online, network TV, cable TV, and radio, the NCI is designed to provide news consumers, journalists and researchers with hard data about what stories and topics the media are covering, the trajectories of major stories and differences among news platforms. Following a rotation system, 35 outlets each weekday were selected as well as 7 newspapers each Sunday.

For its News Index PEJ monitors 48 different news outlets each week (35 per week-day) from five different media sectors

Newspapers (Thirteen in all, Sun-Fri)

NY Times every day

Code 2 out of these 4 every day
Wash Post
LA Times
USA Today
Wall Street Journal

Code 2 out of these 4 every day
The Boston Globe
Star Tribune
Austin American-Statesman
Albuquerque Journal

Code 2 out of these 4 every day
The Sun Chronicle
Star Beacon
The Chattanooga Times Free Press
The Bakersfield Californian

Web sites (Five in all, Mon-Fri)

CNN.com
Yahoo News
MSNBC.com
Google News
AOL News

Network TV (Seven in all, Mon-Fri)

Morning shows
ABC – Good Morning America
CBS – Early Show
NBC – Today

Evening news
ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS – Newshour with Jim Lehrer

Cable TV (Fifteen in all, Mon-Fri)

Daytime (1–1:30 pm) – code 2 out of 3 every day
CNN
Fox News
MSNBC

Nighttime CNN – code 3 out of the 4 every day
Lou Dobbs Tonight
Situation Room (7 pm)
Paula Zahn Now
Anderson Cooper 360

Nighttime Fox News – code 3 out of the 4 every day
Special Report w/ Brit Hume
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity & Colmes

Nighttime MSNBC – code 2 out of the 4 every day
Tucker (6 pm)
Hardball (7 pm)
Countdown w/ Keith Olbermann
Scarborough Country

Radio (Eight in all, Mon-Fri)

Headlines every day
ABC Radio headlines at 9am and 5pm
CBS Radio headlines at 9am and 5pm
NPR Morning Edition every day

Talk Radio
Rush Limbaugh every day

1 out of 2 additional conservatives each day
Sean Hannity
Michael Savage

1 out of 2 liberals each day
Ed Schultz
Randi Rhodes

From that content, PEJ analyzes all stories with a national or international focus that appearing as follows:

On the front page of newspapers
In the entirety of commercial network evening newscasts.
The first 30 minutes of network morning news, the PBS evening news, and all cable programs
The top 5 stories on each website at the time of capture

The resulting universe of stories was coded by a team, which is made up of 8 trained coders, a coding administrator, and a senior research methodologist. The complete methodology for the weekly NCI has further details on the coding system and coder reliability.

This report aggregates the NCI from April 1, 2007- June 29, 2007. The resulting universe totals 18,010 news stories, 459 hours of broadcast content (148 hours from network TV, 224 hours from cable, 87 hours from radio), 2.16 million words in newspapers, and 1.10 million words from news websites.

The following is some additional methodology information that applies specifically to the quarterly report.

Intercoder Reliability Testing

In order to continue to assure that our coding is performed with a high level of reliability, we have performed multiple tests of intercoder agreement with all of our coders.

During the 2nd quarter of 2007 we have had 8 professional coders work on the Weekly News Index project. All were included in our intercoder testing.

Housekeeping Variables

We first conducted a test of the “housekeeping” variables of the index in April. For this test, we selected a random sample of 151 stories from each of the 5 media sectors we cover. Each story was coded by two different coders. This represented more than 10% of the number of stories we code in a given week.

Of those stories, 32 were print stories (newspaper and online) and 119 were broadcast stories (television and radio). For our housekeeping variables, we achieved the following levels of agreement:

Print (32 cases)
Story Date: 100%
Source: 100%
Story word count (+- 20 words): 90%
Placement: 94%

Broadcast (119 cases)
Story Date: 100%
Source: 100%
Broadcast start time: 100%
Headline: 100%
Story start time (+- 6 seconds): 91%
Placement: 93%
Story end time (+- 6 seconds): 92%

Main Variables

Having demonstrated that we had a high level of agreement for all of our housekeeping variables, we then had the coders participate in an additional test to determine the level of agreement for the main variables of the index.

We randomly selected 116 stories from both print and broadcast mediums, which represent about 8% of the stories we code in a typical week. The level of agreement for each of our key variables was as follows:

Format: 89%
Big Story: 91%
Sub-storyline: 87%
Geographic Focus: 91%
Topic: 85%

We will continue to conduct further tests of intercoder agreement in the coming months in order to assure continued quality in our coding processes.

Lexis-Nexis Search for Candidate Mentions

Note: The following methodology applies only to PEJ’s search for candidate names (LINK) and not to the majority of data included in this report and the weekly News Coverage Index reports. For a detailed methodology about how the weekly News Index data is compiled, go here.

The Lexis-Nexis database search was conducted for each of the three leading candidates, from both the Republican and Democratic fields; and also for Fred Thompson and Mayor Bloomberg.

The newspapers included in this search were:

            New York Times
            Washington Post
            Los Angeles Times
            USA Today
            Boston Globe
            Star Tribune (MN)
            Chattanooga Times Free Press
            Albuquerque Journal
            Austin American-Statesmen

We also searched news transcripts from the three major network television stations. The transcripts for the network television stations include a number of programs that are not included in PEJ’s weekly sample. Also, Lexis has transcripts for the entire programs so some segments that do not appear in the weekly index coding would be included in this Lexis search.

The television programs included in the search were:

ABC

                        World News Tonight
                        This Week
                        20/20
                        Good Morning America

CBS

                        60 Minutes
                        CBS Evening News
                        The Early Show
                        Face the Nation

NBC

                        Meet the Press
                        NBC Nightly News
                        Today

PBS

NewsHour with Jim Lehrer

For both newspapers and the television transcripts, searches were conducted using the last names of the candidates that appeared in either the headline or lead paragraphs of the story. The reason for searching for the names in the headline or lead paragraphs was to determine the number of stories that focused on the candidates. Another option would have been to search for mentions anywhere in the article or transcript which would have yielded more results, but would have also included many stories where the candidates were not central to the story.

The exceptions for using the candidate’s last name were Hillary Clinton; John Edwards Fred Thompson, and Mayor Bloomberg. Because of former President Bill Clinton, a search of the name “Clinton” would yield many articles about him and not the current Senator from New York. In addition, many articles refer to the candidate as “Hillary Clinton” while others refer to her as “Hillary Rodham Clinton”. Therefore, we searched for both options in order to get the proper total number of stories. In locating stories on “John Edwards” and “Fred Thompson”, full names of the candidates were used due in part that Edwards and Thompson are both widely used last names in the US. For Mayor Bloomberg, “Mayor Bloomberg” was used as a search term to avoid including stories on Bloomberg industries or individuals with the same last name. Many articles refer to Michael Bloomberg as “Mayor Bloomberg.”