Non-Profit News

The study, Non-Profit News: Assessing a New Landscape in Journalism, involved several phases, all of which were performed in-house by PEJ researchers.

The primary PEJ staff members conducting the research, analysis and writing included: Tricia Sartor, Weekly News Index manager; Kevin Caldwell, researcher/coder; Nancy Vogt, researcher/coder; Jesse Holcomb, research associate; Amy Mitchell, deputy director; Tom Rosenstiel, director.

Other staff members who made substantial contributions to the report were: Christine Bhutta, research associate; Paul Hitlin, senior researcher; Dana Page, communications and creative design manager.

Copy-editing was done by Molly Rohal, communications coordinator for Pew Research Center. Number-checking was done by Steve Adams, researcher/coder.

The following Pew Research Center staff provided design assistance on the interactive tool: Russell Heimlich, web developer; Michael Piccorossi, director of digital strategy and IT; Michael Keegan, graphics director for Pew Social & Demographic Trends and Pew Hispanic Center; Carrie Madigan, informational graphic designer for Pew Social & Demographic Trends and Pew Hispanic Center.

The first phase of research was to identify the media universe. This occurred in May-June 2010. The second phase was conducting audits of the online sites (June-August 2010). The third phase was the content capture and analysis (September 2010-January 2011). Finally, once the coding was complete, several scales were developed as a means of presenting the specific findings on ideology, transparency and productivity.

Details of each of these phases follow.

Defining the Media Universe

Researchers took several steps to define the universe of non-profit news organizations.

Researchers compiled the universe of sites by using three different techniques. First, researchers consulted lists of such news outlets that had already been compiled by academics and journalists who monitor this emerging field. These sources include:

Michele McClellan's (Reynolds Journalism Institute) list of promising community news sites
Brant Houston's (University of Illinois) list of investigative news consortia, discussed in the 2010 report "Ethics for the New Investigative Newsroom."
Knight Digital Media Center
The Poynter Institute (including David Sheddon's Transformation Tracker/Josh Stearns Groundswell)
Newseum/Online News Association's Online Journalism Awards lists
Online Journalism Review
Online News Association
Nieman Journalism Lab

Second, Researchers searched the web for news organizations by using the following terms:  

"[state]" and "News organization"
"[state]" and "News" and "Privately Funded"
"[state]" and "News" and "Non-profit"
"[state]" and "News" and "Independent"
"privately funded news"
"privately funded media"
"general interest journalism online /sites"
"new online site"

Third, researchers identified and contacted each state's press association and asked for a list of credentialed online media outlets. In addition, researchers asked the associations if they were aware of any additional online journalism startups in their state.   

The field was narrowed according to the following criteria:

Outlets must produce at least one original story per week
Outlets must exist entirely online
Outlets must have launched no earlier than 2005
Outlets must list editorial/reportorial staff
Content must consist primarily of reported news, as opposed to opinion and analysis
Outlets may not be the creation of a larger, established media company (i.e., The Nation Institute)
The mission of the outlet must be primarily to produce content, as opposed to treating content creation as an ancillary item, subservient to a different mission.
Outlets must be based in the U.S.
Outlets must focus on news at the state or national level, not the local or hyperlocal level
Outlets must not be dedicated to niche topic areas such as health or finance news, but instead must focus on a broader range of subjects

While the focus of the study is non-profit news outlets, researchers did not limit their searches to non-profit sites alone; commercial or for-profit sites that met all the criteria above were included in the study to serve as a point of comparison.

The final universe for the study consisted of the following 46 online news outlets (in alphabetical order):

Alaska Dispatch
Alaska Watchdog
Cal Watchdog
California Watch
Colorado Independent
Colorado News Agency
Connecticut Mirror
CT News Junkie
Idaho Reporter
Illinois Statehouse News
Iowa Independent
Kansas Watchdog
Maine Watchdog
Maryland Reporter
Michigan Messenger
MinnPost
Missouri News Horizon
Missouri Watchdog
Montana Watchdog
Nebraska Watchdog
Nevada News Bureau
New Hampshire Watchdog
New Jersey Newsroom
New Jersey Spotlight
New Mexico Watchdog
New West
Oklahoma Watchdog
Pennsylvania Independent
Progress Illinois
ProPublica
Sunshine State News
Tennessee Report
Tennessee Watchdog
Texas Tribune
Texas Watchdog
The Daily Caller
The Florida Independent
The Minnesota Independent
The Nerve
The New Mexico Independent
The North Carolina Independent News (now closed)
The Texas Independent
The Washington Independent (now closed)
Virginia Statehouse News
Virginia Watchdog (Old Dominion Watchdog)
West Virginia Watchdog

Site Audits

Once the online news sites were selected, researchers conducted an audit of each site and the primary funding or underwriting organizations behind it, using a codebook specifically developed for this phase of the study. Researchers primarily limited themselves to the information that was readily available on the site, so if particular pieces of information could not be found, this was noted. In some instances-particularly relating to underwriters-researchers reached beyond the site to try to obtain the missing information by conducting additional web searches and consulting other investigative news reports about the sites.

The audit assessed the following variables:

Name of outlet
Year launched
Does the outlet contain editorial/reporting staff (yes/no)
Staff makeup
Number of editorial/reporting staff listed
Intended geographic scope
Stated general focus
Frequency of original reporting
Frequency of blog updates
Tax status
Revenue streams
Transparency of news site:
      How transparent is the outlet in describing its mission?
      Does the outlet provide full contact information?
      Do stories include bylines?
      How transparent is the outlet in revealing its funding sources?
Underwriting structure
About the primary funder(s):
      If foundation/non-profit (describe)
      If private company/firm (describe)
      If individual/family (describe)
Transparency of funder(s):
      How transparent is the funder in describing its mission?
      How transparent is the funder regarding its own financial information?

Capture and Content Retrieval

For one full month (September 7-October 5, 2010), researchers captured and archived all relevant news content (text and audio/video) appearing Monday through Friday on home pages of the sites. All news stories-including the occasional interview and analysis piece-were captured. If a story continued beyond the home page, the entire story was captured. Content clearly identified as blog posts or opinion pieces were not captured. News roundups and/or stories originating from other outlets were not captured.

For the majority of sites, capturing was done the day following publication. In other words, coders captured all of Monday's stories on Tuesday morning. The exception was a select group of sites that regularly published a high volume of content. In order to ensure that none of these sites' content disappeared by the time researchers were scheduled to capture, one staff member was designated to capture stories around 4 p.m. on the day of publication.

Determining the Coding Sample and Content Analysis

The capture and content retrieval phase of the project resulted in a total of 2325 stories. The chronological lists of stories contained in each of the individual subfolders were printed out, and the actual coding sample was determined by selecting every other story captured for each of the sites. Thus, 50% of all stories captured were coded, resulting in a sample of 1122 stories.[1]

A team of five experienced PEJ coders conducted a content analysis on the universe of stories using a codebook specifically designed for the study. Coders went through an extensive training procedure, and achieved an inter-coder reliability of 82% across all variables. Coders achieved a reliability of 80% across the three variables which together measured the presence of ideology in the content-story theme, range of viewpoints, and target of exposes.

In addition to several ‘housekeeping' variables (date coded, source, story date) the content analysis included the following key variables:

• Story Describer - a short description of the content of the story
• Story Topic - determines the broad topic categories addressed by a story, such as government, crime, education, labor, economy, immigration etc.
• Range of Viewpoints - a measure of the effort of the news organization to present a balanced story by presenting multiple viewpoints on a topic or issue that involves some level of possible controversy
• Presence of Journalist Opinion - a simple yes-or-no variable that indicates presence of journalist's opinion
• Target of Wrongdoing - identification of the focus of an investigation or evaluation of specific allegations of malfeasance, corruption, improper behavior, ethical breaches, etc.
• Story Themes - an assessment of whether the total number of assertions in a story drive home a political or partisan message. The following themes were studied:
  • "The hand of government can make society fairer, better, and can bring benefits"
  • "Government regulations, oversight and taxes tend to interfere with progress and the power of the market."
  • Democratic/liberal figure(s)- Positive steps/signs/actions
  • Democratic/liberal figure (s)- negative steps/signs/actions
  • Democratic/liberal figure (s)- is corrupt
  • Republican/conservative figure(s)- Positive steps/signs/actions
  • Republican/conservative figure(s)- negative steps/signs/actions
  • Republican/conservative figure(s)- is corrupt
  • Dem./liberal AND Repub./conservative figures- Pos. steps/signs actions
  • Dem./liberal AND Repub./conservative figures- neg. steps/signs actions
  • Dem./liberal AND Repub./conservative figures- is corrupt
  • Other figure(s)- Positive steps/signs/actions
  • Other figure(s)- negative steps/signs/actions
  • Other figure(s)- is corrupt
  • Wedge issue- liberal point of view
  • Wedge issue- conservative point of view

Data Analysis

When it came to analyzing the content of the sites, several variables were observed in conjunction to indicate the ideological orientation of an outlet or group of outlets.[2] These included the Range of Viewpoints variable, the Target of Exposes variable, and the Story Theme variable.

Researchers combined the results of these three variables into a scale in order to create a summary measure of ideology that would be more reliable than any of the individual indicators alone.

The same was done for the five key transparency indicators that were measured in the audit: transparency about a site's mission and funding sources; the site's accessibility; and the transparency of the site's primary funders about their mission and their own funding sources.

A third scale was created to measure the various indicators of a site's productivity, including the size of a site's editorial/reporting staff; its volume of news articles per week; and the amount of blog or opinion content produced by the site in a typical week.

The sites were given scores on the three 0-100 point rating scales in the following ways.

Ideology scale

Story theme

• Very partisan: 50% or more of site's stories tilted to one ideological point of view, and by a ratio of at least 2-to-1. (Score is 100).
• Somewhat partisan: 40%-49% of a site's stories tilted to one ideological POV, with at least 2-to-1 ratio. (Score is 66.7).
• Slightly partisan: 25%-39% of a site's stories tilted to one ideological POV, with at least 3-to-2 ratio. (Score is 33.3).
• Non-partisan: 0%-24% of a site's stories tilted to one ideological POV, AND there is less than a 2-to-1 ratio of one POV over another. (Score is 0).
• Non-thematic: At least 60% of stories had no theme, and less than 25% leaned in any one direction. (Score is 0).

Target of exposes

• Very partisan: Site targets only liberal actors or conservative actors. (Score is 100).
• Somewhat partisan: Site targets both liberal and conservative actors, but targets one of these more often, by a ratio of at least 2-to-1. (Score is 50).
• Non-partisan: Site does not target any liberal or conservative actors, or, site targets liberal and conservative actors in equal measure. (Score is 0).

Range of viewpoints

• Highly one-sided: 60% or more of a site's stories featured a single POV. (Score is 100).
• Somewhat one-sided: A majority, but less than 60%, of a site's stories featured a single POV. (Score is 66.7).
• Slightly one-sided: A plurality of the site's stories featured a single POV, but not a majority. (Score is 33.3).
• Even-handed: A plurality or majority of the site's stories featured two or more POVs. (Score is 0).

In addition to a content analysis, researchers evaluated the sites on transparency and productivity. Each of those items consisted of several related variables that were scaled together as follows.

Transparency scale

Transparency
of mission

Mission statement

• Very transparent: Site contains detailed ‘about us' section, including description of mission, reason for launch, and what its objectives are. (Score is 100).
• Somewhat transparent: Site contains brief ‘about us' section that hints at mission. (Score is 50).
• Not transparent: No ‘about us' statement. (Score is 0).

Primary funder(s) mission statement

• Very transparent: Funder provides detailed ‘about us' section, including description of mission, reason for launch, and what its objectives are. (Score is 100).
• Somewhat transparent: Funder provides brief ‘about us' section that hints at mission. (Score is 50).
• Not transparent: No ‘about us' statement. (Score is 0).

Site availability

• Very transparent: complete contact information (address, e-mail, phone number). (Score is 100).
• Somewhat transparent: limited contact information (one or two of the above). (Score is 50).
• Not transparent: no contact information. (Score is 0).

Transparency of funding

Site's funding transparency

• Very transparent: Site is very clear about its funding source(s). Provides a list of key funders and/or individual donors, as well as links to foundations. Outlets that provide financial documents on their sites and provide dollar amounts in connection with supporters are considered highly transparent. (Score is 100).
• Somewhat transparent: Site is somewhat clear about its funding source(s), and provides a name but either no link or no description of those sources. (Score is 50).
• Not transparent: no mention of where the site's funding comes from. (Score is 0).

Primary underwriter's funding transparency

• Very transparent: Underwriter is very clear about its funding source(s). Provides a list of funders or a description of the source of its own financial resources, with names if applicable, as well as links to foundations and descriptions of them. (Score is 100).
• Somewhat transparent: Underwriter is somewhat clear about its funding source(s), and provides a name but either no link or no description of those sources. (Score is 50).
• Not transparent: no mention of where the underwriter's funding comes from. (Score is 0).

Productivity scale

Stories per week

• High volume: 20 or more stories per week. (Score is 100).
• Moderate volume: 10-19.9 stories per week. (Score is 66.7)
• Low volume: 5-9.9 stories per week. (Score is 33.3).
• Very low volume: 0-4.9 stories per week. (Score is 0).

Blogging frequency

• High volume: more than 5 posts/week (Score is 100).
• Moderate volume: 3-5 posts/week (score is 66.7).
• Low volume: 1-2 posts/week (Score is 33.3).
• Very low volume: 0 posts/week (Score is 0).

Staffing size

• Very large: 11 or more (Score is 100).
• Large: 5-10 (Score is 66.7)
• Small: 2-4 (Score is 33.3)
• Very small: 1 (Score is 0).

The various components of each of the three scales were each given weights. The sum of those weights is equal to 100% for each scale.

For the ideology scale, the range of viewpoints component and story theme component are worth 45% each, while Target of exposes is worth 10%. For the transparency scale, each of the three components has equal weight (33.3%). For the productivity scale, the stories per week component is worth 30% in the final tally, the blogging frequency component is worth 20%, and the staffing size component is worth 50%.

Presentation of Data in Interactive Tool

Ideological scores were only presented for outlets and groups whose number of stories studied exceeded 20. Transparency and productivity scores were presented for all outlets.

Outlets and groups were labeled as highly ideological, somewhat ideological, or non-ideological, depending on their scale scores. The placement of a site or group into one of these categories was determined by evenly dividing the scores into three equal ranges between the lowest ideology score achieved (0) and the highest score achieved (70). The ranges were as follows:

0-23: non-ideological
24-47: somewhat ideological
48-70: highly ideological

Likewise outlets and groups were labeled as transparent, somewhat transparent, or slightly transparent, depending on their scale scores. Once again, the placement of a site our group into one of these categories was determined by evenly dividing the scores into three equal ranges between the lowest transparency score achieved (20) and the highest (100). The ranges were as follows:

20-46: slightly transparent
47-75: somewhat transparent
76-100: transparent

Finally, outlets and groups were labeled as having high productivity, medium productivity, or low productivity, depending on their scale scores. As with the other measures, the placement of a site or group into one of these categories was determined by evenly dividing the scores into three equal ranges between the lowest productivity score (0) and the highest (100). The ranges were as follows:

0-33: low productivity
34-67: medium productivity
68-100: high productivity

[1]Because the sum of Watchdog.org stories was somewhat small, an exception was made for this group, and every Watchdog.org story that was captured was coded (this was done to ensure valid statistical analysis). This brought the sample of stories analyzed in the study up to 1,203.

[2] Because many sites contained small samples of stories, statistical analysis could not be conducted on their content alone. Therefore, the Watchdog.org and American Independent News Network families of sites, plus the Statehouse News Online consortium, were each analyzed as a unit (as opposed to deriving statistics from the individual sites that make up those groups). The rest of the sites in the study were analyzed individually.