December 20, 2012

Methodology: In Social Media and Opinion Pages, Newtown Sparks Calls for Gun Reform

http://www.journalism.org/2012/12/20/social-media-and-opinion-pages-newtown-sparks-calls-gun-reform/

This special report by the Pew Research Center’s Project for Excellence in Journalism on media reaction to the shootings in Newtown, Connecticut uses data derived from four different methodologies.

Data regarding the tone of conversation on social media (Twitter and blogs) were derived with a combination of PEJ’s traditional media research methods, based on long-standing rules regarding content analysis, along with computer coding software developed by Crimson Hexagon. That software is able to analyze the textual content from millions of posts on social media platforms. Crimson Hexagon (CH) classifies online content by identifying statistical patterns in words.

Data regarding the framing of coverage in the newspaper editorials and op-eds were derived from the PEJ’s in-house coding operation.

Data about the guests on cable and network television news programs were compiled through the use of transcripts and recorded broadcasts of the relevant shows.

Finally, data on the views on YouTube were collected from that site’s publicly accessible lists.

Coding of Social Media Using a Computer Algorithm

For determining the tone of the conversation on social media, the study employed media research methods that combine PEJ’s content analysis rules developed over more than a decade with computer coding software developed by Crimson Hexagon. This report is based on separate examinations of more than 11 million tweets and 45,000 blog posts.

Crimson Hexagon is a software platform that identifies statistical patterns in words used in online texts. Researchers enter key terms using Boolean search logic so the software can identify relevant material to analyze. PEJ draws its analysis samples from several million blogs, all public Twitter posts and a random sample of publicly available Facebook posts. Then a researcher trains the software to classify documents using examples from those collected posts. Finally, the software classifies the rest of the online content according to the patterns derived during the training.

According to Crimson Hexagon: “Our technology analyzes the entire social internet (blog posts, forum messages, Tweets, etc.) by identifying statistical patterns in the words used to express opinions on different topics.” Information on the tool itself can be found at http://www.crimsonhexagon.com/ and the in-depth methodologies can be found here http://www.crimsonhexagon.com/products/whitepapers/.

Crimson Hexagon measures text in the aggregate and the unit of measure is the ‘statement’ or assessment, not the post or Tweet. One post or Tweet can contain more than one statement if multiple ideas are expressed. The results are determined as a percentage of the overall conversation.

The time frame for the analysis began at 10 am ET on December 14 and went through noon ET on December 17.

PEJ used Boolean searches to narrow the universe to relevant posts. Common terminology posted by users varies for each platform. Therefore, PEJ used slightly different search filters for each.

For Blogs, PEJ used the following search filter:

(Newtown OR Newton OR Connecticut OR gun OR guns OR amendment OR Lanza OR “mental health” OR “sandy hook” OR Obama)

For Twitter, the search filter was:

(Newtown OR Newton OR CT OR Connecticut OR gun OR guns OR control OR amendment OR 2nd OR second OR shooting OR weapon OR Lanza OR mental OR health OR sandy OR hook OR Obama)

Human Coding of Newspaper Editorials and Op-eds

The list of eleven newspapers used in this sample included a variety of geographic locations, ownership groups and circulation sizes. The sample is the close to the same one used in PEJ’s weekly News Coverage Index (NCI), the methodology of which explains the newspaper selection process.

The eleven papers used in this study were as follows:

The New York Times
Los Angeles Times
USA Today
The Wall Street Journal
The Washington Post
The Denver Post
Dallas Morning News
Orlando Sentinel
Traverse City Record-Eagle (MI)
The Daily Herald (WA)
The Eagle-Tribune (MA)

All editorials and op-eds from the print copies of these papers from December 15 to December 18 were examined. Hard copies of the papers were either delivered to the PEJ offices in Washington, D.C., or digital versions were acquired through the papers’ websites or the web service Pressdisplay.

Experienced human coders determined the frame and tone of each relevant article. PEJ has used similar variables in previous studies, and intercoder testing has revealed that such coding meets rigorous academic standards.

Data Regarding the Guests on Television News Shows

PEJ recorded the guests on television news shows on eight channels: ABC, NBC, CBS, PBS, Fox, Fox News, CNN and MSNBC.

All guests who discussed an issue relevant to the Connecticut shootings were included in the tally. Regular contributors to the programs were excluded, as were people directly involved with the incident such as eye-witnesses and family members.

Broadcasts from noon ET on December 14 through 10 am ET on December 17 were included.

PEJ was able to review recorded broadcasts or transcripts of most of the national news programming on each of the seven channels during that time.

For some channels, transcripts and digital recordings were not available and could not be included in our sample.

The programs and times examined were as follows:

Cable

Friday, December 14

Fox News
Daytime shows from noon to 3:00 pm ET
All evening shows from 5:00 pm until 11 pm ET

MSNBC
All programming from noon to 11:00 pm ET

CNN
All programming from 10:30 am to 10:00 pm ET

Saturday, December 15

Fox News
America’s News HQ
Fox & Friends Weekend
Huckabee
Greta

MSNBC
Up w/ Chris Hayes
Elementary School Tragedy
Andrew Mitchell Reports
Melissa Harris-Perry

CNN
All programming from 7:00 am to 11:00 pm ET

Sunday, December 16

Fox News
Fox News Sunday

MSNBC
The Chris Matthews Show

CNN
All programming from 6:00 am to 11:00 pm ET

Monday, December 17

CNN
All programming from 6:00 am to 9:00 am ET

Network

Friday, December 14

ABC
World News Tonight
20/20 Special
Nightline

CBS
CBS Evening News
CBS News Special

NBC
NBC Nightly News
NBC Dateline Special

PBS
PBS NewsHour

Saturday, December 15

ABC
Good Morning America
World News Saturday

CBS
CBS This Morning
CBS Evening News
48 Hours Special Edition

NBC
Today Show
NBC Nightly News

Sunday, December 16

ABC
Good Morning America
This Week with George Stephanopoulos
World News Tonight

CBS
Face the Nation
60 Minutes

NBC
Today Show
The Chris Matthews Show
NBC Nightly News

Fox
Fox News Sunday with Chris Wallace

Monday, December 17

ABC
Good Morning America

CBS
CBS This Morning

NBC
Today Show

Data on YouTube Views

On Wednesday, December 19, at 1:30 pm ET, a PEJ staff member captured the list of most viewed news and politics videos on YouTube over the previous week. These videos are categorized as such on the YouTube site and are often a mix of mainstream news reports, raw footage relating to breaking events, or other types of public affairs clips. PEJ determined the most viewed videos as they are listed on YouTube’s page at the time of capture.