NEW METRICS

Making Impact a Key Performance Indicator

New tools aim to measure the real-world impact of journalism.

By Nicole Zhaoqi Zhong
Updated May 10, 2018
3 min read

Good journalism promotes public accountability and an informed citizenry, but proving that the work matters may seem to be a subjective exercise. Traditionally publications have pointed to stories that won awards, inspired new laws, and jailed corrupt politicians. More often, though, quantifying the real-world  impact of reporting can be difficult.

With the rise of philanthropic funding for news organizations, publishers are discovering a need to demonstrate the real-world benefits of their journalism. Commercial metrics, such as page views and the number of unique visitors, stress scale over quality and cannot possibly indicate how well the most effective newsrooms and nonprofit journalism organizations perform.

But with better technology to collect and analyze data, new digital tools are available to track and measure impact. Below are some of the best:

NewsLynx

NewsLynx is an open-source platform to track, store, and analyze impact, developed by Michael Keller and Brian Abelson in 2015 at Columbia University’s Tow Center for Digital Journalism. It has a bot that searches social networks for key influencers who mention relevant keywords or share links to stories. Then journalists evaluate whether these “impact events” resulted from the reporting. They can then categorize the events in one of four impact categories (Citation, Change, Achievement, or Promotion) on one of five levels (Individual, Community, Institution, Media, or Internal).

Newsrooms can filter and view the performance of stories and make comparisons. Besides documenting the qualitative impact added by journalists, NewsLynx pulls quantitative data from Google Analytics, showing how often stories are promoted, including appearances on homepages or social media shares. The statistics—combining quantitative and qualitative data—provide a more comprehensive context for measuring impact.

Impact Tracker

Impact Tracker is the interactive database that grew out of an open-source code released by the Center for Investigative Reporting in 2016. Developer Lindsay Green-Barber, then CIR’s director of strategic research, has since moved on to found The Impact Architects, a consulting firm that works with media and human rights organizations as well as foundations.

Journalists can enter qualitative descriptions about the impacts of their stories and select various levels of impact from macro (a government investigation, for example) to micro (a reader response, say). On the dashboard, they can filter story results by different criteria and view a timeline documenting when each impact event had taken place. The database can be exported in CSV or Excel for further analysis.

Chalkbeat’s MORI

Education-news nonprofit Chalkbeat developed this WordPress plugin to track story impacts. MORI, or Measures of Our Reporting’s Influence, became open-source in 2016. It allows journalists to enter qualitative descriptions of story impacts and to categorize them.

Here is a table comparing features of the three impact tracking tools mentioned above.

Tool NewsLynx CIR Impact Tracker ChalkBeat MORI
Standalone? Platform PlatformWordPress Plugin
Impact Category Citation
Change
Achievement
Promotion
Citation: Someone mentioned your journalism, like your article is mentioned by another article or a stakeholder you identified as a target for the story

Change: Your journalism led to something happening, like a law being passed

Achievement: Your journalism stands out in some way, like an award, or record page views

Promotion: You posted an article to a social account, you held an event around a reporting topic or a reporter appeared on a broadcast network to discuss his or her work
Macro
Meso
Micro
Media
Macro: Changes that happen at the level of an institution or power holder, resulting in structural changes

Meso: Changes that happen at the level of a group as a result of its consumption of media

Micro: Changes that happen at the level of an individual as a result of his or her consumption of media

Media: Other media responding to content increases the reach of content, amplifying its message and giving it an extended life.

http://impact.cironline.org
 Informed Action
Informed Debate
Informed Action: Formal protest or petition, Government investigation, Institutional action, Lawsuit filed, Legislative action, School- or classroom-level change in policy or practice, State- or district-level change in policy or practice

Informed Debate: Award, Non-press organization cites our work in a report, Public official refers to report, Chalkbeat staffer makes a public appearance or interview, Story picked up by another publication or organization
Impact Level Individual
Community
Institutional
Media
Internal
Individual: A single person. For example, an individual changed their habit after reading your story

Community: A group of people, loosely defined by you. For example, a group of people interested or affected by your story has formed to discuss and share their experience

Institutional: A government or organization. For example, an investigation is launched based on your reporting

Media: A media organization that republishes, picks up or cites your work. For example, your story causes other newsrooms to look at the issue

Internal: Specific to your organization. For example, a given article or project became a model for future stories or was cited by the top editors as particularly on-mission
N.A. N.A.
Automatic Tracking Bot searching preset social media accounts for mentions of the topic or shares from your URL Not Yet Not Yet
Visualization Comparison Point of medium or average of selected stories Timeline of when impact events take place Not Yet
Filter Yes Yes Not Yet
Quantative Data Yes Not Yet Not Yet
Export Not Yet Yes Not Yet
Public? Open-Source Open-Source Open-Source

 

Airtable by ICFJ

The International Center for Journalists (ICFJ) created its own impact-tracking database using Airtable, a cloud-based platform that combines the features of spreadsheets and more complex databases.

As an ICFJ Knight Fellow in Sao Paulo, Pedro Burgos developed Impacto with support from Google News Lab Brazil. Impacto uses bots to search social networks, government websites, blogs, and news sites for mentions that could show potential impacts. Impacto has already been tested by five Brazilian publications, and the ICFJ is planning to release it in other countries this year.

Impact Monitor

The European Journalism Centre (EJC) released Impact Monitor, which tracks the effects of selected projects in Tweets, mentions, links, backlinks, and comments.

How impact tracking can improve

With better machine learning and artificial intelligence technology, new digital tools may be able to analyze the quality of impact events and their relevance to a newsroom’s monitored project. Mission-driven journalism and nonprofit news organizations are promoting impact tracking tools. In time, newsrooms will get better at drawing conclusions from their impact records, showing stakeholders that their funding matters and understanding how to use investigative reporting resources more efficiently.

Key quotes

We hope to create a flexible framework that can help media organizations both define and measure the real-world changes their work can provoke, and better communicate with the public about why their work matters. We want to promote a dialogue for evaluating journalism that goes beyond Web metrics and the pursuit of clicks and into something deeper, even if it requires a bit more work.

We ultimately agreed that more needed to be done by journalists and media to make a clearer case for their value, and that better automated tools would greatly simplify measurement of impact beyond news products. Doing so would build trust, which would make news media more competitive in the fight for audience and revenues. It may even make the difference between viability and extinction.

Newsrooms pride themselves on stories that make a difference. Those stories can be good for business, too. Impact is a point of differentiation in a crowded marketplace for commercial news and a powerful motivator for philanthropists who support public media and non-profit journalism.

This may sound like common sense but even the most forward-thinking media organizations have had to do a lot of soul searching and changed their traditional ways of working to align their work with specific, measurable goals rather than claiming, more vaguely, that all their work was done in the public interest.

Why is this important?

With more news organizations seeking donations and foundation dollars, some publishers are using digital tools to track and measure the real-world impact of their journalism..

Killer links

People to follow

Nicole Zhaoqi Zhong

Nicole Zhong graduated from Nanyang Technological University with a bachelor’s degree in Information Engineering and Media in December 2016. She is interested in digital product design and interactive media.