Making Impact a Key Performance Indicator
New tools aim to measure the real-world impact of journalism.
Good journalism promotes public accountability and an informed citizenry, but proving that the work matters may seem to be a subjective exercise. Traditionally publications have pointed to stories that won awards, inspired new laws, and jailed corrupt politicians. More often, though, quantifying the real-world impact of reporting can be difficult.
With the rise of philanthropic funding for news organizations, publishers are discovering a need to demonstrate the real-world benefits of their journalism. Commercial metrics, such as page views and the number of unique visitors, stress scale over quality and cannot possibly indicate how well the most effective newsrooms and nonprofit journalism organizations perform.
But with better technology to collect and analyze data, new digital tools are available to track and measure impact. Below are some of the best:
NewsLynx is an open-source platform to track, store, and analyze impact, developed by Michael Keller and Brian Abelson in 2015 at Columbia University’s Tow Center for Digital Journalism. It has a bot that searches social networks for key influencers who mention relevant keywords or share links to stories. Then journalists evaluate whether these “impact events” resulted from the reporting. They can then categorize the events in one of four impact categories (Citation, Change, Achievement, or Promotion) on one of five levels (Individual, Community, Institution, Media, or Internal).
Newsrooms can filter and view the performance of stories and make comparisons. Besides documenting the qualitative impact added by journalists, NewsLynx pulls quantitative data from Google Analytics, showing how often stories are promoted, including appearances on homepages or social media shares. The statistics—combining quantitative and qualitative data—provide a more comprehensive context for measuring impact.
Impact Tracker is the interactive database that grew out of an open-source code released by the Center for Investigative Reporting in 2016. Developer Lindsay Green-Barber, then CIR’s director of strategic research, has since moved on to found The Impact Architects, a consulting firm that works with media and human rights organizations as well as foundations.
Journalists can enter qualitative descriptions about the impacts of their stories and select various levels of impact from macro (a government investigation, for example) to micro (a reader response, say). On the dashboard, they can filter story results by different criteria and view a timeline documenting when each impact event had taken place. The database can be exported in CSV or Excel for further analysis.
Education-news nonprofit Chalkbeat developed this WordPress plugin to track story impacts. MORI, or Measures of Our Reporting’s Influence, became open-source in 2016. It allows journalists to enter qualitative descriptions of story impacts and to categorize them.
Here is a table comparing features of the three impact tracking tools mentioned above.
|Tool||NewsLynx||CIR Impact Tracker||ChalkBeat MORI|
|Impact Category||Citation Change AchievementPromotion
Citation: Someone mentioned your journalism, like your article is mentioned by another article or a stakeholder you identified as a target for the story Change: Your journalism led to something happening, like a law being passed Achievement: Your journalism stands out in some way, like an award, or record page views Promotion: You posted an article to a social account, you held an event around a reporting topic or a reporter appeared on a broadcast network to discuss his or her work
|Macro Meso Micro Media
Macro: Changes that happen at the level of an institution or power holder, resulting in structural changes Meso: Changes that happen at the level of a group as a result of its consumption of media Micro: Changes that happen at the level of an individual as a result of his or her consumption of media Media: Other media responding to content increases the reach of content, amplifying its message and giving it an extended life.
|Informed ActionInformed Debate
Informed Action: Formal protest or petition, Government investigation, Institutional action, Lawsuit filed, Legislative action, School- or classroom-level change in policy or practice, State- or district-level change in policy or practice Informed Debate: Award, Non-press organization cites our work in a report, Public official refers to report, Chalkbeat staffer makes a public appearance or interview, Story picked up by another publication or organization
Individual: A single person. For example, an individual changed their habit after reading your story Community: A group of people, loosely defined by you. For example, a group of people interested or affected by your story has formed to discuss and share their experience Institutional: A government or organization. For example, an investigation is launched based on your reporting Media: A media organization that republishes, picks up or cites your work. For example, your story causes other newsrooms to look at the issue Internal: Specific to your organization. For example, a given article or project became a model for future stories or was cited by the top editors as particularly on-mission
|Automatic Tracking||Bot searching preset social media accounts for mentions of the topic or shares from your URL||Not Yet||Not Yet|
|Visualization||Comparison Point of medium or average of selected stories||Timeline of when impact events take place||Not Yet|
|Quantative Data||Yes||Not Yet||Not Yet|
|Export||Not Yet||Yes||Not Yet|
Airtable by ICFJ
The International Center for Journalists (ICFJ) created its own impact-tracking database using Airtable, a cloud-based platform that combines the features of spreadsheets and more complex databases.
As an ICFJ Knight Fellow in Sao Paulo, Pedro Burgos developed Impacto with support from Google News Lab Brazil. Impacto uses bots to search social networks, government websites, blogs, and news sites for mentions that could show potential impacts. Impacto has already been tested by five Brazilian publications, and the ICFJ is planning to release it in other countries this year.
The European Journalism Centre (EJC) released Impact Monitor, which tracks the effects of selected projects in Tweets, mentions, links, backlinks, and comments.
How impact tracking can improve
With better machine learning and artificial intelligence technology, new digital tools may be able to analyze the quality of impact events and their relevance to a newsroom’s monitored project. Mission-driven journalism and nonprofit news organizations are promoting impact tracking tools. In time, newsrooms will get better at drawing conclusions from their impact records, showing stakeholders that their funding matters and understanding how to use investigative reporting resources more efficiently.
We hope to create a flexible framework that can help media organizations both define and measure the real-world changes their work can provoke, and better communicate with the public about why their work matters. We want to promote a dialogue for evaluating journalism that goes beyond Web metrics and the pursuit of clicks and into something deeper, even if it requires a bit more work.
We ultimately agreed that more needed to be done by journalists and media to make a clearer case for their value, and that better automated tools would greatly simplify measurement of impact beyond news products. Doing so would build trust, which would make news media more competitive in the fight for audience and revenues. It may even make the difference between viability and extinction.
Newsrooms pride themselves on stories that make a difference. Those stories can be good for business, too. Impact is a point of differentiation in a crowded marketplace for commercial news and a powerful motivator for philanthropists who support public media and non-profit journalism.
This may sound like common sense but even the most forward-thinking media organizations have had to do a lot of soul searching and changed their traditional ways of working to align their work with specific, measurable goals rather than claiming, more vaguely, that all their work was done in the public interest.
Why is this important?With more news organizations seeking donations and foundation dollars, some publishers are using digital tools to track and measure the real-world impact of their journalism..
- MediaShift How the USA Today Network Is Measuring Impact in Its Newsrooms
- MediaShift 5 Needs and 5 Tools for Measuring Media Impact
- ProPublica Issues Around Impact
- Nieman Lab How can journalists measure the impact of their work? Notes toward a model of measurement
- Nieman Lab Metrics, metrics everywhere: How do we measure the impact of journalism?
- Nieman Lab Suffering From Realness, Pivoting to Impact
- Ijnet How can we measure the impact of journalism?
People to follow
Lindsay Green-Barber is the founder of The Impact Architects.
Anjanette Delgado is the digital director of Gannett’s lohud.com.
Richard Tofel is the president of ProPublica.
Michael Keller is an investigative reporter at The New York Times covering artificial intelligence and technology.
Natalie Marinkovich is the grants and impact associate at KQED.
Media Impact Project is a hub for collecting, developing and analyzing approaches for measuring the impact of media.
Center for Investigative Reporting is a nonprofit news organization based in Emeryville, California, and has conducted investigative journalism since 1977.
ProPublica is a nonprofit organization based in New York City that produces investigative journalism in the public interest.