What’s the point of journalism? That’s a question Brian Boyer has taken deeply to heart. As the Visuals Editor of NPR, he’s been hard at work for the last six months on a project called Carebot, which tracks how well news stories drive audiences to take action and aims to “change how newsrooms measure and celebrate success.”
Carebot was funded through a Knight Foundation Prototype Grant, which is now wrapping up. I talked with Brian about what his team has learned – in the business of making people care.
Image: Courtesy NPR Visuals
Sacha: What are you measuring with Carebot right now?
Brian: We’ve played around with a lot of different metrics – the “linger rate” with a graphic, how many people read a story all the way to the end, Facebook shares per pageview. Carebot is not a set formula. It’s more of a framework for how you look at your analytics.
What’s most important is to first define what success means to you. On the Visuals Team, we decided that we believe that journalism has the power to make people care. So we tried to measure that in a variety of different ways. But this goal has been isolated to our team. NPR’s Visuals team is a small group of technologists who work within the newsroom. So we really like to experiment. Other NPR desks and news organizations will – and should – have a different definition of success.
Sacha: It sounds like you’re trying to move past the traditional journalistic metric of pageviews.
Brian: Pageviews is a metric that’s intended to serve the needs of online advertising. Really, all of Google Analytics is set up that way, because Google is an advertiser.
For us, a pageview is an indication of intent. The reader clicks and says, “I might be interested in that.” But we want to know what the reader does from there.
Sacha: It’s interesting that Carebot delivers analytics directly to your reporters about how their stories perform. Has this had any effect on their reporting – even just to provoke discussions?
Brian: Some news organizations have chosen not to send performance statics to their reporters. Famously, the New York Times doesn’t do this. I understand why they would want to keep certain numbers out of the newsroom. They don’t reflect journalistic quality; they reflect the fact that people are interested in cats and celebrities.
But it’s also totally insane to say, “Hey newsroom: act in a vacuum.” Really, it’s about the communication. After you’ve agreed on how you want to measure success, you can celebrate it in a shared team space.
Practically, we’ve seen a few things happen since we started sharing metrics with reporters:
- Happier teams – Reporters are like, “Holy crap, people are reading my thing!”
- Adjusting how we work – We’ve staffed up and down in certain ways. For example, we had a lot more evidence that we needed to hire more graphic designers.
Sacha: You mentioned in a recent talk at the Frank Gathering that one story that performed surprisingly well with Carebot was an extended look at the civil war in Yemen. What have you learned from this and other stories about the kinds of visuals that do – and don’t – work?
Brian: Yeah, that Yemen story was incredible. About 60 percent of people completed the story – one of our highest. It was a long and difficult story. But people cared about it. We copied the shape of that story down the road.
Another thing we changed was of often we “change modes.” What I mean by that is: early on, a common trend in “multimedia storytelling” was one big package, with a long body of text, slideshow, and audio. But when we looked at the completion rates of that kind of stories, we realized that – as soon as you changed modes (from text to video, for example) people would leave.
Now we avoid switching modes, and if we do switch, we’re really careful about it. Take a look at this story about Ebola in Liberia. As you flip through the slides, there’s only ambient sound. If there is any narrative audio, it’s subtitled.
Sacha: So what’s next for Carebot?
Brian: We hope other news organizations will find it useful. We recently participated in a Mozilla Open News event where news outlets hacked the code. The Texas Tribune also recently tested it out.
Our primary goal right now is to make Carebot more modular, so it’s easier to build your own metric in Carebot. So for example, if a news organization wants to measure comments per pageview, we’d make it easier for to build plugins. We’re finishing up process guides and instructions for that right now.
My dream scenario would be to have an open source, WordPress-type community around Carebot. The code is all on Github. Setting up Carebot is pretty easy for a technologist. It takes less than a day to fire up. The much harder thing is figuring out what you want to measure.
Of course, it was harder for us to make Carebot open source. But part of the mission of public radio is to be a public service. Carebot won’t have a long, wonderful life unless other people contribute to it.
Share your stories implementing Carebot or other journalism analytics @Carebot.