Hi there! This is TITLE-ABS-KEY(“science journalism“), a newsletter about science journalism research. In the inaugural issue, I read a paper about reporting science as a process and had some laughs about the academic writing habit of saying It is widely acknowledged and citing sources which hardly acknowledge anything like that.
Today, I’m reading a study of science stories written by AI and by humans and ponder whether it’s not too late to switch from writing to coding. Let’s go!
Today’s paper: Henestrosa, Angelica Lermann, Hannah Greving, and Joachim Kimmerle. Automated journalism: The effects of AI authorship and evaluative information on the perception of a science journalism article. Computers in Human Behavior (2022): 107445. DOI: 10.1016/j.chb.2022.107445
Why this paper: I mean, robots. Every other issue of this newsletter will probably have AI-generated images very soon.
Abstract: Texts produced by artificial intelligence (AI) are becoming increasingly prevalent in digital journalism. Research suggests that these texts do not differ from human-written texts in their perceived credibility or trustworthiness where simple and short text types are concerned. However, it is unclear how AI-written texts beyond simple fact reporting are perceived. Therefore, this research aimed to expand upon the existing literature on automated journalism by investigating the influence of AI authorship (vs. human authorship) and evaluative information presentation (vs. neutral information presentation). The results of three preregistered experimental studies revealed no differences in perceived credibility and trustworthiness between AI-written and human-written texts. However, presenting information in an evaluative way decreased the perception of credibility and trustworthiness. Moreover, the AI was perceived as less anthropomorphic than the human author. The belief in the machine heuristic was stronger for an AI than for a human author, particularly when participants had actually read an article allegedly written by an AI. A pooled analysis across the data of all three studies underpinned the main effect of information presentation. Concluding, we discuss the findings against the background of AI perception theory and suggest implications for future research.
Wow, frankly, from reading the abstract and intro, I am slightly terrified that apparently not only are short and fact-based stories written by AI all over digital journalism (let’s face it, that was kinda the inevitable future way back when I was telling students about the LA Times earthquake bot), but they are also just as credible to the readers. Is it that people aren’t that afraid of the robots despite the best efforts of multiple Hollywood franchises — or that flesh-based journalists are not doing well in the credibility department?
By the way, speaking of things that happened in 2014 when I was just starting out in journalism instruction: [r]egarding the quality of the output, findings suggest that already in the early days of news automation, readers did not seem to be able to distinguish between automatically and human-written texts (Clerwall, 2014).
So the issue is clearly not detecting AI vs human writing; that ship has sailed by now. It’s about whether clearly labeled (as it should be) AI writing feels more or less credible and trustworthy. That’s what the team set out to study, rightly noting that most readers have no idea whatsoever how automated news generation works — and so (this is me) any cursory labeling, without links to extensive explainers, is likely to be even less informative than the “partner content“ or “presents“ labels for native ads.
In general, people tend to be algorithm averse. Algorithm aversion is a phenomenon observed during the emergence of algorithms: People preferred to interact with human agents even though algorithms outperformed humans in many tasks.
We are avenged, fellow flesh-based journalists: readers do dislike machines! Let us free ourselves from the Wordsmith chains and —
As people’s daily experiences have become increasingly digital, this aversion may have decreased. In fact, regarding short and simple texts, algorithms have the advantage of being perceived as writing objectively. This notion fits what Sundar and Kim referred to as the machine heuristic, a mental shortcut indicating that an operating machine is perceived as being objective, accurate, and free from ideological bias. For instance, it has been shown that news selected by a machine were rated more favorably compared to news chosen by a journalist
.
Are you kidding me?… It should be noted, however, that the citation for that example there is for a 2001 paper, i.e. pre-algorithmic news aggregators and pre-renewed appreciation for curation as a journalistic task.
But basically, yes, that machine heuristic likely makes readers feel that a robot is not going to get distracted by a bumblebee and, erm, accidentally typo 2350 into 2035 and embarrass the IPCC. In a meta-analysis, Graefe and Bohlken (2020) compared 12 experimental and descriptive studies that were conducted between 2017 and 2020. They found advantages for human-written content in terms of quality and readability. Experimental evidence also suggests higher credibility when participants simply were told that a human had written the article. Overall and across various topics, however, the meta-analysis found no differences in the perceived credibility of human and AI-written news.
But what if we go beyond facts and figures? Automated sports reporting is already building stories where the narrative itself strongly depends on a complex set of parameters (sometimes the hook is not in who won but rather in individual player stats etc). And here it’s time for little protein-based me to finally feel that vindication: Even though AI might technically outperform humans in the speed and accuracy of written language, genuinely human activities like interpreting, explaining, and evaluating information will not necessarily be perceived as adequate when expressed by AI.
So how are we doing on credibility and trustworthiness here? The little research we do have suggests readers may indeed be wary of AI’s judgement, i.e. when its texts are set to be generated as evaluative rather than objective, there’s a drop in credibility (we want you to get the numbers right but please keep your “opinions“ to yourself, thanks!) — but findings are sparse and inconsistent. Part of this involves the uncanny valley effect: readers are creeped out by AI imitating distinctly human behaviors like referencing personal experience. And yet, of course, we also like to anthropomorphise entities displaying human-like cues, so my non-academic conclusion aligns with the paper: it’s a real mess.
Final point from the paper, though: the mess is also content- and context-dependent, i.e. for readers, AI may feel less equipped to handle complex and/or controversial topics where expert opinions are often not just welcome but necessary to make sense of the facts.
To investigate the mess, the team designed and pre-registered three studies where (human) readers rated stories, both automatically generated and written entirely by human journalists — but also different in terms of style neutrality — on their credibility and trustworthiness. In one of the studies, participants also rated the perceived intelligence and anthropomorphism of human and AI authors.
The stories were about wolves in Germany and autonomous driving. They were all in German in the experiments but you can check out the English translations in the supplementary materials. Here’s a taste of the autonomous driving story:
All in all, the challenges of autonomous driving are by no means impossible, and the opportunities far outweigh them — provided people recognize them. In the long run, autonomous driving will definitely make life easier, more environmentally friendly, and, above all, enormously safer.
Well, I guess this could have been written by AutomatedTXT, a fictional (?) news algorithm which supports its driving brethren. So what happened?
Contrary to our expectations, and even though readers might not be familiar with algorithms writing opinionated texts, credibility and trustworthiness did not decrease when an alleged AI author presented the information about a scientific topic in an evaluative way.
Balderdash! (okay, I confess, I googled ‘polite alternatives to shit‘ and came across a list of 50 useful swear word replacements)
Moreover, we did not find any differences between the human and the AI author when the article was written neutrally. Taken together, in all three studies and the additional analysis of the pooled data, which would have been able to detect even very small differences, only information presentation influenced the main dependent variables, regardless of the declared authorship.
Barnacles!
But then it gets interesting: people did rate AutomatedTXT consistently as less anthropomorphic than its German colleagues with a heartbeat. This leads the authors to believe that it is possible that the participants ultimately did not care who wrote a text, even though they were apparently aware that the authors differed from one another.
So, are readers free of carbon chauvinism? Do they truly not care if the life form that produced their science news is based on the same chemical element as they are? Fascinating stuff. Moreover, more elaborate labels explaining the specific nature of news automation did not affect the readers’ perceptions, so apparently it’s not media literacy either?
Eh, perhaps not on that last one. Apart from the actual research, authors also asked their participants for comments after the studies. There were only very few participants who doubted the existence of the AI author. At the same time, some comments expressed admiration of the AI instead. However, as a few participants also wondered whether the AI was only performing copy and paste or if a human had preselected the information, future research should further pursue what understanding of AI authorship readers have in mind.
Finally, the authors did find evidence of the machine heuristic in action and did see a drop in credibility for evaluative writing — but both for AI and humans. (We should all keep our opinions to ourselves, apparently.)
Wow, this has been a wild ride! (No algorithms were involved in writing this issue, and authors of the paper also disclose no robotic coauthorship.)
That’s it! If you enjoyed this issue, let me know. If you also have opinions or would like to suggest a paper for me to read in one of the next issues, you can leave a comment or just respond to the email.
Cheers! 👩🔬