This post is solely my opinion; not reflecting any views of my coauthors, my university, etc, and was written in my free time at home. I am just putting my current thoughts in writing, with the hope of stimulating some discussion. My post is based on some ruminations I’ve had over recent years, in which I’ve seen a lot of change happening in how science’s self-correcting process works, and the levels of openness in science, which are trends that seem likely to only get more intense.
That’s what this post ponders- where are we headed and what does it mean for scientists and science? Please stay to the end. It’s a long read, but I hope it is worth it. I raise some points at the end that I feel strongly about, and many people (not just scientists) might also agree with or be stimulated to think about more.
I’ve always tried to be proactive about correcting my (“my” including coauthors where relevant) papers, whether it was a publisher error I spotted or my/our own; I’ve done at least 5 such published corrections. Some of my later papers have “corrected” (by modifying and improving the methods and data) my older ones, to the degree that the older ones are almost obsolete. A key example is my 2002 Nature paper on “Tyrannosaurus rex was not a fast runner“- a well-cited paper that I am still proud of. I’ve published (with coauthors aplenty) about 10 papers since then that explore various strongly related themes, the accuracy of assumptions and estimates involved, and new ways to approach the 2002 paper’s main question. The message of that paper remains largely the same after all those studies, but the data have changed to the extent that it would no longer be viable to use them. Not that this paper was wrong; it’s just we found better ways to do the science in the 12 years since we wrote it.
I think that is the way that most of science works; we add new increments to old ones, and sooner or later the old ones become more historical milestones for the evolution of ideas than methods and data that we rely on anymore. And I think that is just fine. I cannot imagine it being any other way.
If you paid close attention over the past five months, you may have noticed a kerfuffle (to put it mildly) raised by former Microsoft guru/patent afficionado/chef/paleontologist Nathan Myhrvold over published estimates of dinosaur growth rates since the early 2000′s. The paper coincided with some emails to authors of papers in question, and some press attention, especially in the New York Times and the Economist. I’m not going to dwell on the details of what was right or wrong about this process, especially the scientific nuances behind the argument of Myhrvold vs. papers in question. What happened happened. And similar things are likely to happen again to others, if the current climate in science is any clue. More about that later.
But one outcome of this kerfuffle was that my coauthors and I went through (very willingly; indeed, by my own instigation) some formal procedures at our universities for examining allegations of flaws in publications. And now, as a result of those procedures, we issued a correction to this paper:
Hutchinson, J.R., Bates, K.T., Molnar, J., Allen, V., Makovicky, P.J. 2011. A computational analysis of limb and body dimensions in Tyrannosaurus rex with implications for locomotion, ontogeny, and growth. PLoS One 6(10): e26037. doi: 10.1371/journal.pone.0026037 (see explanatory webpage at: http://www.rvc.ac.uk/SML/Projects/3DTrexGrowth.cfm)
The paper correction is here: http://www.plosone.org/article/info%3Adoi/10.1371/journal.pone.0097055. Our investigations found that the growth rate estimates for Tyrannosaurus were not good enough to base any firm conclusions are, so we retracted all aspects of growth rates from that paper. The majority of the paper, about estimating body mass and segment dimensions (masses, centres of mass, inertia) and muscle sizes as well as their changes through growth and implications for locomotor ontogeny, still stands; it was not in question.
For those (most of you!) who have never gone through such a formal university procedure checking a paper, my description of it is that it is a big freakin’ deal! Outside experts may be called in to check the allegations and paper, you have to share all your data with them and go through the paper in great detail, retracing your steps, and this takes weeks or months. Those experts may need to get paid for their time. It is embarassing even if you didn’t make any errors yourself and even if you come out squeaky clean. And it takes a huge amount of your time and energy! My experience started on 16 December, reached a peak right around Xmas eve (yep…), and finally we submitted our correction to PLoS and got editorial approval on 20 March. So it involved three months of part-time but gruelling dissection of the science, and long discussions of how to best correct the problems. Many cooks! I have to admit that personally I found the process very stressful and draining.
Next time you wonder why science can be so slow at self-correction, this is the reason. The formal processes and busy people involved mean it MUST be slow– by the increasingly speedy standards of modern e-science, anyway. Much as doing science can be slow and cautious, re-checking it will be. Should be?
My message from that experience is to get out in front of problems like this, as an author. Don’t wait for someone else to point it out. If you find mistakes, correct them ASAP. Especially if they (1) involve inaccurate data in the paper (in text, figures, tables, whatever), (2) would lead others to be unable to reproduce your work in any way, even if they had all your original methods and data, or (3) alter your conclusions. It is far less excruciating to do it this way then to have someone else force you to do it, which will almost inevitably involve more formality, deeper probing, exhaustion and embarassment. And there is really no excuse that you don’t have time to do it. Especially if a formal process starts. I can’t even talk about another situation I’ve observed, which is ongoing after ~3 years and is MUCH worse, but I’ve learned more strongly than ever that you must demonstrate you are serious and proactive about correcting your work.
I’ve watched other scientists from diverse fields experience similar things– I’m far from alone. Skim Retraction Watch and you’ll get the picture. What I observe both excites me and frightens me. I have a few thoughts.
1) The drive to correct past science is a very good development and it’s what science is meant to be about. This is the most important thing!
2) The digital era, especially trends for open access and open data for papers, makes corrections much easier to discover and do. That is essentially good, and important, and it is changing everything about how we do science. Just watch… “we live in interesting times” encapsulates the many layers of feelings one should react with if you are an active researcher. I would not dare to guess what science will be like in 20 years, presumably when I’ll be near my retirement and looking back on it all!
3) The challenge comes in once humans get involved. We could all agree on the same lofty principles of science and digital data but even then, as complex human beings, we will have a wide spectrum of views on how to handle cases in general, or specific cases.
This leads to a corollary question– what are scientists? And that question is at the heart of almost everything controversial about scientific peer review, publishing and post-publication review/correction today, in my opinion. To answer this, we need to answer at least two sub-questions:
1–Are we mere cogs in something greater, meant to hunker down and work for the greater glory of the machine of science?
(Should scientists be another kind of public servant? Ascetic monks?)
2–Are we people meant to enjoy and live our own lives, making our own choices and value judgements even if they end up being not truly optimal for the greater glory of science?
(Why do we endure ~5-10 years of training, increasingly poor job prospects/security, dwindling research funds, mounting burdens of expectations [e.g., administrative work, extra teaching loads, all leading to reduced freedoms] and exponentially growing bureaucracies? How does our experience as scientists give meaning to our own lives, as recompense?)
The answer is, to some degree, yes to both of the main questions above, but how we reconcile these two answers is where the real action is. And this brew is made all the spicier by the addition of another global trend in academia: the corporatization of universities (“the business model”) and the concomitant, increasing concern of universities about public image/PR and marketing values. I will not go any further with that; I am just putting it out there; it exists.
The answer any person gives will determine how they handle a specific situation in science. You’ve reminded your colleague about possible errors in their work and they haven’t corrected it. Do you tell their university/boss or do you blog and tweet about it, to raise pressure and awareness and force their hand? Or do you continue the conversation and try to resolve it privately at any cost? Is your motive truly the greater glory of science, or are you a competitive (or worse yet, vindictive or bitter) person trying to climb up in the world by dragging others down? How should mentors counsel early career researchers to handle situations like this? Does/should any scientist truly act alone in such a regard? There may be no easy, or even mutually exclusive, answers to these questions.
We’re all in an increasingly complex new world of science. Change is coming, and what that change will be like or when, no one truly knows. But ponder this:
Open data, open science, open review and post-publication review, in regards to correcting/retracting past publications: how far down the rabbit hole do we go?
The dinosaur growth rates paper kerfuffle concerned numerous papers that date back to earlier days of science, when traditions and expectations differed from today’s. Do we judge all past work by today’s standards, and enforce corrections on past work regardless of the standards of its time? If we answer some degree of “yes” to this, we’re in trouble. We approach a reductio ad absurdum: we might logic ourselves into a corner where that great machine of science is directed to churn up great scientific works of their time. Should Darwin’s or Einstein’s errors be corrected or retracted by a formal process like those we use today? Who would do such an insane thing? No one (I hope), but my point is this: there is a risk that is carried in the vigorous winds of the rush to make science look, or act, perfect, that we dispose of the neonate in conjunction with the abstergent solution.
There is always another way. Science’s incremental, self-correcting process can be carried out quite effectively by publishing new papers that correct and improve on old ones, rather than dismantling the older papers themselves. I’m not arguing for getting rid of retractions and corrections. But, where simple corrections don’t suffice, and where there is no evidence of misconduct or other terrible aspects of humanity’s role in science, perhaps publishing a new paper is a better way than demolishing the old. Perhaps it should be the preferred or default approach. I hope that this is the direction that the Myhrvold kerfuffle leans more toward, because the issues at stake are so many, so academic in nature, and so complex (little black/white and right/wrong) that openly addressing them in substantial papers by many researchers seems the best way forward. That’s all I’ll say about that.
I still feel we did the right thing with our T. rex growth paper’s correction. There is plenty of scope for researchers to re-investigate the growth question in later papers. But I can imagine situations in which we hastily tear down our or others’ hard work in order to show how serious we are about science’s great machine, brandishing lofty ideals with zeal– and leaving unfairly maligned scientists as casualties in our wake. I am reminded of outbursts over extreme implementations of security procedures at airports in the USA, which were labelled “security theatre” for their extreme cost, showiness and inconvenience, with negligible evidence of security improvements.
The last thing we want in science is an analogous monstrosity that we might call “scientific theatre.” We need corrective procedures for and by scientists, that serve both science and scientists best. Everyone needs to be a part of this, and we can all probably do better, but how we do it… that is an interesting adventure we are on. I am not wise enough to say how it should happen, beyond what I’ve written here. But…
A symptom of scientific theatre might be a tendency to rely on public shaming of scientists as punishment for their wrongs, or as encouragement for them to come clean. I know why it’s done. Maybe it’s the easy way out; point at someone, yell at them in a passionate tone backed up with those lofty ideals, and the mob mentality will back you up, and they will be duly shamed. You can probably think of good examples. If you’re on social media you probably see a lot of it. There are naughty scientists out there, much as there are naughty humans of any career, and their exploits make a good story for us to gawk at, and often after a good dose of shaming they seem to go away.
But Jon Ronson‘s ponderings of the phenomenon of public shaming got me thinking (e.g., from this WTF podcast episode; go to about 1 hr 9 min): does public shaming belong in science? As Ronson said, targets of severe public shaming have described it as “the worst pain ever”, and sometimes “there’s no recourse” for them. Is this the best way to live together in this world? Is it really worth it, for scientists to do to others or to risk having done to them? What actually are its costs? We all do it in our lives sometimes, but it deserves introspection. I think there are lessons from the dinosaur growth rates kerfuffle to be learned about public shaming, and this is emblematic of problems that science needs to work out for how it does its own policing. I think this is a very, very important issue for us all to consider, in the global-audience age of the internet as well as in context of the intense pressures on scientists today. I have no easy answers. I am as lost as anyone.
What do you think?
EDIT: I am reminded by comments below that 2 other blog posts helped inspire/coagulate my thoughts via the alchemy of my brain, so here they are:
http://dynamicecology.wordpress.com/2014/02/24/post-publication-review-signs-of-the-times/ Which considers the early days of the Myhrvold kerfuffle.
http://blogs.discovermagazine.com/neuroskeptic/2014/01/27/post-publication-cyber-bullying/ Which considers how professional and personal selves may get wounded in scientific exchanges.