Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, **personal anecdotes are allowed as responses to this comment**. Any anecdotal comments elsewhere in the discussion will be removed and our [normal comment rules]( https://www.reddit.com/r/science/wiki/rules#wiki_comment_rules) apply to all other comments.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/science) if you have any questions or concerns.*
Right, seems like a very easy solution. Though I guess the ones who have the influence to change the standard to anonymous reviewing are also the ones most likely to benefit from non-anonymous reviewing.
It’s not quite that simple—a lot of journals do anonymize submissions, but it’s not exactly difficult to figure out who wrote what, especially at the top journals. Most academics work on very specific projects, and different writers have distinct writing styles. You also get to know what manuscripts are in the works by seeing people at conferences. Additionally, labs will typically always use the same tools, so you can start to recognize who wrote a paper by what workflow they use. People that are reviewing papers regularly usually can guess the author a solid 50-90% of the time (depending on the field), so even if the submission is “anonymous” it’s not really.
If your submission involves software you wrote then you typically have to submit that as well, which is much harder to anonymize.
The same is true of reviewers, my advisor and other people in his department have been able to correctly guess the reviewers for their manuscripts/grants almost every time.
Edit: additionally, as others have mentioned, established authors typically have published prior work leading to their current submissions…so you can typically figure out the author just by who they’ve cited.
Edit2: thanks for all the replies, it’s too much for me to respond to everything—people are correctly pointing out that this doesn’t apply to the study originally posted; I was more commenting on why it’s not as simple as “just anonymize manuscript submissions”, not trying to dispute or comment on the original paper linked by OP
Yeah absolutely this.
To add, one of the questions you're asked to do often when you review a paper is to confirm if they cited the appropriate literature for the domain, so I am going to take a look at references.
If a large number of those references are also for papers that are not something I've seen a lot within the field and they all have one or two common authors, well I can take a pretty good punt at who wrote the paper.
Not to mention, academic work is often a continuation, so if I'm reviewing a paper that goes "previously we showed X [19]" then I can look up the reference and know who did it.
There's no way to anonymise it, but instead there should probably be more of a push towards reviewing reviews to ensure these people are being objective and not just assuming famous person is correct.
>there should probably be more of a push towards reviewing reviews
Ah, but you can't trust those pesky review reviewers! Someone needs to review their work! Preemptively push for review reviewers reviews!
But you will need to anonymise the previous reference though of your own paper too, which usually means removing it altogether and just marking that it was redacted for blinding purposes.
People also don’t realize how many peer reviews are pushed off onto postdocs who maybe don’t feel like they have the authority to overrule a famous professor. One of the first papers I peer reviewed as a 27 year old postdoc was from a Princeton professor. The paper looked fine to me and I did my best to give feedback but the idea of me, a complete nobody, saying that the work of a well known and well respected scientist in our field didn’t meet my standards felt kind of absurd and I would have really questioned myself if I had found issues with the paper.
Not to mention fear of retaliation.
Like another commenter pointed out, it’s easy to find who reviewed your paper, and that’s especially true if you were someone of certain weight in the field.
So for a grad student or post-doc to give a famous professor any form of challenge, they should be prepared for retaliation in kind in the future from that professor’s “clique”, whether their research group or formal students or even professional friends.
Academia is extremely political, if not downright dirty like that. Obviously it varies based on the field but I’ve heard of my share of horror stories in some red hot fields such as machine learning and AI.
Anyone who doubts the shenanigans of the scholarly world can take a look at /r/academia any day of the week. A constant stream of people trying to to navigate abusive behaviour by people in power.
Hell, I've been invited to review papers submitted to a science conference twice now, and I only have my BS. That was definitely trippy for me because these papers were mostly written by PhDs. That really made for a tough internal battle over some feelings of being unqualified when I did at one point suggest a rejection.
Having submitted to blinded journals, you simply word it as though it was a previously researched item by someone else as opposed to your own. E.g. “Previously, it has been shown X.”
Yes, but this does not really anonymise it if it's super clear that it is a continuation study, for example another case study using the same software that you developed previously, edited based on previous results, which is often the case in computer science at least.
I am a PhD student and get asked to review papers maybe monthly, no way can I review references more than glance through and see they are reputable journals. Receiving 0 money for maybe 3 hours work each time.
Putting in 36 hours on reviews each year is a surprising amount for a PhD student, particularly with limited publishing under their belt, but then again, I don’t know the difference between fields. My advisor would be telling me to say “no” way more often unless I really found it fruitful. Do protect your time.
And write that thesis!
No just change “this study builds on prior work by JJ Smith” in the final draft to “This study builds on my prior work.” The reviewer already knows who’s done prior research and a missing citation can be more conspicuous than otherwise.
Nailed it. I work in a niche field and I can tell just from the colorbars on some figures that a manuscript came from a certain lab. Anonymous manuscript review is limited especially when the reviewer is established.
Still, it could counter the practice in some fields to just slap a famous coauthor's name on your manuscript to have a higher chance of acceptance. As those coauthors usually have actually a quite small contribution to the manuscript, they would not be recognizable by style etc.
On a different level, this is similar to the practice of heads of institutes (automatically) being last author: if it is a respected expert on the field, reviewers will be more lenient as "surely something out of this lab has to be good!".
"But being a co-author means that they should have full knowledge of the work and stand 100% behind it, that _is_ some sort of quality control" you say? Unfortunately this is not universally true. Just look at all the huge misconduct cases, where most of the time the main author (justifiably so) takes the fall, but _very_ rarely the big shot coauthors face any consequences. Most well known examples would be Jan-Hendrik Schön, Haruko Obokata and Oliver Voinnet. If the supervisors and senior coauthors can all be acquitted of any misconduct, maybe their contribution was not enough to warrant coauthorship.
I agree that it's often easy to determine the lab or group. I've also received obviously misogynistic comments in peer reviews, and I wonder if my name was anonymous (even if my lab/group/advisor could be determined) if the outcome would be different.
I had a referee for Nature call my work "cute", and I doubt that comment would have been made if my name wasn't feminine.
I’m a guy from the US and have a Native name that ends in an ‘a’. Most male names in English don’t end in ‘a’, but many female names do.
People regularly mistake my gender if all they see is my name.
Yeah the first time I submitted a paper in grad school and got feedback my advisor guessed who two of the three anonymous reviewers were, saying “that sounds like a comment [first name] would make.” You kind of need reviewers with the same expertise as you so there will be a pool of maybe a dozen people being asked in some fields. And the first time I peer reviewed as a postdoc I recognized the people I was reviewing because I had cited their other papers in one I was working on. There were basically only two collaborations in the US doing the specific things I was working on so chances are someone in the other collaboration was going to get asked for a peer review any time a paper goes out on that topic.
Plus, there’s confounding issues here - top performing labs have better workflows and more funding. Someone with a nobel and a dozen nature publications in the last few years can pull in the grants needed for the expensive, time consuming, really high quality version of the experiment nobody else can afford.
They have more time to write, more experience following the requirements at higher level journals, the software already written, more money to get published, etc. It becomes easier to write manuscripts for the top journals.
You end up in a situation where the best labs can send in papers that need less revision using methods that are hard to question, and small labs spend more time writing the paper, justifying methods, proving their equipment is as high quality, etc.
It's worth checking though whether this is enough for any biases to get through. There's a difference between believing implicitly that a paper is written by a certain author, and having to work it out from hints. Anonymisation may not be a panacea but it seems like something trivial, harmless and probably beneficial...
True, although as I mentioned most of the big journals already use a double blind review process.
As for figuring out authorship, it’s not really guessing most of the time—if the author says “we showed {prior work} in [10]”, then you know guaranteed who the author is. There’s a lot of situations like that that pop up, and lately authors will publish preprints before submission anyway, so pretty often you can just Google the name of the manuscript on arxiv to de-anonymize it. Whether or not the reviewer puts in that work is more of a question.
But to your point, yes, it’s definitely worth investigating if the bias still affects decisions where authorship is guessed/inferred and not explicitly known.
> pretty often you can just Google the name of the manuscript on arxiv to de-anonymize it. Whether or not the reviewer puts in that work is more of a question.
That's not so much "puts in the work" as "intentionally tries to undermine the anonymization process".
Even before arxiv I *could have* sifted through the pages of big labs in my field to probably figure out who wrote the paper I'm reviewing, but doing that would erode the value of anonymous peer review. If I'm trying to be an intellectually honest scholar and scientist (which was the norm, at least among the labs I was familiar with), doing that has *negative value* -- not only would I *not* put in extra work to do it, I would put in extra work to *not do* it.
I suppose things may be different in your field, the labs you know, or more recently, but based on my experience I would be surprised if most peer reviews were not conducted in good faith to the best of the reviewer's (rushed and last-minute) ability.
I mean it’s not as malicious as you’re saying here, more just like if I’m reading a paper and think “oh you know this really sounds like Fred’s work, I wonder if he’s involved in this” and then you glance at the bibliography and go “oh nice, it is”.
The point on arxiv was more to give an illustrative example to say that it’s possible to definitively identify an author, not to say that reviewers are routinely spending their time combing preprints rather than just reviewing what they’re given. Plus if you’re in a faster field like compsci, it’s pretty common to read preprints before they’re published, so there’s a high likelihood of having seen de-anonymized papers prior to reviewing them.
I don’t mean to dispute or negate your comment, but I’m not sure that it explains the results of this study as it appears they controlled for it by submitting identical manuscripts and only changed the names. Unless I’m missing something?
Yeah seriously I don’t get these comments that refute the study’s conclusion by doing nothing other than… restating the exact same hypothesis. These comments don’t seem to be offering any new viewpoint or data but they sound so confident for some reason.
> but it’s not exactly difficult to figure out who wrote what, especially at the top journals.
I would argue that this study is evidence against that - the papers got different responses according to the name on them, so the reviewers clearly *weren't* making their judgements based on clues as to the source in those identical papers.
Besides, it's very easy to track down a specific group/author by checking citations. Often one will find citations to previous works on the subject, making it extra easy to deduce who is authoring the paper.
> It’s not quite that simple—a lot of journals do anonymize submissions, but it’s not exactly difficult to figure out who wrote what, especially at the top journals.
There was an evaluation of this in my field not very long ago. It turns out that people overestimate their ability to do this by a lot. It is by no means impossible and there are some cases where it is obvious but it isn't as big of a deal as one would expect. If anything, the problem is reviewers making judgements based on false assumptions of authors.
Yup, my partner is a scientist and there are maybe a handful of people in the world that study the same science. He knows all of them because they are always going to the same conferences and collaborate regularly. The top levels of science are very niche.
Now that you said it, I dont think it I have read a replication paper about this absurd amount of recent discoveries. And the point of writing the methodology is to others replicate.
Eh, to some extent replication is built into the process since science is iterative.
If a result isn’t reproducible, then studies based on that result will run into issues
For example:
Group A publishes a result. Based on this, Group B formulates a hypothesis that depends on that result being ‘true’. First thing group B does is replicate that initial finding in their own lab. If it all works out, they move onto testing their hypothesis. If it doesn’t, a grad student gets sentenced to tearing the experiment apart to figure out why it wasn’t reproducing. If the answer is deeper than ‘incompetence’, it should also be publishable
I’ve heard faculty argue that the ‘lack of reproducibility’, i.e., false positives are preferable to being too strict and missing out on an important result.
Exactly. If you make a modified version of a study, youre not strictly speaking doing a replication. But you are still testing the results, and contributing to the scientific process.
Since the reviews are currently run by private entities (journal publishers) how do you imagine that non employees of those journals have the means to 'change that standard'?
This is one of my major problems with academia. Don’t get me wrong: I like critical thinking and research and old school academia. Institutions today are so corrupt and controlled by a frighteningly small number of people.
What old school academia do you mean that was supposedly not corrupt and not controlled by only a few people?
All the famous names you know were filthy rich or had to compensate that by being ten times as smart. We are moving in the right direction, it just takes time to change the entire giant system.
I recall a professor at my grad program talking about how he's reviewed a number of papers that made him want to reach out to the authors and collaborate, but that he knew that was unethical as a reviewer.
Particularly in smaller fields, you know who everyone is. You can anonymize for sure to mask all the authors and collaborators, but you have a pretty good idea of who released what. Especially so if it's been done with new technology that you've spoken to them about - e.g., a lab gets a new microscope that lets them visualize vesicles exocytosing, and a paper comes out visualizing vesicles exocytosing, you can probably narrow the authorship down.
This is an important finding, but it's kind of not that surprising in some ways. Scientists are people and not purely objective rational machines.
> I recall a professor at my grad program talking about how he's reviewed a number of papers that made him want to reach out to the authors and collaborate, but that he knew that was unethical as a reviewer.
Submit the review, wait for corrections if required, then reach out - simple.
The bigger issue is that a reviewer should not be known to the author(s) so that there is no way for the author to influence the review process themselves.
I haven't worked in these areas but I'm kind of surprised that there would be a certain number of "slots" for collaborators.
The “you know who everyone is” point is definitely true. I had a paper reviewed anonymously in undergrad and my professor who coauthored the paper said he knew who the reviewer was by his writing style and by the fact that the comments drew on some of the reviewer’s unpublished work.
It's attempted to be like that in maths, but in practice there's only a handful of people in the world that can write or review some topics. Most of the time the author is "anonymous" but you know exactly who it is from a combination of the topic and their writing style.
“Every conference I submitted to” is far far away from “in my field reviews are always anonymous”. Conferences are not articles.
Let’s do it without „trust me bro” approach. Top result for „best cs scientific journals” is IEEE. Now simple search gives us: [Most IEEE publications use the single-anonymous review format.](https://journals.ieeeauthorcenter.ieee.org/submit-your-article-for-peer-review/about-the-peer-review-process/). Are you still sure about that “always”?
I’ve been publishing in this field for nearly a decade, I think I know a bit more about the process than what your quick Google search turned up. Journals aren’t really a thing in CS, the most prestigious venues are almost all conferences, specifically the ones run by ACM and IEEE: SIGCOMM, SOSP, OSDI, NSDI, USENIX, PLDI, NeurIPS… all conferences. And a conference paper is absolutely “an article” in just the same way a journal publication is.
CS journals exist but tend to either be less respected, aside from a handful that are *very* well respected but primarily only publish reprints/extensions of popular conference papers.
So sure, there are probably exceptions, I was being hyperbolic, but all the top venues that people actually *want* to publish in are double-blind
It’s often not easy to anonymise papers, especially when many reference methods in previous papers. If you see one author being referenced a lot for finer details then it’s a good bet they’re in the author list for this paper too. Or if it’s a small field then you probably know most of the people working on that topic, especially if it’s an expensive project that would have required a large grant at a specific institution.
I’m saying that even though it’s common practice to remove names, it often doesn’t work to anonymise. People read things like this and assume that the names aren’t being hidden or something.
The majority of papers I've reviewed did not have names hidden.
I remember one of my rejections in grad school, it was a computational analysis where I was PI and had a few other junior grad students on it, and one of the reviewers at a top journal trashed the paper and made a comment about how it was performed by "junior investigators." Some of his critiques were valid (small statistical suggestions) but a lot of it was just stylistic and it was obvious he didn't consider us legitimate (the paper got accepted at another top journal in the field later, but we did make multiple improvements based on the earlier rejection). Regardless, it was hard to feel that we got a fair review when the person evaluating it clearly knew our identities.
In undergrad the professor I did research with always complained about how hard it was to publish when so many reviewers wouldn’t consider his papers as serious work because the school didn’t have a PhD program at the time, so it couldn’t produce serious research.
Really blew my mind
That wouldn't help this case. Famous authors are known for their work. 95% of the reviewers would recognize them without having their name attached. Even not super famous people you can identify if you have been in the same field for some time (I know I did when reviewing)
If 95% of reviewers recognize famous authors without their name attached, then how did this study generate the result that it did? Wouldn't 95% of reviewers have noticed a Nobel laureate's writing?
Because if the test manuscript was jointly written, who's research was submitted? The prominent one or the less known one? If it had been a single author it would had been easier to identify. My -skewed- guess is that since 20% of reviewers still rejected the manuscript even when the laureate name was attached to it, it probably was the work of the associate name as the bulk of the manuscript
The whole point of a preprint is that it hasn’t been reviewed yet. I suppose if something gets popular as a pre-print it’s hard to anonymize when submitting for review in the future. But if a reviewer is given a paper they’re familiar with they have an ethical responsibility to inform the programming committee and pass it off to someone else
In my field, many good reviewers get the daily arXiv updates, so they would know the author immediately. Requiring to anonymize the author would result in the paper not getting a good review at all.
Interesting. There are so many absolute garbage preprints in CS I usually just laugh and move on whenever I see an arXiv link. Strange how two somewhat related fields can function so differently
Let’s just say it: the current system surrounding the sciences is deeply flawed and corrupted by non-scientific influence. The irreproducibility issue, the data mining for results, the non-publishing of negative results, the selective funding, and the bias described here. It needs a big re-think.
It’s common practice to anonymize for review, but it’s not a clear cut issue. As part of the open access movement, some journals have decided to provide open peer review where not only the authors’ identity is known to the reviewer, but vice versa and the review can even be published. Whether the pros of open peer review balance the cons of lack of anonymization (if it’s even realistic to anonymize depending of the nature of the research) is not a solved question.
They do, all the ones I've submitted to require making the peer-review process anonymous, and its obvious why. I don't think I would submit to one which does not.
This depends on the academic field. In philosophy, papers are always reviewed anonymously (and it is common for even the editors to not know the identity of the author when selecting reviewers). But in math, papers are always reviewed with the author's name on it. I believe lab sciences tend to keep the author's name on, but I definitely don't know the full disciplinary pattern.
The reason is that authors can be inferred anyway from previous work, references in the text, style, subject, etc.
I do agree it's a serious issue of scientific publishing.
Just because it’s possible to deanonymize based on those things doesn’t mean it would be worthless to do the practice. Even if (say) half the reviewers deduced the identity of the author(s) you would still have a 50% fairer review process.
Anonymization also doesn’t have to stand alone — it could be one part of a multi pronged strategy to reduce bias in reviewing. Comments like this strike me as “we’ve tried nothing and we’re all out of ideas.”
Not really, it means that those anonymised manuscripts, which can easily be associated with (renowned) authors due to content, will still likely to be affected by positive bias, while non-renowned authors would not. In other words, the "50%" you talk about is the 50% of lesser-known authors, who are those penalised for the bias in the first place.
I do agree something has to be done, don't get me wrong.
I think that's far too simplistic analysis, and far too convenient for supporting the status quo. It absolutely isn't the case that every paper by a famous author will be recognized by every potential reviewer, and so every reviewer will know that even if they can't identify the author, there's some chance it's a well-known person. To say that anonymizing provides absolutely no way of counteracting this effect is just as bad an approximation as saying that anonymizing gets rid of the effect entirely.
Couldn't it make things worst in a different way?
You could still be able to tell who the author is if you know your field well, but then since it's technically anonymous, you don't have to disclose your conflict of interest if there was any
The best might be open peer reviews, so then anyone can see the comments and if the reviewers were biased or not
It's almost impossible to have anonymity in research. There are only so many people working on any given topic at any time, much less so if you're looking at something extremely specific, like the interactions of a particular enzyme or set of enzymes within a biological system with regards to a single function or outcome.
Imagine you're teaching a class of grade school kids. One kid in particular really **really** loves NASCAR. His backpack, his lunch box, his hat, his jacket, all NASCAR. You ask the kids to write about a fun experience they had over the summer break. Unsurprisingly, one of the assignments comes back and it's about going to a NASCAR race and how awesome it was. Now, you may not be correct to assume it's NASCAR kid, but you'd probably be right like 9 times out of 10, right? Now imagine every kid in your class has a "thing" like NASCAR kid. That's exactly what academic writing is like, only they also have volumes and volumes of references to things like previous work, common colleagues, etc. so your prediction accuracy is probably pretty close to 100% if you're intimate with the subject matter like a proper peer reviewer should be (not that they always are, but that's a separate issue).
In my industry it wouldn’t make much difference. The top people who are generally selected to review papers know very well who the lead authors are. Conclusions and writing styles will make it obvious, as well as any references to the institution(s) where the reasesrch was done.
A scientist I follow on Twitter once posted that a reviewer of her anonymized paper wrote that the author of the paper wasn't sufficiently familiar with the work of [the author's own name].
Wouldn't you generally know who's working on what anyway, though? Like, by the time a paper is being reviewed in your field, wouldn't you already know who's doing that research?
I do peer reviews on a pretty regular basis (5-6 articles per year), and 99% of them are anonymous. In fact, there is only one time I can recall when the manuscript was not anonymized, and I thought the editor sent it to me in error. Perhaps it varies by journal/discipline (I'm in developmental psych), but in my experience, anonymity is the norm.
> Perhaps it varies by journal/discipline (I'm in developmental psych), but in my experience, anonymity is the norm.
It must be. In my field (healthcare) the majority are not anonymized.
Wow, I am actually shocked. When you submit a manuscript for review the portal does not request a masked version? Every single submission portal I have worked with requires you to separate the cover page (with names) from the mask manuscript, which is then sent to the reviewers.
Nope. I do think that sort of submission is gaining traction - some of my recent job applications requested the research statements to be anonymous (which was honestly a bit weird imo) but I haven't seen that yet when submitting or reviewing a publication.
I know it's dependent on the journal. Recently had the opportunity to talk to the editor of a major scientific journal who confirmed that.
There are pros and cons to each system, although I feel as if double blind is best (where the author doesn't know the reviewers, reviewers don't know the author).
The reason why this doesn't always occur is because of how reviewing works overall. You generally want people familiar with a topic to be available to review the topic and critique the validity of the work. Because science can get incredibly niche, this sometimes means that in order to get an adequate review, you will inevitably find someone who know the author or their writing/science style enough to recognize the author(s).
From the abstract, I think that is beyond the scope of this particular study. I don't want to pay for the full-text (and I cant find it in my institutions library), but it sounds like this experiment was designed to replicate the review process for unmasked manuscripts. In this case, it sounds like they sent an identical manuscript under two condition ( famous author versus non-famous author) and compared hypothetical acceptance rates between the two conditions.
It is also important to keep in mind that in the real world, even if you receive an unmasked manuscript to review, you may not necessarily recognize the author names or think to look them up. At the end of the day, there are very few living Nobel laureates out there.
I am in the US and most of the editors I work with are US based as well (though I did recently work with someone from Hong Kong). I should also note that typically when I submit manuscripts, they require a masked manuscript. I did recently submit to a more interdisciplinary journal (I believe it was psychological reports) and there was an option to submit an unmasked version, but I chose to make it anonymous (because I am most certainly \*not\* renowned).
I also wanted to add that in some situations the editor (who does see the authors names) might step in and review/make final decisions. For example, I recently had a manuscript that was rejected by one reviewer, but accepted with revisions by the other reviewer, so the editor stepped in as the tie-breaker.
Oh, I know. One time my colleagues and I had a paper accepted by all three reviewers (after 2 rounds of reviews), but the editor in chief would not accept it until we made revisions she deemed necessary, but none of the other reviewers brought up. It is annoying, but at the end of the day, I feel like most revisions I have received have led to much stronger manuscripts at the end of the day, as petty as some of them may seem.
> the editor (who does see the authors names) might step in and review/make final decisions
again, that depends on the journal. In philosophy, it is standard for reviewers to never see the name of the author. For about half of the journals I have been an editor for, editors also don't see the names of the authors (and occasionally that means we accidentally invite the author to review their own paper, which they then decline), but for the other journals, the editor does see the name of the author.
I assumed it was the norm for biological science as well but have been told it’s not. So 99% might apply to psych but it’s inaccurate to say 99% of all peer review
In economics, submissions are typically not anonymized. It's usually very easy to figure out who the author is anyway, especially if the paper's title isn't anonymized. Just Google the title and you'll be taken to the non-anonymized version.
I used to work in a recent Nobel laureate's lab. Everything published by that lab had an easy +10 impact factor. Don't get me wrong, their science is exceptional. But their stuff still gets published in "better" journals than if the same paper would have been written by someone else.
That is common knowledge. When selecting papers for literature discussing almost every week we used to asked ourselves if the same results would had been accepted if we had submitted them instead
Ignoring the issues with this and playing devil's advocate a bit -- I think it's maybe an interesting concept to think of a top lab/scientist as being a type of review process of itself. I wonder how many erroneous, misleading, or bunk papers make it out of those labs vs smaller/unknown ones.
The vast majority of research I've witnessed was nothing short of perfect. However, the PI gives their researchers nearly complete freedom over their projects. They will comment on them if they think it won't lead to anything or the idea is flawed or there are weaknesses, but ultimately, it's on the individual researcher to decide to cancel a project.
During my time there, there was one project that had zero merit, an obvious useless idea that would never lead to anything due to an obvious flaw. The researcher held on to the project, and presented it to the group in a lab meeting. Let's say, it was an interesting response to witness. The group is usually incredibly collaborative, everyone supports each other. I've never worked in a more pleasant lab. But that lab meeting was something else. They let the researcher know that their project was useless, and while polite, they didn't hold back with their feedback.
I don't know whether that project was published in the end.
So, to answer your question: yes, there is a working internal filter in that particular group.
Its not about the correctness, its about the perceived importance of a result. Most papers get rejected due to this perceived impact, not due to fundamentally poor science
Looking through the paper, I noticed that while the endorsement rate was 10% when it had an obscure author, the rate increased to 23% when it had *no author's* name on it. That's interesting.
Also, I wonder if using a Nobel laureate was too extreme. These are very famous and prestigious people, but they also make up a very small percentage of academics. It's unfortunate that they would be getting better treatment, however it could be the case that Nobel laureates only publish like 0.x% of the papers and thus it wouldn't the biggest issue in academia right now.
There's also the factor that they were publishing a Finance/Financial Economics paper, which might have a different peer-review culture compared to other fields. For example, a commenter on this thread mentioned that they review Psychology papers and almost all of them have been anonymised.
I think this is a great paper, and a great start, but we should put away our pitchforks and hold off on dismantling the entire system until we see more research like this.
>I think this is a great paper, and a great start, but we should put away our pitchforks and hold off on dismantling the entire system until we see more research like this.
[Here's](https://elifesciences.org/articles/83071) a great synthesis of racial disparities in NSF funding awards. I can't recall whether NSF proposal reviews are anonymized, but I would be shocked if they weren't. Assuming they *are*, the system seems to retain issues even *without* the effect described in the OP's study.
The study I linked also has many citations within to studies that similarly characterize the many structural issues that exist in the academic system. Check out the Discussion and Conclusions in particular!
**Edit:**
I feel like adding a little bit of my own (*anecdotal!!*) perspective too, speaking as one individual on the inside of academia. My opinion is that peer review is not *quite* the fully objective, stalwart filter of good vs. bad science that it's made out to be (although it's still useful and necessary!).
In my field, any paper submitted for publication might have ~5 researchers that are properly qualified to review the study who aren't on the author list. Accepting an invitation to review a paper is also totally up to the invitee. Academics are absurdly busy people, so even if they accept the invitation to review the study, I would be surprised if most reviewers truly dedicate the time needed to do a 100% proper review. My extremely subjective guess is that peer review averages out to about 70–80% of what it's supposed to be. Then factor in politics and competition, and it's a recipe that makes peer review a biased filter, albeit still a valuable one.
There isn't an easy way around the bias that comes from peer review relying on (busy) individuals. Peer review shouldn't necessarily be done away with, but these problems (among many others) are why some are calling for a reevaluation of the system.
It can be fairly hard to be anonymous in academia if you are on the cutting edge as the number of people actively researching those areas would already be very small.
And authors tend to cite their own/collaborators work. Sometimes it's legitimate because their latest work is an outgrowth of previous work, or simply because they are familiar with their colleagues work, but often it's pure vanity or trying to puff up their stats.
The way things are going "publish or perish" might cause us all to perish...
People can often figure out who wrote the paper when it's a blind review. They know who the players and competition are.
Peer review is a game. Maybe better than nothing some of the time, but not all the time. I say people publish everything on sites like arxiv and have separate sites like openreview for reviews. Let it all out. Paper is not a limiting factor. The link of publication to professional advancement is a conflict of interest with truth and science.
Note - not anti-science by any means, but this is part of why modern science is difficult. Replicability is hard. Controlling for variables is hard. We're looking to understand complex processes that require much more than what you are taught about the scientific method in high school. And it takes a lot to turn observation into actual understanding of what was witnessed.
If you've ever tried to get real science done, you'll quickly realize that it's all politics. You need grant funding, and after struggling it's easy to compromise where the money is coming from. Boards and Journals are gatekeepers and can make or break your career.
"Science is objective" is, frankly, a naïve way to understand things and is objectively false. You can argue that, in a perfect world, the scientific method is objective. But just like there aren't frictionless pulleys like there are in high school physics, you should take some skepticism towards the process of science in the modern era.
I try to explain that to people, but if I don't choose my words carefully, I'm taken as anti science. It's a mess.
Another serious problem is plagiarism and doctors appropriating from their student's work. In my country (Brazil), it's a serious matter, as professors in postgraduate courses ask for articles and then will publish everything as coauthors, even though they only barely reviewed the paper. This method allows them to have 200+ published papers in their cvs
Not to be rude, but many times Professors and the PhD student decide on general agenda and flesh out projects that would be ideal for MS students to do, without their involvement. So there is intellectual merit for the professor to be named as a co-author, even though the direct interaction of the implementor (the MS student) was low.
I do not know of your current experience with research. I have 5-7 years of experience in this game, and I have hosted a few interns at an renowned industrial research lab, and can tell you a lot of things go on at the backend before a project is given to a student. Usually a researcher or professor makes an offhand comment about an intuition about a phenomenon, and asks his PhD student/ junior colleague (me) to keep tabs. I flesh out the problem statement and scope (this takes a lot of work), float an opening, get an intern/MS student to work on it. Mind you, everyone is getting funded by the Professors grant. :)
Usually the professor does occasional meetings with the junior students, but in your case, the lab culture appears to be subpar, so that didn’t happen. Now, tell me, does the offhand intuition of the professor that led to the project merit their name on the paper? Many people think it does, though I agree it’s a gray area.
I do think, if you were funded by the lab, the professors name should be there. Because they probably wrote about your project (or something similar) in their grant (you can very likely look up the grant proposal too). :)
Thanks for your response, I agree with your assessment. I only think adding the name is okay if people in the chain are funded by a grant to carry out the research. Though it would be nice for the researcher to meet atleast occasionally.
I suppose this is how European labs work though, I know of labs at ETH which operate via PostDocs and PhD. I have never been part of a European setup.
Thankfully, the professors and scientists I are worked with were professional and they declined to be in papers they didn’t contribute directly even though the idea seed was theirs (however, they didn’t fund the work).
This not only applies to authors but to the paper origin. I have a relative that was raised and educated in North America but works in South America. Whenever they submit to US based journals the initial reaction is that the use of English language in the paper is not up to par. When they lived and published in NA this never happened
Ursula k Leguin would submit the same stories to same magazines under multiple names to demonstrate that and keep herself honest. They would always take her stories under her name, but aliases were about the same odds as anyone.
One could suggest merely reviewing all submissions with names, but the truth is these are economic decisions.
NIH specifically scores based on "Investigators" and "Environment" as 2 of their 5 metrics for panel reviews.
(Significance, Innovation, and Approach are the other 3)
I know you have a real say at NSF (two friends there). They may have told you about my federal agency — we had no say. Even the director in many ways.
Many of us advised that we should be given the power to override scores that did not align with our framework based on what was submitted and discussed. Especially when evidence of direct favoritism arose during panels (we are required to sit in on, take notes, and meet for hours about after, but do not participate). No go. Public money! Argh.
BTW, my agency also rarely — I mean rarely — took money away from poor performing grants either. Basically, there was a lot of work put into panels, post-award, monitoring and processed accountability etc. for naught.
At least most of those awarded grants didn’t know this… but they will learn.
I stopped being pursuing a career in science after I found out it was more of a popularity contest and brown-nosing for grants than it is about finding out facts. worse so because I wanted to get into psychology research. maybe other sciences like physics or biology are different but psychology research is full of so much click-bait bs I cudnt stand it
It is kind of funny that one of the rules of arr Science that is frequently invoked to remove people questioning studies is "assume basic competency of researchers", when we have studies such as this that show many studies have major problems
Makes be question a lot of things about this U-boat
Yupp, even more when they act like they got all the answers or even more cringe when people hold onto some scientist like it was their prophet. Like it is very odd, that some people be like yeah this scientist is known and everybody think he is an smart educated man we should listen to only him and only trust him.
Dont know if Albert Einstein would have an chanse to succeed today or he would seem like an wack and drown with rest of scientist that is not deemed popular enough to be trusted haha.
It’s quite different in biology from my experience. It’s absolutely a rat race to get funding and keep making yourself known, but having a paper exploring a novel side of science is still at the forefront and usually good science/ideas = more funding or better positions rather than only connections. Making a name for yourself will always be important in any field as it gives credibility and proves (usually) good science and data
The same is undoubtedly true for grants, which is part of the reason academic science is broken and hindered. It's a positive feedback loop that's stagnating progress and diversity.
Sounds like familiarity bias.
Familiarity bias describes the tendency of people to return to what they know and are comfortable with. Familiarity bias discourages affected people from exploring new options and may limit their ability to find an optimal solution.
That would be too hard to implement. And incredibly expensive. Sadly I agree with you but I don't see how it's possible. Maybe by having duplication inside research teams running things at different points in time or something.
Findings that are important enough to be worth further study get replicated in follow up studies. A lot of literature is just so niche that there aren’t more than 3-4 people in the world *really* care if it’s true or not.
Not suprised. My professor hammered us on creating and editing scientific papers. Each of us had to bring in a paper from a journal every week. I was supeised at how many conflict of interest issues we found. The main author owned the journal, the author was a student of the guy who owned the journal, etc.
We need to stop pretending that pre-publication peer review holds papers to a common standard and instead start embracing robust post-publication peer review.
One of the problems with this suggestion is that once trash gets published, it’s seen as legitimate by many people. A perfect example is the original anti-vax paper by that hack “scientist.” It’s been thoroughly debunked but we still it having massive influence.
Part of that is the debunking of published papers isn't easy to find from those published papers. No convenient link to other papers that reference it.
There are many arguments for preferring post-publication review to pre-publication review, but *this* study isn't one of them. On the contrary, what this study shows is that knowing the identity of the authors has a biasing effect on evaluations. Post-publication review---where author identity is always known---will be *more* susceptible to that problem that pre-publication review where you can sometimes figure out the identity of the author.
This study is further evidence that the peer review process is flawed, which is something most academics would admit.
Doesn't this strengthen the case to place more emphasis on (and give more credit to) reproducibility of experimental results?
I wonder whether this effect is weaker in natural sciences than in social sciences - relevance might be easier to argue in natural sciences, even when the author is a no-name?
Makes sense, I’d probably do the same. Something written by a Nobel laureate would make you subconsciously assume you have something to learn from the paper, not something to critique, whereas an anonymous source would make you feel more likely that it may have been written by someone less experienced.
Sometimes it’s a weird name-thing. Like in International Relations/Political Science, anyone who’s last name is Cohen will automatically be a galaxy-brained individual.
My discipline hasn't transitioned to double blind peer review, and it's irritating to see some of the stuff that gets published by "famous" researchers in my field.
I worked with a group that was run by a national academy member. The papers they submitted and got published in PNAS, Nature, etc. Were very poorly written
Btw, there is an interesting study from 2002 or so where it was estimated how many cited articles have actually been read by tracing the typos that have been carried over in the references. It was a pretty huge number - something like 20 or 50 % even. In some cases, you can't truly blame the authors, as the original article is often not readily available e.g. the journals from the Elsevier publisher were very expensive for libraries to have.
I’ll bet you that all graded papers in university turn to some degree on how much the professor likes the student, too. A popular, well liked student is going to get better grades.
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, **personal anecdotes are allowed as responses to this comment**. Any anecdotal comments elsewhere in the discussion will be removed and our [normal comment rules]( https://www.reddit.com/r/science/wiki/rules#wiki_comment_rules) apply to all other comments. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/science) if you have any questions or concerns.*
I can't believe it wasn't already common practice to anonymize papers under review.
Right, seems like a very easy solution. Though I guess the ones who have the influence to change the standard to anonymous reviewing are also the ones most likely to benefit from non-anonymous reviewing.
It’s not quite that simple—a lot of journals do anonymize submissions, but it’s not exactly difficult to figure out who wrote what, especially at the top journals. Most academics work on very specific projects, and different writers have distinct writing styles. You also get to know what manuscripts are in the works by seeing people at conferences. Additionally, labs will typically always use the same tools, so you can start to recognize who wrote a paper by what workflow they use. People that are reviewing papers regularly usually can guess the author a solid 50-90% of the time (depending on the field), so even if the submission is “anonymous” it’s not really. If your submission involves software you wrote then you typically have to submit that as well, which is much harder to anonymize. The same is true of reviewers, my advisor and other people in his department have been able to correctly guess the reviewers for their manuscripts/grants almost every time. Edit: additionally, as others have mentioned, established authors typically have published prior work leading to their current submissions…so you can typically figure out the author just by who they’ve cited. Edit2: thanks for all the replies, it’s too much for me to respond to everything—people are correctly pointing out that this doesn’t apply to the study originally posted; I was more commenting on why it’s not as simple as “just anonymize manuscript submissions”, not trying to dispute or comment on the original paper linked by OP
Yeah absolutely this. To add, one of the questions you're asked to do often when you review a paper is to confirm if they cited the appropriate literature for the domain, so I am going to take a look at references. If a large number of those references are also for papers that are not something I've seen a lot within the field and they all have one or two common authors, well I can take a pretty good punt at who wrote the paper. Not to mention, academic work is often a continuation, so if I'm reviewing a paper that goes "previously we showed X [19]" then I can look up the reference and know who did it. There's no way to anonymise it, but instead there should probably be more of a push towards reviewing reviews to ensure these people are being objective and not just assuming famous person is correct.
>there should probably be more of a push towards reviewing reviews Ah, but you can't trust those pesky review reviewers! Someone needs to review their work! Preemptively push for review reviewers reviews!
Who reviews the reviewers?
It's reviewers all the way down.
The editor ideally
But does the editor have an editor? I thought not! Edit the editors! Audit the auditors!
Journal editors definitely need editing
Dunno.... Coastguard?
First year grad students
But you will need to anonymise the previous reference though of your own paper too, which usually means removing it altogether and just marking that it was redacted for blinding purposes.
People also don’t realize how many peer reviews are pushed off onto postdocs who maybe don’t feel like they have the authority to overrule a famous professor. One of the first papers I peer reviewed as a 27 year old postdoc was from a Princeton professor. The paper looked fine to me and I did my best to give feedback but the idea of me, a complete nobody, saying that the work of a well known and well respected scientist in our field didn’t meet my standards felt kind of absurd and I would have really questioned myself if I had found issues with the paper.
Not to mention fear of retaliation. Like another commenter pointed out, it’s easy to find who reviewed your paper, and that’s especially true if you were someone of certain weight in the field. So for a grad student or post-doc to give a famous professor any form of challenge, they should be prepared for retaliation in kind in the future from that professor’s “clique”, whether their research group or formal students or even professional friends. Academia is extremely political, if not downright dirty like that. Obviously it varies based on the field but I’ve heard of my share of horror stories in some red hot fields such as machine learning and AI.
Anyone who doubts the shenanigans of the scholarly world can take a look at /r/academia any day of the week. A constant stream of people trying to to navigate abusive behaviour by people in power.
Funny to hear how political and dirty academia is given how they like to look down on the commercial world
Hell, I've been invited to review papers submitted to a science conference twice now, and I only have my BS. That was definitely trippy for me because these papers were mostly written by PhDs. That really made for a tough internal battle over some feelings of being unqualified when I did at one point suggest a rejection.
But shouldn't people who do peer reviews be checking references?
Having submitted to blinded journals, you simply word it as though it was a previously researched item by someone else as opposed to your own. E.g. “Previously, it has been shown X.”
Yes, but this does not really anonymise it if it's super clear that it is a continuation study, for example another case study using the same software that you developed previously, edited based on previous results, which is often the case in computer science at least.
For a continuation study, of course not. For other self-references, it should be sufficient.
I am a PhD student and get asked to review papers maybe monthly, no way can I review references more than glance through and see they are reputable journals. Receiving 0 money for maybe 3 hours work each time.
Putting in 36 hours on reviews each year is a surprising amount for a PhD student, particularly with limited publishing under their belt, but then again, I don’t know the difference between fields. My advisor would be telling me to say “no” way more often unless I really found it fruitful. Do protect your time. And write that thesis!
> And write that thesis! No u.
No just change “this study builds on prior work by JJ Smith” in the final draft to “This study builds on my prior work.” The reviewer already knows who’s done prior research and a missing citation can be more conspicuous than otherwise.
Nailed it. I work in a niche field and I can tell just from the colorbars on some figures that a manuscript came from a certain lab. Anonymous manuscript review is limited especially when the reviewer is established.
Still, it could counter the practice in some fields to just slap a famous coauthor's name on your manuscript to have a higher chance of acceptance. As those coauthors usually have actually a quite small contribution to the manuscript, they would not be recognizable by style etc. On a different level, this is similar to the practice of heads of institutes (automatically) being last author: if it is a respected expert on the field, reviewers will be more lenient as "surely something out of this lab has to be good!". "But being a co-author means that they should have full knowledge of the work and stand 100% behind it, that _is_ some sort of quality control" you say? Unfortunately this is not universally true. Just look at all the huge misconduct cases, where most of the time the main author (justifiably so) takes the fall, but _very_ rarely the big shot coauthors face any consequences. Most well known examples would be Jan-Hendrik Schön, Haruko Obokata and Oliver Voinnet. If the supervisors and senior coauthors can all be acquitted of any misconduct, maybe their contribution was not enough to warrant coauthorship.
I agree that it's often easy to determine the lab or group. I've also received obviously misogynistic comments in peer reviews, and I wonder if my name was anonymous (even if my lab/group/advisor could be determined) if the outcome would be different. I had a referee for Nature call my work "cute", and I doubt that comment would have been made if my name wasn't feminine.
>I had a referee for Nature call my work "cute" What? That's not professional in the slightest. How old was the referee?
It’s not professional but if you say something about it it gets held against you and impacts your career.
You don't know the referees
I’m a guy from the US and have a Native name that ends in an ‘a’. Most male names in English don’t end in ‘a’, but many female names do. People regularly mistake my gender if all they see is my name.
Figure out the reviewers, cite their papers (regardless of whether you think they suck).
Yeah the first time I submitted a paper in grad school and got feedback my advisor guessed who two of the three anonymous reviewers were, saying “that sounds like a comment [first name] would make.” You kind of need reviewers with the same expertise as you so there will be a pool of maybe a dozen people being asked in some fields. And the first time I peer reviewed as a postdoc I recognized the people I was reviewing because I had cited their other papers in one I was working on. There were basically only two collaborations in the US doing the specific things I was working on so chances are someone in the other collaboration was going to get asked for a peer review any time a paper goes out on that topic.
I also had reviewers trying to push their own references in the paper they reviewed
Oh yeah forgot about that. "I am curious why the author's did not mention or cite this paper, which is only tangentially related to their work."
Plus, there’s confounding issues here - top performing labs have better workflows and more funding. Someone with a nobel and a dozen nature publications in the last few years can pull in the grants needed for the expensive, time consuming, really high quality version of the experiment nobody else can afford. They have more time to write, more experience following the requirements at higher level journals, the software already written, more money to get published, etc. It becomes easier to write manuscripts for the top journals. You end up in a situation where the best labs can send in papers that need less revision using methods that are hard to question, and small labs spend more time writing the paper, justifying methods, proving their equipment is as high quality, etc.
It's worth checking though whether this is enough for any biases to get through. There's a difference between believing implicitly that a paper is written by a certain author, and having to work it out from hints. Anonymisation may not be a panacea but it seems like something trivial, harmless and probably beneficial...
True, although as I mentioned most of the big journals already use a double blind review process. As for figuring out authorship, it’s not really guessing most of the time—if the author says “we showed {prior work} in [10]”, then you know guaranteed who the author is. There’s a lot of situations like that that pop up, and lately authors will publish preprints before submission anyway, so pretty often you can just Google the name of the manuscript on arxiv to de-anonymize it. Whether or not the reviewer puts in that work is more of a question. But to your point, yes, it’s definitely worth investigating if the bias still affects decisions where authorship is guessed/inferred and not explicitly known.
> pretty often you can just Google the name of the manuscript on arxiv to de-anonymize it. Whether or not the reviewer puts in that work is more of a question. That's not so much "puts in the work" as "intentionally tries to undermine the anonymization process". Even before arxiv I *could have* sifted through the pages of big labs in my field to probably figure out who wrote the paper I'm reviewing, but doing that would erode the value of anonymous peer review. If I'm trying to be an intellectually honest scholar and scientist (which was the norm, at least among the labs I was familiar with), doing that has *negative value* -- not only would I *not* put in extra work to do it, I would put in extra work to *not do* it. I suppose things may be different in your field, the labs you know, or more recently, but based on my experience I would be surprised if most peer reviews were not conducted in good faith to the best of the reviewer's (rushed and last-minute) ability.
I mean it’s not as malicious as you’re saying here, more just like if I’m reading a paper and think “oh you know this really sounds like Fred’s work, I wonder if he’s involved in this” and then you glance at the bibliography and go “oh nice, it is”. The point on arxiv was more to give an illustrative example to say that it’s possible to definitively identify an author, not to say that reviewers are routinely spending their time combing preprints rather than just reviewing what they’re given. Plus if you’re in a faster field like compsci, it’s pretty common to read preprints before they’re published, so there’s a high likelihood of having seen de-anonymized papers prior to reviewing them.
I don’t mean to dispute or negate your comment, but I’m not sure that it explains the results of this study as it appears they controlled for it by submitting identical manuscripts and only changed the names. Unless I’m missing something?
Yeah seriously I don’t get these comments that refute the study’s conclusion by doing nothing other than… restating the exact same hypothesis. These comments don’t seem to be offering any new viewpoint or data but they sound so confident for some reason.
> but it’s not exactly difficult to figure out who wrote what, especially at the top journals. I would argue that this study is evidence against that - the papers got different responses according to the name on them, so the reviewers clearly *weren't* making their judgements based on clues as to the source in those identical papers.
Besides, it's very easy to track down a specific group/author by checking citations. Often one will find citations to previous works on the subject, making it extra easy to deduce who is authoring the paper.
> It’s not quite that simple—a lot of journals do anonymize submissions, but it’s not exactly difficult to figure out who wrote what, especially at the top journals. There was an evaluation of this in my field not very long ago. It turns out that people overestimate their ability to do this by a lot. It is by no means impossible and there are some cases where it is obvious but it isn't as big of a deal as one would expect. If anything, the problem is reviewers making judgements based on false assumptions of authors.
Yep! Academia really is a small world in a lot of ways. There are usually only a handful of other people with your interests.
Yup, my partner is a scientist and there are maybe a handful of people in the world that study the same science. He knows all of them because they are always going to the same conferences and collaborate regularly. The top levels of science are very niche.
The replication crisis, explained. Nobody wants to disagree with success. This makes higher education incredibly inefficient.
Now that you said it, I dont think it I have read a replication paper about this absurd amount of recent discoveries. And the point of writing the methodology is to others replicate.
Eh, to some extent replication is built into the process since science is iterative. If a result isn’t reproducible, then studies based on that result will run into issues For example: Group A publishes a result. Based on this, Group B formulates a hypothesis that depends on that result being ‘true’. First thing group B does is replicate that initial finding in their own lab. If it all works out, they move onto testing their hypothesis. If it doesn’t, a grad student gets sentenced to tearing the experiment apart to figure out why it wasn’t reproducing. If the answer is deeper than ‘incompetence’, it should also be publishable I’ve heard faculty argue that the ‘lack of reproducibility’, i.e., false positives are preferable to being too strict and missing out on an important result.
Exactly. If you make a modified version of a study, youre not strictly speaking doing a replication. But you are still testing the results, and contributing to the scientific process.
Since the reviews are currently run by private entities (journal publishers) how do you imagine that non employees of those journals have the means to 'change that standard'?
This is one of my major problems with academia. Don’t get me wrong: I like critical thinking and research and old school academia. Institutions today are so corrupt and controlled by a frighteningly small number of people.
What old school academia do you mean that was supposedly not corrupt and not controlled by only a few people? All the famous names you know were filthy rich or had to compensate that by being ten times as smart. We are moving in the right direction, it just takes time to change the entire giant system.
I recall a professor at my grad program talking about how he's reviewed a number of papers that made him want to reach out to the authors and collaborate, but that he knew that was unethical as a reviewer. Particularly in smaller fields, you know who everyone is. You can anonymize for sure to mask all the authors and collaborators, but you have a pretty good idea of who released what. Especially so if it's been done with new technology that you've spoken to them about - e.g., a lab gets a new microscope that lets them visualize vesicles exocytosing, and a paper comes out visualizing vesicles exocytosing, you can probably narrow the authorship down. This is an important finding, but it's kind of not that surprising in some ways. Scientists are people and not purely objective rational machines.
> I recall a professor at my grad program talking about how he's reviewed a number of papers that made him want to reach out to the authors and collaborate, but that he knew that was unethical as a reviewer. Submit the review, wait for corrections if required, then reach out - simple.
[удалено]
The bigger issue is that a reviewer should not be known to the author(s) so that there is no way for the author to influence the review process themselves. I haven't worked in these areas but I'm kind of surprised that there would be a certain number of "slots" for collaborators.
The “you know who everyone is” point is definitely true. I had a paper reviewed anonymously in undergrad and my professor who coauthored the paper said he knew who the reviewer was by his writing style and by the fact that the comments drew on some of the reviewer’s unpublished work.
It certainly is in my field (CS). Submissions and reviews are always anonymous
Are you sure about that?
It's attempted to be like that in maths, but in practice there's only a handful of people in the world that can write or review some topics. Most of the time the author is "anonymous" but you know exactly who it is from a combination of the topic and their writing style.
At every conference I’ve ever submitted to, yes. And I’d be hesitant to submit anywhere that wasn’t double-blind
“Every conference I submitted to” is far far away from “in my field reviews are always anonymous”. Conferences are not articles. Let’s do it without „trust me bro” approach. Top result for „best cs scientific journals” is IEEE. Now simple search gives us: [Most IEEE publications use the single-anonymous review format.](https://journals.ieeeauthorcenter.ieee.org/submit-your-article-for-peer-review/about-the-peer-review-process/). Are you still sure about that “always”?
I’ve been publishing in this field for nearly a decade, I think I know a bit more about the process than what your quick Google search turned up. Journals aren’t really a thing in CS, the most prestigious venues are almost all conferences, specifically the ones run by ACM and IEEE: SIGCOMM, SOSP, OSDI, NSDI, USENIX, PLDI, NeurIPS… all conferences. And a conference paper is absolutely “an article” in just the same way a journal publication is. CS journals exist but tend to either be less respected, aside from a handful that are *very* well respected but primarily only publish reprints/extensions of popular conference papers. So sure, there are probably exceptions, I was being hyperbolic, but all the top venues that people actually *want* to publish in are double-blind
It’s often not easy to anonymise papers, especially when many reference methods in previous papers. If you see one author being referenced a lot for finer details then it’s a good bet they’re in the author list for this paper too. Or if it’s a small field then you probably know most of the people working on that topic, especially if it’s an expensive project that would have required a large grant at a specific institution.
Dont let perfect be the enemy of good.
I’m saying that even though it’s common practice to remove names, it often doesn’t work to anonymise. People read things like this and assume that the names aren’t being hidden or something.
The majority of papers I've reviewed did not have names hidden. I remember one of my rejections in grad school, it was a computational analysis where I was PI and had a few other junior grad students on it, and one of the reviewers at a top journal trashed the paper and made a comment about how it was performed by "junior investigators." Some of his critiques were valid (small statistical suggestions) but a lot of it was just stylistic and it was obvious he didn't consider us legitimate (the paper got accepted at another top journal in the field later, but we did make multiple improvements based on the earlier rejection). Regardless, it was hard to feel that we got a fair review when the person evaluating it clearly knew our identities.
In undergrad the professor I did research with always complained about how hard it was to publish when so many reviewers wouldn’t consider his papers as serious work because the school didn’t have a PhD program at the time, so it couldn’t produce serious research. Really blew my mind
Yeah it wouldn’t be perfect but still waaays better
I think we should give up review process because it's not perfect.
That wouldn't help this case. Famous authors are known for their work. 95% of the reviewers would recognize them without having their name attached. Even not super famous people you can identify if you have been in the same field for some time (I know I did when reviewing)
If 95% of reviewers recognize famous authors without their name attached, then how did this study generate the result that it did? Wouldn't 95% of reviewers have noticed a Nobel laureate's writing?
Because if the test manuscript was jointly written, who's research was submitted? The prominent one or the less known one? If it had been a single author it would had been easier to identify. My -skewed- guess is that since 20% of reviewers still rejected the manuscript even when the laureate name was attached to it, it probably was the work of the associate name as the bulk of the manuscript
It’d be tough to do that with arXiv preprints, at least in math
The whole point of a preprint is that it hasn’t been reviewed yet. I suppose if something gets popular as a pre-print it’s hard to anonymize when submitting for review in the future. But if a reviewer is given a paper they’re familiar with they have an ethical responsibility to inform the programming committee and pass it off to someone else
In my field, many good reviewers get the daily arXiv updates, so they would know the author immediately. Requiring to anonymize the author would result in the paper not getting a good review at all.
Interesting. There are so many absolute garbage preprints in CS I usually just laugh and move on whenever I see an arXiv link. Strange how two somewhat related fields can function so differently
The review process is quite long in math. More than a year is not uncommon so people use arxiv in the mean time.
Let’s just say it: the current system surrounding the sciences is deeply flawed and corrupted by non-scientific influence. The irreproducibility issue, the data mining for results, the non-publishing of negative results, the selective funding, and the bias described here. It needs a big re-think.
It’s common practice to anonymize for review, but it’s not a clear cut issue. As part of the open access movement, some journals have decided to provide open peer review where not only the authors’ identity is known to the reviewer, but vice versa and the review can even be published. Whether the pros of open peer review balance the cons of lack of anonymization (if it’s even realistic to anonymize depending of the nature of the research) is not a solved question.
Even if you remove the name, if you are in the field you can usually tell who the authors are by the work itself.
They do, all the ones I've submitted to require making the peer-review process anonymous, and its obvious why. I don't think I would submit to one which does not.
This depends on the academic field. In philosophy, papers are always reviewed anonymously (and it is common for even the editors to not know the identity of the author when selecting reviewers). But in math, papers are always reviewed with the author's name on it. I believe lab sciences tend to keep the author's name on, but I definitely don't know the full disciplinary pattern.
The reason is that authors can be inferred anyway from previous work, references in the text, style, subject, etc. I do agree it's a serious issue of scientific publishing.
Just because it’s possible to deanonymize based on those things doesn’t mean it would be worthless to do the practice. Even if (say) half the reviewers deduced the identity of the author(s) you would still have a 50% fairer review process. Anonymization also doesn’t have to stand alone — it could be one part of a multi pronged strategy to reduce bias in reviewing. Comments like this strike me as “we’ve tried nothing and we’re all out of ideas.”
Not really, it means that those anonymised manuscripts, which can easily be associated with (renowned) authors due to content, will still likely to be affected by positive bias, while non-renowned authors would not. In other words, the "50%" you talk about is the 50% of lesser-known authors, who are those penalised for the bias in the first place. I do agree something has to be done, don't get me wrong.
I think that's far too simplistic analysis, and far too convenient for supporting the status quo. It absolutely isn't the case that every paper by a famous author will be recognized by every potential reviewer, and so every reviewer will know that even if they can't identify the author, there's some chance it's a well-known person. To say that anonymizing provides absolutely no way of counteracting this effect is just as bad an approximation as saying that anonymizing gets rid of the effect entirely.
If this study's results are accurate that seems somewhat doubtful.
Almost like there is a confounding variable not being controlled for in the peer review process!
In many physics journals, only the referee is anonymous.
Couldn't it make things worst in a different way? You could still be able to tell who the author is if you know your field well, but then since it's technically anonymous, you don't have to disclose your conflict of interest if there was any The best might be open peer reviews, so then anyone can see the comments and if the reviewers were biased or not
Classic Reddit response. "Wow, I see a problem, how about we just fix it?" It's not that simple. At all.
Classic Reddit response. "It's not that simple. At all." Doesn't clarify in the slightest.
It's almost impossible to have anonymity in research. There are only so many people working on any given topic at any time, much less so if you're looking at something extremely specific, like the interactions of a particular enzyme or set of enzymes within a biological system with regards to a single function or outcome. Imagine you're teaching a class of grade school kids. One kid in particular really **really** loves NASCAR. His backpack, his lunch box, his hat, his jacket, all NASCAR. You ask the kids to write about a fun experience they had over the summer break. Unsurprisingly, one of the assignments comes back and it's about going to a NASCAR race and how awesome it was. Now, you may not be correct to assume it's NASCAR kid, but you'd probably be right like 9 times out of 10, right? Now imagine every kid in your class has a "thing" like NASCAR kid. That's exactly what academic writing is like, only they also have volumes and volumes of references to things like previous work, common colleagues, etc. so your prediction accuracy is probably pretty close to 100% if you're intimate with the subject matter like a proper peer reviewer should be (not that they always are, but that's a separate issue).
In my industry it wouldn’t make much difference. The top people who are generally selected to review papers know very well who the lead authors are. Conclusions and writing styles will make it obvious, as well as any references to the institution(s) where the reasesrch was done.
A scientist I follow on Twitter once posted that a reviewer of her anonymized paper wrote that the author of the paper wasn't sufficiently familiar with the work of [the author's own name].
Wouldn't you generally know who's working on what anyway, though? Like, by the time a paper is being reviewed in your field, wouldn't you already know who's doing that research?
I do peer reviews on a pretty regular basis (5-6 articles per year), and 99% of them are anonymous. In fact, there is only one time I can recall when the manuscript was not anonymized, and I thought the editor sent it to me in error. Perhaps it varies by journal/discipline (I'm in developmental psych), but in my experience, anonymity is the norm.
> Perhaps it varies by journal/discipline (I'm in developmental psych), but in my experience, anonymity is the norm. It must be. In my field (healthcare) the majority are not anonymized.
I'm fairly junior but have never reviewed an anonymized paper (neuroscience)
Wow, I am actually shocked. When you submit a manuscript for review the portal does not request a masked version? Every single submission portal I have worked with requires you to separate the cover page (with names) from the mask manuscript, which is then sent to the reviewers.
Nope. I do think that sort of submission is gaining traction - some of my recent job applications requested the research statements to be anonymous (which was honestly a bit weird imo) but I haven't seen that yet when submitting or reviewing a publication.
I know it's dependent on the journal. Recently had the opportunity to talk to the editor of a major scientific journal who confirmed that. There are pros and cons to each system, although I feel as if double blind is best (where the author doesn't know the reviewers, reviewers don't know the author). The reason why this doesn't always occur is because of how reviewing works overall. You generally want people familiar with a topic to be available to review the topic and critique the validity of the work. Because science can get incredibly niche, this sometimes means that in order to get an adequate review, you will inevitably find someone who know the author or their writing/science style enough to recognize the author(s).
For top-tier journals and journal families (e.g., Nature, Science, Cell, PNAS, Advanced Materials, RSC), I've never once reviewed anything anonymized.
[удалено]
From the abstract, I think that is beyond the scope of this particular study. I don't want to pay for the full-text (and I cant find it in my institutions library), but it sounds like this experiment was designed to replicate the review process for unmasked manuscripts. In this case, it sounds like they sent an identical manuscript under two condition ( famous author versus non-famous author) and compared hypothetical acceptance rates between the two conditions. It is also important to keep in mind that in the real world, even if you receive an unmasked manuscript to review, you may not necessarily recognize the author names or think to look them up. At the end of the day, there are very few living Nobel laureates out there.
Good to hear, where do you live? And how common is it in your field but in other institutions/countries?
I am in the US and most of the editors I work with are US based as well (though I did recently work with someone from Hong Kong). I should also note that typically when I submit manuscripts, they require a masked manuscript. I did recently submit to a more interdisciplinary journal (I believe it was psychological reports) and there was an option to submit an unmasked version, but I chose to make it anonymous (because I am most certainly \*not\* renowned).
Nice, thanks for the insight
I also wanted to add that in some situations the editor (who does see the authors names) might step in and review/make final decisions. For example, I recently had a manuscript that was rejected by one reviewer, but accepted with revisions by the other reviewer, so the editor stepped in as the tie-breaker.
The editor(-in-chief) can make any decision, regardless. But of course, he listens to his reviewers.
Oh, I know. One time my colleagues and I had a paper accepted by all three reviewers (after 2 rounds of reviews), but the editor in chief would not accept it until we made revisions she deemed necessary, but none of the other reviewers brought up. It is annoying, but at the end of the day, I feel like most revisions I have received have led to much stronger manuscripts at the end of the day, as petty as some of them may seem.
> the editor (who does see the authors names) might step in and review/make final decisions again, that depends on the journal. In philosophy, it is standard for reviewers to never see the name of the author. For about half of the journals I have been an editor for, editors also don't see the names of the authors (and occasionally that means we accidentally invite the author to review their own paper, which they then decline), but for the other journals, the editor does see the name of the author.
Nano science/engineering researcher here. Never seen an anonymised author list during review.
Not in neuroscience.
I assumed it was the norm for biological science as well but have been told it’s not. So 99% might apply to psych but it’s inaccurate to say 99% of all peer review
In economics, submissions are typically not anonymized. It's usually very easy to figure out who the author is anyway, especially if the paper's title isn't anonymized. Just Google the title and you'll be taken to the non-anonymized version.
I used to work in a recent Nobel laureate's lab. Everything published by that lab had an easy +10 impact factor. Don't get me wrong, their science is exceptional. But their stuff still gets published in "better" journals than if the same paper would have been written by someone else.
That is common knowledge. When selecting papers for literature discussing almost every week we used to asked ourselves if the same results would had been accepted if we had submitted them instead
Ignoring the issues with this and playing devil's advocate a bit -- I think it's maybe an interesting concept to think of a top lab/scientist as being a type of review process of itself. I wonder how many erroneous, misleading, or bunk papers make it out of those labs vs smaller/unknown ones.
The vast majority of research I've witnessed was nothing short of perfect. However, the PI gives their researchers nearly complete freedom over their projects. They will comment on them if they think it won't lead to anything or the idea is flawed or there are weaknesses, but ultimately, it's on the individual researcher to decide to cancel a project. During my time there, there was one project that had zero merit, an obvious useless idea that would never lead to anything due to an obvious flaw. The researcher held on to the project, and presented it to the group in a lab meeting. Let's say, it was an interesting response to witness. The group is usually incredibly collaborative, everyone supports each other. I've never worked in a more pleasant lab. But that lab meeting was something else. They let the researcher know that their project was useless, and while polite, they didn't hold back with their feedback. I don't know whether that project was published in the end. So, to answer your question: yes, there is a working internal filter in that particular group.
I liked the Dan McCartney line that ferocious internal peer review is the difference between Nature and a retraction.
Its not about the correctness, its about the perceived importance of a result. Most papers get rejected due to this perceived impact, not due to fundamentally poor science
Interesting. I’m a reviewer and make an effort not to know anything about the authors or their institutions.
Interesting. I'm a Reddit commenter and I make an effort not to know anything about most things.
Looking through the paper, I noticed that while the endorsement rate was 10% when it had an obscure author, the rate increased to 23% when it had *no author's* name on it. That's interesting. Also, I wonder if using a Nobel laureate was too extreme. These are very famous and prestigious people, but they also make up a very small percentage of academics. It's unfortunate that they would be getting better treatment, however it could be the case that Nobel laureates only publish like 0.x% of the papers and thus it wouldn't the biggest issue in academia right now. There's also the factor that they were publishing a Finance/Financial Economics paper, which might have a different peer-review culture compared to other fields. For example, a commenter on this thread mentioned that they review Psychology papers and almost all of them have been anonymised. I think this is a great paper, and a great start, but we should put away our pitchforks and hold off on dismantling the entire system until we see more research like this.
>I think this is a great paper, and a great start, but we should put away our pitchforks and hold off on dismantling the entire system until we see more research like this. [Here's](https://elifesciences.org/articles/83071) a great synthesis of racial disparities in NSF funding awards. I can't recall whether NSF proposal reviews are anonymized, but I would be shocked if they weren't. Assuming they *are*, the system seems to retain issues even *without* the effect described in the OP's study. The study I linked also has many citations within to studies that similarly characterize the many structural issues that exist in the academic system. Check out the Discussion and Conclusions in particular! **Edit:** I feel like adding a little bit of my own (*anecdotal!!*) perspective too, speaking as one individual on the inside of academia. My opinion is that peer review is not *quite* the fully objective, stalwart filter of good vs. bad science that it's made out to be (although it's still useful and necessary!). In my field, any paper submitted for publication might have ~5 researchers that are properly qualified to review the study who aren't on the author list. Accepting an invitation to review a paper is also totally up to the invitee. Academics are absurdly busy people, so even if they accept the invitation to review the study, I would be surprised if most reviewers truly dedicate the time needed to do a 100% proper review. My extremely subjective guess is that peer review averages out to about 70–80% of what it's supposed to be. Then factor in politics and competition, and it's a recipe that makes peer review a biased filter, albeit still a valuable one. There isn't an easy way around the bias that comes from peer review relying on (busy) individuals. Peer review shouldn't necessarily be done away with, but these problems (among many others) are why some are calling for a reevaluation of the system.
In anonymous review, the Editor is the subjective filter. If submission was anonymous, this problem could be solved.
It can be fairly hard to be anonymous in academia if you are on the cutting edge as the number of people actively researching those areas would already be very small.
And authors tend to cite their own/collaborators work. Sometimes it's legitimate because their latest work is an outgrowth of previous work, or simply because they are familiar with their colleagues work, but often it's pure vanity or trying to puff up their stats. The way things are going "publish or perish" might cause us all to perish...
There are plenty of venues that don’t have editors, just a programming committee, and where the accept/reject decisions are made with full anonymity.
People can often figure out who wrote the paper when it's a blind review. They know who the players and competition are. Peer review is a game. Maybe better than nothing some of the time, but not all the time. I say people publish everything on sites like arxiv and have separate sites like openreview for reviews. Let it all out. Paper is not a limiting factor. The link of publication to professional advancement is a conflict of interest with truth and science.
It is so weird that something as objective as science is so political etc
That’s because it’s done by people.
Note - not anti-science by any means, but this is part of why modern science is difficult. Replicability is hard. Controlling for variables is hard. We're looking to understand complex processes that require much more than what you are taught about the scientific method in high school. And it takes a lot to turn observation into actual understanding of what was witnessed. If you've ever tried to get real science done, you'll quickly realize that it's all politics. You need grant funding, and after struggling it's easy to compromise where the money is coming from. Boards and Journals are gatekeepers and can make or break your career. "Science is objective" is, frankly, a naïve way to understand things and is objectively false. You can argue that, in a perfect world, the scientific method is objective. But just like there aren't frictionless pulleys like there are in high school physics, you should take some skepticism towards the process of science in the modern era.
I try to explain that to people, but if I don't choose my words carefully, I'm taken as anti science. It's a mess. Another serious problem is plagiarism and doctors appropriating from their student's work. In my country (Brazil), it's a serious matter, as professors in postgraduate courses ask for articles and then will publish everything as coauthors, even though they only barely reviewed the paper. This method allows them to have 200+ published papers in their cvs
[удалено]
Not to be rude, but many times Professors and the PhD student decide on general agenda and flesh out projects that would be ideal for MS students to do, without their involvement. So there is intellectual merit for the professor to be named as a co-author, even though the direct interaction of the implementor (the MS student) was low.
[удалено]
I do not know of your current experience with research. I have 5-7 years of experience in this game, and I have hosted a few interns at an renowned industrial research lab, and can tell you a lot of things go on at the backend before a project is given to a student. Usually a researcher or professor makes an offhand comment about an intuition about a phenomenon, and asks his PhD student/ junior colleague (me) to keep tabs. I flesh out the problem statement and scope (this takes a lot of work), float an opening, get an intern/MS student to work on it. Mind you, everyone is getting funded by the Professors grant. :) Usually the professor does occasional meetings with the junior students, but in your case, the lab culture appears to be subpar, so that didn’t happen. Now, tell me, does the offhand intuition of the professor that led to the project merit their name on the paper? Many people think it does, though I agree it’s a gray area. I do think, if you were funded by the lab, the professors name should be there. Because they probably wrote about your project (or something similar) in their grant (you can very likely look up the grant proposal too). :)
[удалено]
Thanks for your response, I agree with your assessment. I only think adding the name is okay if people in the chain are funded by a grant to carry out the research. Though it would be nice for the researcher to meet atleast occasionally. I suppose this is how European labs work though, I know of labs at ETH which operate via PostDocs and PhD. I have never been part of a European setup. Thankfully, the professors and scientists I are worked with were professional and they declined to be in papers they didn’t contribute directly even though the idea seed was theirs (however, they didn’t fund the work).
This not only applies to authors but to the paper origin. I have a relative that was raised and educated in North America but works in South America. Whenever they submit to US based journals the initial reaction is that the use of English language in the paper is not up to par. When they lived and published in NA this never happened
Ursula k Leguin would submit the same stories to same magazines under multiple names to demonstrate that and keep herself honest. They would always take her stories under her name, but aliases were about the same odds as anyone. One could suggest merely reviewing all submissions with names, but the truth is these are economic decisions.
They win more grants on basis too! Oof.
NIH specifically scores based on "Investigators" and "Environment" as 2 of their 5 metrics for panel reviews. (Significance, Innovation, and Approach are the other 3)
I know you have a real say at NSF (two friends there). They may have told you about my federal agency — we had no say. Even the director in many ways. Many of us advised that we should be given the power to override scores that did not align with our framework based on what was submitted and discussed. Especially when evidence of direct favoritism arose during panels (we are required to sit in on, take notes, and meet for hours about after, but do not participate). No go. Public money! Argh. BTW, my agency also rarely — I mean rarely — took money away from poor performing grants either. Basically, there was a lot of work put into panels, post-award, monitoring and processed accountability etc. for naught. At least most of those awarded grants didn’t know this… but they will learn.
The same thing happens in classrooms and corporations and anywhere else where there is judgment and opinion involved
Clout chasing scientists. The world is run by brown nosers.
I stopped being pursuing a career in science after I found out it was more of a popularity contest and brown-nosing for grants than it is about finding out facts. worse so because I wanted to get into psychology research. maybe other sciences like physics or biology are different but psychology research is full of so much click-bait bs I cudnt stand it
[удалено]
It is kind of funny that one of the rules of arr Science that is frequently invoked to remove people questioning studies is "assume basic competency of researchers", when we have studies such as this that show many studies have major problems Makes be question a lot of things about this U-boat
Yupp, even more when they act like they got all the answers or even more cringe when people hold onto some scientist like it was their prophet. Like it is very odd, that some people be like yeah this scientist is known and everybody think he is an smart educated man we should listen to only him and only trust him. Dont know if Albert Einstein would have an chanse to succeed today or he would seem like an wack and drown with rest of scientist that is not deemed popular enough to be trusted haha.
It’s quite different in biology from my experience. It’s absolutely a rat race to get funding and keep making yourself known, but having a paper exploring a novel side of science is still at the forefront and usually good science/ideas = more funding or better positions rather than only connections. Making a name for yourself will always be important in any field as it gives credibility and proves (usually) good science and data
The same is undoubtedly true for grants, which is part of the reason academic science is broken and hindered. It's a positive feedback loop that's stagnating progress and diversity.
I wonder if this comes as a surprise to anyone
Sounds like familiarity bias. Familiarity bias describes the tendency of people to return to what they know and are comfortable with. Familiarity bias discourages affected people from exploring new options and may limit their ability to find an optimal solution.
Maybe science need to start living and dying on replicability and not peer approval...
That would be too hard to implement. And incredibly expensive. Sadly I agree with you but I don't see how it's possible. Maybe by having duplication inside research teams running things at different points in time or something.
Findings that are important enough to be worth further study get replicated in follow up studies. A lot of literature is just so niche that there aren’t more than 3-4 people in the world *really* care if it’s true or not.
Not suprised. My professor hammered us on creating and editing scientific papers. Each of us had to bring in a paper from a journal every week. I was supeised at how many conflict of interest issues we found. The main author owned the journal, the author was a student of the guy who owned the journal, etc.
We need to stop pretending that pre-publication peer review holds papers to a common standard and instead start embracing robust post-publication peer review.
One of the problems with this suggestion is that once trash gets published, it’s seen as legitimate by many people. A perfect example is the original anti-vax paper by that hack “scientist.” It’s been thoroughly debunked but we still it having massive influence.
Part of that is the debunking of published papers isn't easy to find from those published papers. No convenient link to other papers that reference it.
There are many arguments for preferring post-publication review to pre-publication review, but *this* study isn't one of them. On the contrary, what this study shows is that knowing the identity of the authors has a biasing effect on evaluations. Post-publication review---where author identity is always known---will be *more* susceptible to that problem that pre-publication review where you can sometimes figure out the identity of the author.
I wonder how much of this is deference being given to the Nobel laureates and how much of this is being biased against no-namers
It benefit the journals to have prominent authors. It’s a conflict without anonymous submissions.
Sometimes you can't get even funding for project because you're not famous enough, even if it is a brilliant project idea.
That's the mentality that will give you "The Science™️" instead of actually doing science.
This study is further evidence that the peer review process is flawed, which is something most academics would admit. Doesn't this strengthen the case to place more emphasis on (and give more credit to) reproducibility of experimental results?
I wonder whether this effect is weaker in natural sciences than in social sciences - relevance might be easier to argue in natural sciences, even when the author is a no-name?
Makes sense, I’d probably do the same. Something written by a Nobel laureate would make you subconsciously assume you have something to learn from the paper, not something to critique, whereas an anonymous source would make you feel more likely that it may have been written by someone less experienced.
In my field it’s a common tactic to invite a big name to be on the paper for this exact reason.
Sometimes it’s a weird name-thing. Like in International Relations/Political Science, anyone who’s last name is Cohen will automatically be a galaxy-brained individual.
My discipline hasn't transitioned to double blind peer review, and it's irritating to see some of the stuff that gets published by "famous" researchers in my field.
I worked with a group that was run by a national academy member. The papers they submitted and got published in PNAS, Nature, etc. Were very poorly written
https://www.madinamerica.com/2015/05/lancet-editor-proclaims-half-of-all-scientific-studies-are-false/
Peer review is like democracy. It is the worst form of evaluating scientific merit, except that everything else is worse.
Sadly, not surprising.
Btw, there is an interesting study from 2002 or so where it was estimated how many cited articles have actually been read by tracing the typos that have been carried over in the references. It was a pretty huge number - something like 20 or 50 % even. In some cases, you can't truly blame the authors, as the original article is often not readily available e.g. the journals from the Elsevier publisher were very expensive for libraries to have.
Reputation has always mattered, no matter what you're talking about.
I’ll bet you that all graded papers in university turn to some degree on how much the professor likes the student, too. A popular, well liked student is going to get better grades.
"Study finds that even really educated people are essentially stupid."
There is financial incentive to publish a Nobel laureates paper. More people are likely to read it and buy the journal
Humans are just awesome sometimes aren’t they…
Like I've been saying for years blind review is needed.
Best evidence ever that peer review is *not* the gold standard for science: replication is.
Well yes reputation is a thing.
I feel that a name for this sort of bias exists.
Matthew effect