Hi folks,

I'm a sometime amateur astronomer, generally interested in the sciences since I was a kid, etc. I've done some reading in "bad" astronomy and other sciences, and have made my own observations over the years (about thirty), and on this basis propose the following:

Fundamental change is occasionally needed in the sciences. By such change I mean more than patching theories to match new observations.

Fundamental change upsets ongoing lines of research, possibly negating them completely. For example, imagine the big bang never happened, certainly a recurring topic at the fringe of astronomy. If the big bang theory were to be upset, at least one if not two generations of astronomers in cosmology would be looking for jobs!

Peers who review papers for publication are naturally sensitive to this, and will abhorr fundamental change in their disciplines for this reason. Thus peer review will tend to squash research which may fundamentally change a particular area of science.

My questions to the list are two fold:

Has anyone here observed this effect in their own disciplines (or have you _effected_ this effect)?

What might be a more concilliatory review process to apply to new research, that removes what appears to me to be a conflict of interest?

I hope I've selected the right category for this thread, even though this is not about any particular theory.

Thank you very much!

Boris Starosta

Peer review can be a brutal process but one has to remember that peer review has led to radically new theories in the past and it seems doubtful that it will fail to do so in the future.

Greetings... I honestly think you're way off base here. The "peer review" system in my experience, is not replete with people seeking to protect their own interests. Science is out there to discover the facts... not protect "pet ideas".

Certainly, sometimes there are major changes in accepted theories, but the peer review process in no way shakles the acceptance of new ideas. Actually, it's quite the opposite.

The peer review process is designed to look for fundamental flaws in the experiment, data, or conclusions, NOT estimate the impact of the research. Anyone reviewing a paper is looking for the following:

1) Is the technical basis for the experiment sound?
2) Was the experiment carried out in an unbiased manner?
3) Were multiple trials of the experiment performed?
4) Were multiple independent methods of analysis used?
5) Are there irregularities in the data that can't be adequately explained?
6) Does the data and experiment performed support the conclusions drawn by the author?
(etc.)

I have real trouble with your post because of this line: "If the big bang theory were to be upset, at least one if not two generations of astronomers in cosmology would be looking for jobs!" Here you seem to be implying some "conspiracy" to "protect someone's job". I'm here to tell you, it just doesn't happen that way in the real world. All scientists must keep up with the times, or they become irrelevant. Just like lawyers, doctors, accountants, and other professionals must keep up their training, so too must scientists.

Hope this helps...

Welcome to the Board, Boris!

I recall there have been threads where this was discussed and IIRC, gethen's son had some problems in his scientific study. I don't know the threads offhand so you'll have to search for them. I'm pretty sure they were "Against the Mainstream" threads.

Also, search for a poster named Thomas. I recall some of his posts had discontent with the peer review system.

I for one feel that the peer review system is a good one that does not need large-scale change. I believe most of the significant scientific discoveries in the last 75+ years has been done under peer review. Yes, there can be the occasional problem with 'old school' researchers rigidly sticking to ideas and there is certainly some financial/corporate pressures in some areas (medical, for one) but does that make peer review inadequate?

As for methods to remove conflict of interest I am unsure how it could be done. You need to have reviewers whom are intimately well-versed in the field of the paper they are reviewing. These same reviewers would have, in your view, conflicts of interest with the need to squelch research that threatens their field. But without them, research of poor quality is more likely to be published and that would probably not help advance the field. It would more likely hinder it by having others waste time replicating poor research or following dead-ends.

Here is the thread started by Thomas in which peer review is discussed.

This is what peer review is supposed to be by design. Overall, I think the review process is necessary and a positive thing. Referee reports have led to improvements in papers I've written.

My concern - being that the papers I've written are against the mainstream - is that there is a fine line between questioning the conclusions and rejecting a paper simply because the referee disagrees with the conclusions. This is where the personality of the referee comes into play. Some referees will tolerate interpretations they do not agree with and others will not.

Where a problem arises is when the conclusions ARE consistent with the analysis of the paper but the referee doesn't like the conclusions and on that basis alone rejects the paper. If the analysis is sound then the paper should be accepted regardless of whether or not the referee agrees with the conclusions.

The research field will not be harmed by being exposed to conclusions that are ATM if those conclusions are consistent with the analysis that was conducted. The way it is supposed to work is that if a discrepant result arises, then another research group should do a follow-up study to find out whether or not that discrepant result holds.

If you correspond with people that are trying to publish ATM results you will find that they have stories in which the anonymous referee chose to reject a paper not because of the analysis, but because the referee disagreed with the conclusions drawn from that analysis. In those cases the author should be allowed to present the author's interpretations of the results to the research community (as long as the conclusions are logically drawn from the analysis).

I'd like that peer reviewers be policed by some kind of legal body, so that they feel greater responsibility for their expressed judgements. The system as it is now welcomes all kinds of injustice, corruption and clannish behaviour.

One area to examine is geology/geophysics. Over the past 40 years an ATM idea (continental drift) has become central (plate tectonics).

I agree with dgruss that the peer review process has the danger of injecting predjudice concerning conclusions that do not agree with the reviewer's views. I've thought about that before and haven't found a satisfactory solution. Part is that whether, proper or not, a reviewer is probably not only looking at the internal consistency and reasonableness of the paper itself but also comparing it to the preexisting knowledge base. For myself, that seems to be inescapable. Such review seems to me to be essential, but it can definitely me misused.

__________________
If they can get you asking the wrong questions, they don't have to worry about the answers.

Keep in mind that it is the editor of the journal that makes the final decision to whether to except a paper. The peer referees can only make recommendations.

I would disagree. I would not want a group of people trying to judge the merits of my papers who could not know the field well enough to make an accurate judgement. Do some scientists act "clannish?" Yes, they do, but it usually isn't hard for an editor to tell if a referee is being unreasonable. As a result, an editor may ignore their judgement and publish the paper. This is especially true if a particular referee has a history of trashing papers they don't like.

__________________
Old laser physicists never die, they just become incoherent.
These days, every Tom, Dick, and Harry thinks he knows what a photon is, but he is wrong. - Albert Einstein

"Galileo, it has come to the attention of this body that you have made some statements..."

Nah. Imperfect as the present system is, imagine it with LAWYRERS:

"Define 'titration'..."

__________________
If they can get you asking the wrong questions, they don't have to worry about the answers.

Whoever argued with that wasn't me. I'm all for accurate and decent judgement. But the indecent judgement must be punished substantially. Exposing a few examples of indecent behaviour of reviewers each year could greatly improve the morals among even the nastiest of them.

Thanks Triangle man for the information on Thomas. I'll go study that.

Also thanks to dgruss23 for acknowledging my observations.

This is the crux of the matter: to get published in the journals you have to run this gauntlet of your peers (and competitors, essentially, if your ideas are anti mainstream).

Presumably, the internet has lowered barriers to publishing most anything. Have persons on this board observed any effects the internet might have on this matter? (have you been around long enough? I have not, barely.)

Has the internet had a visible effect allowing the dissemination of "bad science"? Do you have more and more bad science fads to deal with, as a result of internet publication? Certainly there's lots of fringe science visible on the internet, that ordinarily I would never have been exposed to... would'nt have found this forum either...

Boris

That is unlikely to improve things. The moment legal gets into the mix you will have another type of pressure put to bear - fear of lawsuits. This will result in many unworthy papers being published purely because the reviewer is afraid of legal action if he gives it a bad review.

There are real world examples of this. In many countries, organizations involved in Financial Regulation are protected by legislation that prohibits them being sued for declining various applications. This type of protection is becoming the norm now as these organizations found that lawsuit pressures were interfering with their ability to make objective decisions.
So in regulatory circles they are moving away from legal 'policing' because it is inefficient. No reason to think it would be any different with peer reviewing.

Nukular, you wrote:

...one has to remember that peer review has led to radically new theories in the past and it seems doubtful that it will fail to do so in the future.

Well, you are apparently exactly disagreeing with me, and are attempting to set me at ease. You might succeed if you tell me what radically new theories peer review has facilitated? Or name just one by example?

I don't know of any, but then I'm no historian either.

Boris

You're welcome.
First a disclaimer, I have not dealt much with Bad Science pre-internet so I do not have experience in that regard. The very nature of the internet - that it allows you to open a webpage with any content you want with no editor or peer review - would definately increase dissemination of Bad Science. Now one doesn't have to go through peer reviews or book publishers, just book a webname and away you go! Check out www.crank.net for examples of what is now available for dissemination thanks to the internet. Heck, at least two forums here at the BABB (Lunar Conspiracies and Planet X) are dedicated to debunking Bad Astronomy that is mainly propagated via the internet. As the internet serves as a prime example of what would happen to journals if they weren't peer reviewed I am grateful for the peer review process so that when I look at an article in a journal I can get comfortable with the fact that others in the field have reviewed the work.

Disclaimer 2: This stance is in no way supportive of any kind of editing/censorship of internet content. I like the vast freedom of information available on the web, even if a lot of it is kooky, but I am also wary of relying on information found on the internet as well.

Some BABBers, I am sure, must have had the experience of receiving very different feedback from different journals, or from different reviewers for the same journal.

I have seen this from the journal editor’s side (my Head of Unit being editor for a while of a respected ergonomics journal and me doing the donkeywork). I can only describe the frequency with which variant views were expressed by the chosen reviewers as giving much cause for concern. It was greater than I had anticipated.

I don’t know whether the same applies in physics and astronomy. Perhaps I may learn from future posts to this thread.

One thought. I recall a number of occasions on which very necessary amendments were requested to otherwise sound papers. One tried to explain what needed to be done – but the message did not quite get through (e.g. there was a blindness to some aspect), making it impossible in the end to publish the paper. On such occasions, one felt the need to break down the formalities of the peer review process so as to have someone sit down with the author and hammer some sense into him!

The exact rules no doubt vary from journal to journal. Given that the editor (or his accomplice) chooses the referees, there is considerable pressure to follow a majority opinion. Reviews remain on file as a sort of "audit trail" to the operation of a journal, and cannot lightly be disregarded.

__________________
"True skepticism encompasses not dismissing evidence because it seems to defy rational explanation."

Sorry, I misunderstood what you had said. I would still disagree though. I really doubt such a system would improve judgements. The current system's greatest virtue is that a referee can say what they feel is right without fear of reprisals. Such a system that you propose will inveraribly (although unintentionally) crucify a few who did nothing wrong. If you create an enviroment where scientists could be reprimanded for a review, they will be afraid to trash a paper that truly deserves it.

__________________
Old laser physicists never die, they just become incoherent.
These days, every Tom, Dick, and Harry thinks he knows what a photon is, but he is wrong. - Albert Einstein

To answer my own question, this is what Thomas wrote April 20:

Very interesting. But now the question is, responses from whom?

Boris

The irresponsibility of reviewers' judgements is a real problem - would you deny that?
Now how to counter that is an open question. Inviting the regular scavenger lawyers could really make the problem only worse. Still there must some way for the author to defend oneself, something like a right to formally demand re-evaluation by other reviewers, or a right to demand that peer reviews be peer-reviewed themselves

In Seeing Red Arp described an instance where the paper had to be submitted to three referees before the paper was accepted with the third referee commenting that one of the other referees seemed to angry to objectively evaluate the paper. So I do think authors can ask. In those instances the question would be whether or not the scientific editor is fair enough to grant the request.

That's an important point! The referee makes the recommendation and the editor makes the call, but the editor can choose a close friend or someone that the editor knows will shred a controversial paper. I don't know how often that happens, but the fact that there is an editor above the referee making the final call does not guarantee that that process is always fair and objective.

No process involving judgement can ever be truly fair and objective, the question comes down to whether the peer review process is somehow fundamentally flawed and needs to be changed. Any process will make mistakes or be influenced by outside forces. I personally can't see any kind of significant change that could be done to make the peer review system better.

And to answer AgoraBasta's statement, "The irresponsibility of reviewers' judgements is a real problem". I'm still not convinced that there is a significant problem with the peer review system. This is a tough thing to investigate though, ancedotal evidence appears to be the only type of evidence to go on in this situation and that type of evidence is weak. I'm not willing to conclude that the system is flawed because of a few individual examples where people didn't get published, especially when we don't have both sides of the story.

If people feel that personal influences are systemically sabatoging the peer review process, how did you come to that conclusion? (For the record, I've never published a scientific paper so I'm asking the question to try to get further insight into the matter).

I think you all have to realize that there is a distinct difference between science (a process), and scientists (a population of people). So, "science", seen not as a process, but as the work product of scientists, will inevitably suffer from whatever weaknesses the scientists do, no matter what kind of review process you care to invent.

By and large, I would say that the peer review process in place now, is about as good as you can expect. The reviewers are invariably published authors in the field they are reviewing, and so are themselves subject to the peer review of others, when they submit papers. While the potential for conflict of interest is clearly there, I don't think that the reality of such conflict is at all pervasive.

The effect of reviewer prejudice can be large, but is always temporary, as far as I can tell. The early history of radio astronomy is a good example. When Grote Reber submitted his first paper to the Astrophysical Journal (Notes: Cosmic Static, ApJ 91: 621-624, June 1940), no reviewer would accept it. So far as I know, the only real reason was that they could not see any significance to radio from the sky, and probably felt that Reber (an electrical engineer, not an astronomer) had simply made some mistake. But the editor (Otto Struve) published the paper anyway, on the grounds that he would rather publish a questionable paper that turned out to be wrong, than not publish a questionable paper that turned out to be right. But once Oort & van der Hulst, in Nazi occupied Holland, figured out that Reber had detected the neutral hydrogen spin-flip at 21 cm, everybody changed their attitude in a heart beat.

The point here is that even if Reber's paper had been refused, it was really just a matter of time, before its significance would have been appreciated (especially in the light of heavy radio research in WWII). So even if flawed review had quashed this new idea, that radio could emanate from space, it would not have been quashed for long.

And so I think it is in general. At worst, a failure in the review process that blocks a new idea will only block it for a limited time. At worst, the new idea will be blocked until we meet Planck's criterion: "old theories die when old scientists die".

And one other observation. It's not at all about "jobs". The earlier comment that a lot of astronomers would be looking for new jobs if big bang cosmology were overturned is a fanciful notion. I can't think of anyone who's job literally depends on any theory. It's not about reviewers and their jobs, but reviewers and their intellectual turf. Nobody likes to have some young whipper-snapper come along and prove that the last 40 years of their life was spent on a wild goose chase. Pride, as they say, goeth before the fall. Most scientists can work around such an experience, but some can't.

Open peer review is one of the best things that science has going for it. Governments, corporations and the financial system could learn a thing or two from the way scientists work.

__________________
Fraser Cain
Publisher
Universe Today - Free space news delivered by email every weekday.

What are 'open' peer reviews?

The system is almost as good as you can expect. My concern is that papers be accepted/rejected based upon the accuracy of the analysis and not upon the conclusions of the researcher. I had a paper go through four reviews at a journal before being rejected. The ridiculous part was that the first reviewer (twice reviewed it) said to accept the paper but recommended some changes. After twice making suggested changes/re-analysis I kept getting the same result. But the third draft went to a new referee who after sitting on it for 5 weeks opted out of "formally" reviewing the paper saying that the paper was too controversial.

Now common sense says it doesn't take 5 weeks for the referee to assess whether or not he/she can review a paper. I have no doubt that the Scientific editor found a referee qualified to review the paper's content. Common sense also says that after if there were glaring errors in the analysis it should have (and would have) been rejected on those grounds. So by the time I submitted the 4th version in which I had removed all discussion of the controversial aspects of the analysis and simply left the base results the 3rd referee did not feel the paper offered anything new - and further commented that he did not like my previously published paper which he felt "added confusion" to the literature.

So I sat back and reflected upon what I learned from the experience and wrote a new paper which I submitted to a different journal. And I did learn quite a bit about different ways to present the results and how people would respond. The new paper is still under review (been almost 5 months now), and it is a better presentation than the others. I guess my point is that even though I feel the paper should have been accepted at the first journal - a better product has emerged for submission at a different journal as a result of the review process. When you're trying to publish ATM results you hope for fair treatment, but it is unrealistic to expect every paper to get accepted.

Boris asked for revolutionary theories that have been peer reviewed.

I would list quantum mechanics and relativity as revolutionary theories that were peer reviewed. Most of the early papers appeared in Annalen der Physik and Zeitschrift fur Physik, if memory serves me correctly. Along with the papers of Planck, Bohr, Einstein, Heisenberg, Jordan, Born, and Schrodinger, many other revolutionary things occured in the 30's in physics. For one, their was the discovery and confirmation of antiparticles. The discovery of the neutron lead to fundamentally new ways of thinking about nuclear structure. These discoveries lead to even more remarkable phenomena: the eight-fold way and the particle zoo, neutral kaon mixing, the J/psi particle (which seemed to take everyone by surprise), the chiral nature of the weak interaction, as well as the asymptotic behavior of strong interaction.

Much of this list relates to particle/nuclear physics, which is a result of personal bias, but I fail to see how the peer review system ceased to function in all of these cases. A long look in the dusty pages of Physical Review, Nuclear Physics B, or any number of journals will reveal a multitude of papers with predicted theories not born out by experiment or ones consisting of attacks on radical theories in defense of the status quo, but in the end it is experiment which decides with theories become dominant and which don't. While this is not in itself peer review, it plays an enormous role in relating what is known (as well as what has been previous hypothesized) to what should be acceptable/expected of any future publication.

Of course a radical new theory would be nice and probably generate thousand of peer reviewed papers. But as an example, no other theory challenging quantum has survived the test of predicting (using the mighty sword of calculation) the magnetic momentum of the muon -- to current experimental precision, atomic spectra (with the Lamb shift), resonances in the particle zoo, the Quantum Hall effect, or the thousands of other successes that quantum theory has racked up. Of course this is not to say quantum theory or any scientific theory is complete. Science by its nature is an inductive process and is performed by humans (who can often be blinded by dogma and emotions) and is therefore open to error. The secret to the scientific process' success lies in the the fact that it is self correcting and peer review plays an important role in that process.

On a more astronomical note, recent radical theories I would invoke would be the idea that gamma ray bursts are galatic in origin and the recent results from supernovae which seem to indicate that the universe is not only expanding but accelerating. The numbers and predictions of these phenomena are really staggering as well as exciting. As a side note I will offer a current controversial claim, namely the suggestion that alpha (the fine structure "constant") varies in cosmological time. This was suggested by Webb et al, but many authors are pointing out the strange assumptions made in their analysis. Of course only time will resolve this question but it is a nice example of the peer review process in action.

I am sure that one could easily find numerous other examples outside of physics and astronomy, like the discovery of DNA and RNA, plate techtonics (previously mentioned), the KT boundary, etc. In the end, I would suggest caution at the claim that the minor problems of peer review, which usually involve the failing attributes of human beings in general, are true failures of the peer review process. It seems to me to be misplaced and misleading. Peer review is really just a filter and it serves its function well.

On the other hand, the your reply may be dealing with semantics. I admit my original wording is pretty strong, but the idea is that revolutionary theories pass through the peer review process and thereby facilitate other publications. I for one am skeptical of papers/results which don't pass through the peer review process, since experience has show that these papers for one reason or another have been found lacking. I would venture to guess that many other scientists feel the same way. Time is a precious thing and filtering out the wheat from the chafe only works to facilitate further work in science. Continually examining poorly designed experiments or poorly constructed theories, doesn't really lead to more insight but it doesn't help to point out mistakes that should be avoided in the future. It is in this way that I would assert that peer review has led to the publication of many past revolutionary theories and will probably continue to do so in the future.

Punished? Remember Pons and Fleischman? How are their careers going these days...?

Punishment happens on its own when crackpots become outcasts.

Interesting discussion! I have just a few observations to add, for whatever they're worth. I think Boris's starting point is essentially right, that the peer review process is blocking scientific advances, at least in some fields. I would rank the main fields this way: astronomy worst (for blocking new ideas), then physics, then geology, then biology. I can't think of instances where major thrusts in chemistry or math were blocked by the peer review process, but maybe others can.

The problem with peer review is not so much the reviewers themselves. They are just doing their bit to try to make the publication system work. Rather, it is the editors of journals, who unfortunately are far too conservative most of the time. A conservative approach is good in certain fields, but where we are going to the roots of science a broader, more open-minded view is needed and that can be rare among editors, especially of the bigger journals.

At the same time I have to agree with the statements that the peer review process works well most of the time. It keeps a lot of bad stuff out, and the reviewers can often make valuable suggestions to save papers. This at least has been my own experience. When I first starting submitting papers to journals (on evolutionary biology mostly), I tended to write overly long, speculative papers that the reviewers couldn't stomach. Gradually I was able to refine my style to the point where I did publish papers. It was a hard learning process, but one everyone has to go through. Going against the mainstream does make things a lot harder though.

The key role of the editor is in selecting reviewers. If the editor sees a paper that is against the mainstream, but seems to have possible merit, he may send the paper to more open-minded reviewers. If the editor alternatively has suspicions from the outset, he will instead go to his conservative reviewers.

What science could use is some different formats of journals where new ideas have a better chance, but I don't know what those formats could be. Once upon a time there were journals where the editor alone decided what to print. This could still work in certain cases, but somehow you would need to get the smartest, most innovative people to be the editors, not an easy task!

Another approach would be to have some agency fund whole journals which cater to the main alternative veins in science. In astronomy there should be a journal like "Static Cosmologies"; in geology, "Expanding Earth"; in physics, several perhaps like "New Concepts in Relativity" or "Gravitational Mechanisms"; in biology, "Mechanisms of Evolution". And then hope that these too would not be taken over by conservatives! As to who would want to fund innovative science journals, I have no idea. Probably we can exclude governments and big business.

P&F are a prime example of why peer review is necessary, even for potentially ATM ideas. A good review of their experiment would have exposed all of the shoddy work they did and the whole thing might have died on the vine. Instead, they published via press conference and witheld all details of their experiment. Some of this was due to pure greed. If it was real, CF would have been very lucrative and they were possibly attempting to preserve their patent rights. Still, it doesn't excuse their behavior. There are mechanisms that could have allowed for review and still preserved intellectual property.

Nukular's post brings out several examples of what could be called "mainstream" against the mainstream ideas. The possible drift in the fine structure constant is a good example of this. The development of the quark model is another good example. Initially, it was little more than a way of organizing the growing particle "zoo" in a systematic way. If you asked a physicist in the 60's if quarks were real he probably would have responded "It's a nice idea, but quarks aren't real." Only after the discovery of the J/psi in 74 (referred to at the time as the November Revolution) did quarks become a significant part of the current standard model. All of this work was initially published in peer reviewed journals long before it became part of the mainstream.

Finally, most theorists would like nothing more than a discovery that would require a wholesale reworking of the current models. Far from putting them out of work, as the initial post in this thread suggests might be a consequence, it would be job insurance for years to come.

__________________
"I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind." - William Thompson, 1st Baron Lord Kelvin

"If it was so, it might be, and if it were so, it would be, but as it isn't, it ain't. That's logic!" - Tweedledee

This isn't right. This isn't even wrong. - Wolfgang Pauli

There are already refereed journals for alternative ideas in science out there. Two that I'm aware of are the Journal of Scientific Exploration and Apeiron .

Such journals are good for getting a controversial idea in a referenceable form - its published somewhere. The problem is that if your goal is to communicate your results to your research community then you've accomplished almost nothing by publishing in these journals. Your results can be safely ignored when published in such obscure places.

Those are two good journals, although Apeiron at least is now in electronic form only. I am of the view that electronic articles and preprints will do little to change mainstream thinking. You need print for that. I would urge prospective authors with controversial opinions to aim for print journals and not give up. You want to have a permanent record of your work in libraries. When I was suggesting new journals for alternative science, I was thinking of larger productions that would have some of the clout of the mainstream titles.
Someone has to cough up some money for that, as the income from subscribers would not be sufficient.

Another good physics title to add to your list is Physics Essays, though it has become irregular of late.