Pseudodebate

Continuing on the theme of "pseudoskepticism" over the past few weeks (see here and here), I've come upon some of Marcello Truzzi's original comments on the subject and have decided to give some specific responses. You can read the article, originally published in his Skeptical Inquirer, here.

After a brief introduction, Truzzi starts out with discussing the burden of proof of evidence. Most of his article follows on this theme, so I'll just quote this to give the summary. (All italics is his original emphasis.)

In science, the burden of proof falls upon the claimant; and the more extraordinary a claim, the heavier is the burden of proof demanded. The true skeptic takes an agnostic position, one that says the claim is not proved rather than disproved. He asserts that the claimant has not borne the burden of proof and that science must continue to build its cognitive map of reality without incorporating the extraordinary claim as a new "fact." Since the true skeptic does not assert a claim, he has no burden to prove anything. He just goes on using the established theories of "conventional science" as usual. But if a critic asserts that there is evidence for disproof, that he has a negative hypothesis --saying, for instance, that a seeming psi result was actually due to an artifact--he is making a claim and therefore also has to bear a burden of proof.

So here's the situation: Some experiment somewhere did a test for the presence of psi. They came back with results and claimed that these constituted evidence for psi. What is a skeptic to do? Truzzi's strawman of a skeptic would immediately say that the result must have been an artifact or due to some explanation other than psi, and then give no justification for this. But just because no justification is given doesn't mean no justification exists.

Let me illustrate this, once again with the example of "slood," a postulated substance with ill-defined properties. In our world, we can explain almost everything without appealing to slood. Any events that people have attempted to explain with slood are just as easily - if not more easily - explained by known phenomena. There are many models of the underworkings of the universe, and not one of them predicts the existence of slood. In fact, fitting it in to any of these models would require a drastic reworking. Simply put, there's absolutely no reason to believe that slood exists.

Now, let's say one group of never-say-die scientists designs an experiment to test for the presence of slood. If slood exists, this experiment would be expected to give Result A (which is generally a statistically significant deviation from what we would expect in a sloodless universe). The experiment is performed, and Result A is found. The scientists then claim this is evidence of slood.

So what's a skeptic to do here? Accepting slood would require a drastic reworking of known science, and up to now there's been no reason to believe it exists. On the other hand, there are many other possibilities to explain why the experiment resulted in the way it did:

1. A statistical fluke - Generally, statistical significance requires 95% certainty. This means that there's a 5% chance this result was obtained purely as a matter of chance. Of course, some results can be more extreme, and the chance of them happening randomly is less (though never zero).
2. A known mechanism - The experiment might have failed to control for (or properly control for) the possibility that some known physical phenomenon influenced the result. In this case, the result could just mean that this phenomenon was coming into play.
3. An unknown mechanism - Maybe it wasn't slood that caused the result of the experiment. Maybe it was dools, a substance which no one has even thought of yet (but which doesn't require a drastic reworking of science to fit in). Or maybe our universe is all a giant simulation and the designer decided to toy with us by changing the results.
4. Fraud - Many "scientists" make a living promoting the existence of slood, and finding evidence that it exists would be in their benefit. Some such scientists might have been involved in this experiment and had a chance to cheat. Also, it's known from past experiments to test for slood that fraud has indeed occurred.

Adding these four possibilities in, we've got a total of five ways this result could have occurred, only one of which implies the existence of slood. Now, the number of ways it could have occurred alone doesn't give us the chances that it's any one way. To get this, we have to look at the likelihood. From the beginning, we know that accepting slood would require a drastic reworking of science and we've seen very little prior reason to believe it exists. This makes this possibility very unlikely, so it's much more likely that instead one of the other possibilities came into play.

So even despite this experiment's results, it's still most likely that slood doesn't exist. Many amateur skeptics will stop there and simply say the evidence isn't convincing enough. However, some skeptics will then go a step further and analyze the experimental protocols to judge the plausibility of alternatives 2 and 4. Some scientists might even see if they can come up with a theory for a new phenomenon to meet alternative 3.

Generally, the result of these will be the skeptics coming up with ways the experimental controls can be tightened and the scientists with ways to change it to rule out alternative theories. If the original experimenters are serious, they'll then take these changes to heart and perform another, better-designed experiment with a few checks against fraud. If they aren't serious, they'll whine about "pseudoskeptics" and make no progress towards a better experiment.

Sometimes, in the real world, investigators do indeed take recommendations to heart. For instance, this happened in the case of the Ganzfeld experiments. What's happened in every one of these cases I've come across is that the results take a sudden plunge back towards the expected results. Sometimes the significance fades away completely, while other times it remains. In either case, this gives evidence that the skeptics were right in that there were some problems with the experiment.

The next step of inquiry will generally have skeptics and scientists looking over the experimental controls again to see if there are any remaining flaws. If some are found, the process is repeated. If none are, slood proponents will then dance on the rooftops claiming slood is proved to exist, but one important step is still left. This step is replication. What is done here is that a completely independent group should recreate the experiment from scratch, and see if they get the same results. Preferably, no one involved with the first experiment should be involved in the replications, and if possible, those involved in the replications should be neutral on the matter or even negatively biased.

The point of replication is to provide another level of checking against design flaws and fraud. The replicators can start off with all the recommendations over how to keep it tight, and shouldn't have any bias which would lead them to consciously or subconsciously influence the results. It also provides another check against statistical flukes, as it's less likely the same fluke would occur twice. Only once the experiment has been independently replicated do we truly have a reason to believe that slood exists - though we can never rule out alternative explanations. Further experiments should likely be performed to test other proposed properties of slood and to see if alternatives can be ruled out. After a long time has gone on with many tests of slood, none of which it fails, it will eventually become accepted science.

A single experimental result is never enough to change scientific consensus, and considering it most likely that an unusual result is due to an artifact, poor experimental design, or fraud is simply the default position. It takes work to overturn the current scientific consensus, and paranormal enthusiasts rarely want to put in this work. When they actually do, it tends not to work out for them. Skeptics are there to make sure they follow proper procedure and don't try to short-circuit the system, not to bar the way to new findings. If some claim has truth behind it, it can make its way in through the standard system. Accusations of skeptics being pseudoskeptics really just amount to giving up and fighting the wrong the battle.