Packing the perfect baloney detection kit

We all have a collection, great or small, of critical thinking tools to detect baloney. And yes, "everyone" includes the New Agers and Creationists too, even if their Altoid tin-sized thinking kits are half-missing, rusty, and only dragged out to debate used car salesmen or faraway American Idol judges. 

Those of us who treasure our kits are always questioning what we should pack inside. Fortunately, just as there's no shortage of recommendations on how to stuff the perfect outdoor survival kit or travel gadget kit, we're lucky to enjoy a wealth of smart thinkers willing to share the contents of their baloney detection toolboxes. 

Here's an overview of a few great ones:

Michael Shermer's Baloney Detection Kit

None other than the famed author and publisher of Skeptic Magazine Michael Shermer lets us gaze inside his baloney detection kit. Its tools are of the question variety: 

1. How reliable is the source of the claim?
2. Does the source make similar claims?
3. Have the claims been verified by somebody else?
4. Does this fit with the way the world works?
5. Has anyone tried to disprove the claim?
6. Where does the preponderance of evidence point?
7. Is the claimant playing by the rules of science?
8. Is the claimant providing positive evidence?
9. Does the new theory account for as many phenomena as the old theory?
10. Are personal beliefs driving the claim?

There's video to go with it:

Carl Sagan's Baloney Detection Kit

Uh-oh. If Carl Sagan's name is on a baloney-hunting Geiger counter, I would not want to be Oscar Meyer. 

The inimitable astronomer's baloney detection kit, as summarized from Sagan's works by a Planetary Society volunteer, is a longish one. His tools are safeguards against creating baloney – which thus become tools to detect baloney checking whether a claim follows them:

• Wherever possible there must be independent confirmation of the facts
• Encourage substantive debate on the evidence by knowledgeable proponents of all points of view.
• Arguments from authority carry little weight (in science there are no "authorities").
• Spin more than one hypothesis - don't simply run with the first idea that caught your fancy.
• Try not to get overly attached to a hypothesis just because it's yours.
• Quantify, wherever possible.
• If there is a chain of argument every link in the chain must work.
• "Occam's razor" - if there are two hypothesis that explain the data equally well choose the simpler.
• Ask whether the hypothesis can, at least in principle, be falsified (shown to be false by some unambiguous test). In other words, it is testable? Can others duplicate the experiment and get the same result?
• Conduct control experiments - especially "double blind" experiments where the person taking measurements is not aware of the test and control subjects.
• Check for confounding factors - separate the variables.

The compiler follows the above with a summary of fallacies of logic and rhetoric, plus links to further resources. It's good stuff; please take a look. 

Brian Dunning's red flags of pseudoscience

An earlier post ("Here Be Dragons" and the red flags of pseudoscience) listed ten "baloney ahead!" warnings from Skeptoid podcaster Brian Dunning. Below is the list alone; see the linked post for more detail.

1. Appeal to Authority
2. Ancient Wisdom
3. Confirmation Bias
4. Confusion of Correlation with Causation
5. Red Herrings
6. Proof by Verbosity
7. Mystical Energy
8. Suppressed by Authorities
9. All Natural
10. Ideological Support

Building the kit

All right, that's a great bit of material to start from. I have a few other favorite bits of received wisdom that need to be fit into the kit (sorry, sources unknown for these):

• The plural of "anecdote" is not "data"
• Multiple pieces of bad data do not combine to make good data

Let's see, then I have a great list of "Six Problems with thinking that lead to acceptance of bad ideas" by Isenberg School of Management at the University of Massachusetts professor Thomas Kida, via the July 15 2006 Point of Inquiry podcast. My transcription:

• We prefer stories to statistics
• We seek to confirm, not question, our ideas
• We rarely appreciate the role of chance and coincidence in shaping events
• We sometimes misperceive he world around us
• We tend to oversimplify our thinking
• Our memories are often inaccurate

Those aren't baloney-detection tools per se, but are great dies from which to shape some tools. 

Okay. Right now I'm just emptying the drawers full of stuff, to see what I've got squirreled away. Lots of fine raw material, though in a variety of formats: skeptical questions, admonitions, red flags, and more.

Now it's time to sift, sort, and put together my own custom kit. Can much of the above be combined and condensed? What form should the tools in the kit take? Stay tuned – and please let me know the favorite bits you've got in your kits. What vital components do I need to consider? Whose exemplary toolkits am I overlooking?

Let's take the best of what's out there and put together one streamlined, baloney-detectin', baloney-stompin' kit!