Re: In Defense of Anecdotal Evidence

["Rob Carlson" <robc@xmission.com>]

On Thu, 12 Dec 1996 14:12:23 -0800, Huge Cajones Remailer wrote:
>Statistics are a useful tool, but they have their problems.  Their
>accuracy is often in doubt.  Most scientific data comes with an error
>analysis so you can tell what the figure means.  For some reason
>statisticians never do this so we cannot tell whether their numbers
>are accurate to within 0.1%, 1.0%, 10%, or even worse.
>
>There are many other problems.  For instance, users of statistics
>assume they have a random sample, even in cases where that is far from
>clear

[ List of other problems deleted ]

Of course, anecdotal evidence also suffers from all of these problems. And
in greater magnitude. This is true since it is a special case of statistical
evidence. With a non-random sample set of one and no controls for
observer bias.


>The advantage of first hand experience is that it is primary evidence.
>You know it's true because you were there and saw it.

I have seen women cut in half and rabbits made to appear in an
empty hat with my own eyes. I'm quite sure neither of these is true.
Humans are poor observers. The data processing unit is easily
fooled. Many people make a living off of this fallability such as 
magicians and politicians. 

The observer is also tainted by previous experience(or lack of) and individual
needs. This leads to bias unavoidable in even the most honest and
reliable observers. Statistical methods are used to control for this. 
Double-blind techniques are one example. Humans also have a need
to make conclusions when insufficient evidence is available. Witness
the number of people who have an opinion on the innocence of OJ
Simpson based on <2 minutes/day condensation of the evidence.


>The advantage of anecdotal evidence (in the sense we have been using
>it) is that the person who is telling you the anecdote was there and
>saw it.  You can cross-examine them and get a full understanding of
>the evidence provided.

The reporting of evidence involves different issues. If you want to 
believe that women are actually cut in two or that politicians are telling 
the truth anytime their lips are moving, thats one thing. If you want to tell 
me its true because you personally observed it, thats quite another. Given
the failures of humans as observational tools, your story is unverifiable by
me. Perhaps through effective cross examination I can prove you wrong, but
I can never prove you right with such a technique.That will require other
evidence outside the control of the observer ( statistical is just one available ).

Evidence that can be verified independently by many observers increases the
reliability. Experiments and polls can be done by me thus eliminating your bias.
Independent verification can also check for errors and check the parameters
under which the evidence is true. Studies are done with certain assumptions
and controls. The evidence loses its reliability when removed from this context.

This doesn't make studies or statistical evidence true. Just more reliable than
anecdotal evidence. Humans who will lie about their observations will also produce
flawed studies. Again the former (anecdotal) is unverifiable, but I can check the 
latter (statistical) independently. I also don't believe polls can be used to determine 
the truth. They can only tell what a mass of people believe. And then only if done 
correctly.

To make this on topic, how does this apply to cryptography and crypto-politics? This 
issue is a foundation of our discussions here. Shall we accept anecdotal evidence
such as the "If you only knew what we knew" arguments? What is more reliable: IPG's
claims that their product is an OTP because they say it is or Bruce Schneier's book that
can be used to point out the fallacy's in their claims?

Relying on anecdotal evidence makes you susceptible to the magicians of the world. The honest
ones use mirrors and their need is to entertain you enough to get your money. The rest use
anecdotal evidence and emotional arguments (verbal misdirection?). Their needs are left as a
test of the reader's naivete.

Rob Carlson <robc@xmission.com>