Benford revisited:

[jpinson@fcdarwin.org.ec]

In regards to my posting about the New Scientist article on
"Benford's" law (unexpected number ratios in observed data), I 
received many comments along the line of:

> I suggest you look at any slide-rule or log graph paper for the
> simple explanation of this trivial phenomenon.


Actually, the New Scientist article gave a very clear explanation 
of Benford's law, and how it can be a simple artifact of the way
numbers are counted.

The article caught my attention,  when it went on to explain how 
other phenomena, such as alpha particle half-lifes can follow 
Benford's law for completely different reasons.

I won't attempt to retype the entire article, but Buck, Merchant, 
and Perez, reported in the  European Journal of Physics (Vol 14, 
p59) that both expected and observed half-lives followed 
Benford's law.

Quoting New Scientist: "..Why should nature operate in this 
curious manner? Buck and his colleagues point out that in the 
case of alpha decay, there may be a good physical reason.  In 
1928 George Gamow used the newfangled theory of quantum mechanics 
to describe alpha decay.  The alpha particle lives inside a 
"potential well" in which the energy is lower than in an any 
other state that it can normally reach.  It escapes by 
"tunneling" through the potential well.  The probability of 
tunneling occurring within a given time is found by forming a so- 
called "tunneling integral" and raising it to a power that is 
proportional to time.  So tunneling times, and hence half-lives 
naturally correspond to a geometric series, not a arithmetic one. 
If nature chooses the tunneling integral "randomly" with uniform 
probability, then the power-law dependence of alpha-particle 
half-lives on this integral leads to scale-invariant behavior, 
and so to Benford's law.  "

I am not a physicist or a mathematician, so I can't speculate on 
the validity of these statements.

However, all this did make me wonder if advances in fractal 
geometry, or chaos theory, might one day reveal patterns in 
seemingly random numbers derived from real word phenomena.

Since most random numbers used in cryptography are derived from 
real word phenomena,  we should be aware of such potential 
influences.

Actually, we don't have to bring in fractals, chaos, or Benford's 
law to have reason for concern about using real world based 
random numbers.

A method commonly recommended on this list for generating random 
numbers is to take the audio input from a workstation (with no 
mike attached) and then compress the output:

cat /dev/audio | compress - >random-bits-file

First of all, without extensive study of the audio output (from 
the *specific* source) who can say the output is random in the 
first place?  Harmonics from the various electrical circuits in 
the computer could easily produce non-random effects (like a 60 
cycle hum).

The second aspect that concerns me is the use of compression. Unless
you are very familiar with the type of data compression being used,
how can you be sure there aren't various signatures, tables, data
structures, or headers being inserted into the output.

Indeed, I had to go no further that the beginning of a data file
produced by the "Compress" program, to find nonrandom numbers.  All
such files begin with the same three bytes!  If something so obvious
is being overlooked, what about more subtle patterns?

Anyway, it seems to me there can be many factors "contaminating" 
random numbers based on physical phenomena, factors ranging from 
a simple 60 cycle hum to unrecognized fractal patterns.

It is perhaps a good idea to routinely "massage" random numbers 
to remove both recognized and unrecognized contamination.

Xor'ing random numbers derived from two or more different sources 
might be sufficient.  If done properly, such a procedure may 
produce non-Benford, evenly-distributed random numbers, which 
might be considered superior to the original.

Actually, I tend to be rather cautious about such things and am 
always on the lookout for possible "contamination".  I also try 
to keep an open mind to new ideas.

Jim Pinson