Re: The Importance of Filtering

[Ray Cromwell <rcromw1@gl.umbc.edu>]

> * Extropians-style filtering, now being developed as a commercial
> product by Harry Shapiro and Ray Cromwell. (To be clear about things,
> Hugh Daniel, Eric Hughes, etc., expressed their willingness to install
> the Extropians-style software shortly after it became available,
> almost 2 years ago. Various delays ensued, then the offer by the
> authors was put in limbo, then the commercialization phase ensued.)

   Just to clear this up, this freeware code was never put into
limbo. I believe Harry talked to Hugh several times in e-mail
but Hugh didn't have the time to put up with the installation
(hint: it's harder than majordomo to install since they are
tons of configuration options.  You know, like the list statistics,
what filtering agents to use, digest format, incoming filters,
accounting database, etc)

   If anyone is willing to provide a machine, I will gladly
give and/or install my mailing list software, and then subscribe it
to cypherpunks. (during the development phase, this is exactly
how me and Harry tested the list. We subscribed to several mailing
lists and had the list software do the filtering)

   This is more optimal anyway. If the vast majority of people don't
want filtering, there is no need to waste cpu cycles by running
it as the main cypherpunks list. 

> And then of course there what the new list software. This allowed
> folks to exclude authors, threads, etc., at the _point of
> distribution_.

> Is this a good idea.? Well, if one excluded 10% of the traffic, then
> it would "save" having to receive 5-10 messages a day. Big deal.

> I used the ExI software, and found it an interesting experiment, but I
> can't say it save me any real effort. The effort of sending the
> filtering message to the list site, hassling with the formats, etc.,
> clearly outweighed the tiny effort it would have taken to manually
> press "d" to delete the messages when then appeared.
 
    Yes, and this is a recognized problem, and one of the things
we are going to eliminate in the commercial version. Basically,
the "send a msg to a mail server" form of command processing is
too painful to use except infrequently (such as ::resend) The
technique of using "in-band" commands within a post
was my first attempt to reduce the pain of sending commands
(by allowing you to postpone any commands, and then include
then in one of your posts later)

One of the ways to alleviate "transaction cost" of list commands is to
use client side scripts.  For instance, for list administration, I
wrote Harry a "hot key" perl script which takes over elm's Print
function. When Harry needs to do something, he hits 'p', and then
chooses an option. (for instance, to add a user, he types
'pa'. I could easily create a similar thing for anyone using
elm who wanted to exclude based on a single key-press)

> I also found it useful to at least spend the 2-5 seconds to see what
> was being talked about before pressing "D."

   The real use of the Extropians software comes in "exclude all" mode,
resend thread, and digest. Most of the people I see using the
list filtering (from the logs), do something of the following:
1) exclude all
2) read the 'filterlist' every 12 hours. If something looks interesting,
do ::resend thread, and get a digest containing that thread only.
3) include specific threads and authors.

   The only thing I ever use the filtering system for is to exclude
individual users, or annoying threads. (d for everything else, like
you)

> Your mileage may vary, but I doubt that the Extropians-style software
> is going to help much...I used it, and my experiences are what I just
> described. 

   I think it is superior to majordomo and listserv for other reasons
(remember, the filtering stuff is merely a "plug in" agent which could
be replaced with any filtering or reputation system) For instance,
the reputation system writers could use it as a platform to write
a reputation based filtering system if they wanted to.

> Filtering is the wave of the future.

  Definately. I fear there will be no good solution though until
we get atleast some partial natural language understanding.

-Ray