[ANNOUNCE] Eternity server 1.01 alpha release

[Adam Back <aba@dcs.ex.ac.uk>]

Annoucing the `eternity server'.

Alpha testers wanted.  See source code at:

	http://www.dcs.ex.ac.uk/~aba/eternity/

System requirements are:

  unix
  perl5
  news source (local news spool or access to nntp server)
  pgp (2.6.x)
  web space
  ability to install cgi-binaries
  ability to schedule cron jobs a plus but not essential

Unfortunately I have not got a service up and running for you to play
with because I don't meet those requirements (in cgi capability on the
server I would otherwise like to use).

If someone wants to set one up to demo, and post the URL that would be
useful.

The local news spool stuff is tested better than the nntp server
stuff, but it is all there and the nntp stuff was working at one
point.  Remember this is alpha code.  If you can hack perl and want to
stress test the nntp server stuff or the rest of it that would be
helpful.

What is an Eternity Service?

(Some of you may recognize the name `Eternity Service' from Ross
Anderson's paper of that name.  What I have implemented is a related
type of service which shares Anderson's design goals, but has a
simpler design.  Rather than invent a new name, I just borrowed
Anderson's, unless anyone can think of a better name. Anderson has the
paper available on his web pages (look for `eternity') at
http://www.cl.cam.ac.uk/users/rja14/ Also I heard that Eric Hughes
gave a talk on UPS `Universal Piracy Service' I presume this has
similar design goals, though I have not heard further details.)

I have implemented this to give something concrete to think about in
terms of creating something closer to Andersons real eternity service
in robustness.

The eternity server code above provides an interface to a distributed
document store which allows you to publish web pages in such a way
that the attacker (the attacker here is a censor such as perhaps a
government, perhaps your government) can not force you to unpublish
the document.

It's architecture is quite simple.  We already have an extremely
robust distributed document store: USENET news.  We already have the
ability to post messages to the document store anonymously: type I and
type II (mixmaster) remailers.  So theoretically we can publish what
ever we want, just post it to USENET.  However retreiving the data is
cumbersome.  We must search USENET with a search engine, or look
manually for the document, the document will expire after some short
time.  You may still be able to find the document in a search engine
which archives news for longer periods such as dejanews or altavista.

However, it is not as user friendly as a web based interface, there
are no hypertext links between related documents, there are no inline
graphics, etc.

So my `eternity server' is a kind of specialised web search engine
which acts as a kind of web proxy and provides a persistent web based
interface to reading web pages stored in encrypted documents posted to
different USENET groups.  The web search engine provides persitent
virtual URLs for documents even if they move around with different in
different news groups, being posted via different exit remailers.
Signatures on documents allows the user to update document.  The web
proxy has options to cache documents, and otherwise fetches from
a USENET newsspool or nntp news server.

Technical Description

The system offers different levels of security to users.  All
documents must be encrypted.  The subject field of the document should
include the SHA1 hash of the document's virtual URL.  (All eternity
virtual urls are of the form:

	http://[virtualsite].eternity/[documentpath]

that is they would not function as normal urls because the use the
non-existant TLD "eternity".  You may be able to use this facility to
get your browser to look at a particular eternity server to serve
documents of this form).

The lowest level of security is to encrypt the document with PGP -c
(IDEA encryption) with the passphrase "eternity".  This is just
obfuscation to make the documents marginally harder to examine in
USENET without using the eternity service to automate the process.

The next level of security is encrypt the document with the SHA1 of a
1 prepended with the URL.  ie 
	
	passphrase = SHA1( "1". "http://test.eternity/doc/" )

This means that you can not read documents unless you know the URL.
Of course, if you can guess the URL then you can read them.

The third level of security is to additionally to one of the above two
options to encrypt the document with another passphrase of your
chosing.  Then you can give the passphrase only to those amongst your
circle of friends.

Additionally support is given for signed documents.  This is important
because once you've published a web page you might wish to update it.
And yet you don't want other people updating it with a blank page (or
otherwise changing your page) as this would be as good as the ability
to remove the page.

If you include a PGP public key with a document, the eternity server
will associate that PGP key with the hash of your documents URL.  Any
attempts to update that document will only succeed if signed by the
same PGP key.

The document server has the option to cache documents.  Cacheing can
either be turned off (in which case documents are always fetched from
the news spool, this is not that inefficient as the news spool may be
local, and the server keeps a database of USENET group/article number
against document hash, and so can directly fetch the document), or
cacheing can be turned on, or cacheing can be set to encrypted, in
which case documents in the cache are encrypted with a passphrase
derived from the SHA1 of the document's URL by prepending a 1 to the
URL.  In the encrypted mode if the server operator is not keeping
logs, he can't decrypt the documents in his own cache.  If cacheing is
off there are no documents in the cache, only in the USENET news
spool.

In addition some users will have a password which is pasted into an
extra password field on the field

Users can set their own per document cacheing options (they can set no
cacheing to override the options of on or encrypted on the server), or
they can set encrytped to override cachine turned on, etc.

There is a document index at eternity root ( "http://eternity/" ), and
users may optionally request documents to be included in the directory
of eternity documents and virtual hosts.  It makes sense to include
the root document of your set of web pages in the index, and to leave
the rest out of the index.  Then you can link to your own pages from
your original page.

Relative URLs work, inline graphics work, site relative URLs work, and
it includes crude support for mime types, so most normal web data
types will work as normal.

The document format is:

[flags]
[pgp key]
[eternity document]

The flags are can be any of:

URL: http://test.eternity/example1/
Options: directory or exdirectory
Cache: on/yes/off/no/encrypt/encrypted
Description: An example eternity document

The pgp key is an asii armored PGP key, and the eternity document must
also be ascii armored and may optionally be signed (with the
acompanying pgp key).  The eternity document can also be encrypted
again inside that layer with a user key which need never be revealed
to the server.

The overall document is encrypted with PGP -c password "eternity", or
with a password obtained by prepending a 1 to the URL as obtained like
this.

% echo -c 1"http://test.eternity/example1/" | sha1

The flags, pgp key and eternity document can be arbitrarily jumbled
up in order.

The eternity server source code includes an examples directory.

The eternity server has a configuration file which is heavily
commented: eternity.conf.


Threat Models

Think of this as a specialised search engine. If you don't like the
documents you find, then change your search terms, don't complain to
one of the many search engine maintainers! The Eternity Service is
designed to facilitate publishing in a web based form of non
government approved reading or viewing material, or the publishing of
non government approved software, or other "liberated" documents,
programs, etc.  Each user can easily install their own search engine
in a shell account for their own use.

Newsgroups are very robust against attack because they are part of a
distributed system: USENET news. It is difficult to remove a USENET
newsgroup (the Scientologists know, they tried and failed, and they
have seriously deep pockets, and attitude to match). For the eternity
service it probably doesn't matter that much if you do get rid of a
particular newsgroup, eternity documents can be in any
newsgroup. Somehow I don't think you'll succeed in getting rid of all
USENET newsgroups, ie in shutting USENET news down. I'm not sure that
a major government could succeed in that even.

Information wants to be free. Long live USENET news.

Improving robustness

The main problem with this system that I see is that documents don't
live long in USENET news spools as expiry dates are often 1 week or
less.  So to use the service you'd have to set up a cron job to repost
your web documents every week or so.  Some documents remain in the
cache and so survive as long as there is interest in them.

An immediately easy thing to do is to include support for fetching and
looking documents up in altavista's news archive and in dejanews
archive.  This shouldn't be that difficult.  The eternity server
article database already includes newsgroup and message id against
document hash, all that is required is to formulate the requests
correctly.  If anyone wants to add this, feel free.

A mechanism for eternity servers to query each other's caches would be
one mechanism to think about.

Another would be to charge anonymous digicash rates to keep documents
in the cache a couple of $ per Mb/year.

Also you could perhaps think about charging for acccesses to the
service and passing a royalty back to the document author.

There is no `ownership' of eternity virtual domains, or of document
paths.  If Alice puts a document at http://books.eternity/, there is
nothing to stop Bob putting other files under that virtual domain,
such as http://books.eternity/crypto/.  However in general Bob can't
make Alice's document point to his URLs.  Bob can't modify or remove
Alice's pointer from the eternity directry http://eternity/.  But he
can add his own entry.

There can however be race conditions.  If Alice puts up a document on
http://books.eternity/ which points to an inline gif, but hasn't yet
put up that inline gif, Bob can get there first and put up a picture
of Barney until Alice notices and changes here pointer to point to the
picture she wants to point at.

So one solution would be to only accept documents which are signed by
the signator of the next domain up.  First come first served for the
virtual domains, the key signing the root dir of the virtual domain is
checked for each file or directory added under that.

Note even if the URLs are exdirectory, this need not be a problem as
you can just try the path with one directory stripped off.  Could be
done.  Is it worth it though?  It would add overhead to the server,
and there is minimal distruption someone can cause by sharing domains.

Perhaps it is desirable for someone to own a domain by virtue of being
the first person to submit a document with the hash of that domain
together with a key.

Comments?



The rest of this document is things on the to do list, if you feel
like helping out, or discover a limitation or bug.

To do list

It could do with some documentation.  And some stress testing.  The
file locking hasn't really been tested much.  NNTP may be flaky
as I made some mods since last debugging.

It would be nice to have a CGI based form to submitting documents.
Paste signed ascii armored document here.  Paste public key here.
Click radio buttons to choose options.  Enter URL here.  Type in
newsgroup to send to. Click here to submit through a chain of
mixmaster remailers.

There are a few bugs in the above system which ought to be fixed, but
this is alpha code, and they are on the TODO list.  One problem is
that the flags aren't signed.  A change of message format will fix
that in a later version.  The basic problem is that you can't check
the signature until you've decrypted and obtained the key.  At which
point you have to decrypt again.  A simple fix would perhaps to be to
put the public key outside the outer encryption envelope, but that is
unattractive in terms of allowing author correlation even by people
who don't have passphrases or don't know URLs.  Better would be to
just decrypt once, obtain the key if there is one, and then check the
sig again.  The server already keeps a database of PGP keys (not using
PGP's keyring) it stores keys by the SHA1 hash of the armored public
key and indexes against document URL hashes.  In this way public keys
do not need to be included in later document updates.  This could be
used to check a signature on the outer encryption layer.  Then only
the first post needs to decrypt twice (once without key available to
check sig, and then again with key available).  As a nice side effect
this protects against the PGP 0xdeadbeef attack (or 0xc001d00d as Gary
Howland has on his public key).

Another problem is when scanning a newsgroup, if the scanned message
can not be read it just stores it for later, so that when someone
accesses the server who knows the URL for that document it can be
decrypted and scanned for good signature and updated then.  However
the code only has a place holder for one document at the moment.  That
should be changed to a list and appropriate code to scan the list when
the document is accessed.  Once the newest document is found, the list
of documents to be scanned can be discarded and the newest article
stored against the hash in article.db.

Another more subtle problem is that the code may update documents in
the wrong order if multiple versions are posted to different news
groups.  The articles are scanned one group at a time.  So if the
groups are scanned in order alt.anonymous.messages followed by
alt.cypherpunks, and version 2 of the document is in alt.anon.msgs,
and version 1 in alt.cp, then it will update the document with the
older version.

Another to do list entry is the cache replacement policy.  There isn't
one, your cache will grow indefinately as documents are never removed.

Another thing to do is the altavista and dejanews lookup via HTTP by
message-id.  All the place holders are there.

If anyone wants to tackle any of the above tasks feel free.

The program dbmdump prints out .db files in ascii so you can see what
is there.

eternity -u is suitable to run as a cron job, or just at the console.

eternity -r resets everything, clears out the cache, empties database
etc.

eterity?url=http://blah.eternity/blah

is the form to use as a cgi.  It handles both GET and POST methods.

There is a log file for debugging, see eternity (the perl program) for
how to set different logging opitons. You can log most aspects of the
program for debugging purposes.

Adam

-- 
Have *you* exported RSA today? --> http://www.dcs.ex.ac.uk/~aba/rsa/

print pack"C*",split/\D+/,`echo "16iII*o\U@{$/=$z;[(pop,pop,unpack"H*",<>
)]}\EsMsKsN0[lN*1lK[d2%Sa2/d0<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<J]dsJxp"|dc`