[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Spiderspace
>... I was under the impression that the only documents that most web crawlers
>will search are documents that are link-accessible. Are you saying that this
>isn't true? Are you saying that Alta-Vista will search EVERYTHING that's
>publicly accessible, whether by anonymous FTP or web?
Don't archie servers already pick up the anonymous ftp fairly well?
Also, aside from no-robots conventions, you can build a cgi program for
access to files that might be more effective at blocking searches
while still preserving access.
Also, it wouldn't be hard for a web-crawler to follow ftp links,
as long as the root of an anon-ftp site is pointed to by a URL somewhere.
#--
# Thanks; Bill
# Bill Stewart, [email protected], Pager/Voicemail 1-408-787-1281
#
# "Eternal vigilance is the price of liberty" used to mean us watching
# the government, not the other way around....