This post originated from an RSS feed registered with Python Buzz
by Ben Last.
Original Post: Hail Tobor
Feed Title: The Law Of Unintended Consequences
Feed URL: http://benlast.livejournal.com/data/rss
Feed Description: The Law Of Unintended Consequences
A snippet. A Zope robots.txt, as a Python Script, that returns different values for test and production sites. Handy if your development site is also open to the rest of the Net. The date and time are in there so I can see the last time the script ran; I recommend caching these via an AcceleratedHTTPCacheManager.
request = container.REQUEST
RESPONSE = request.RESPONSE
RESPONSE.setHeader('Content-type','text/plain')
# Return a string identifying this script.
host = request.SERVER_URL.lower()
if host.startswith('http://'): host = host[7:]
print "#Robots.txt for host %s" % host
print "#Generated "+str(DateTime())
#This is a list of elements that mark the host as being a
#development server. Edit to put your own in, or set
#devServer according to any criteria you please.
devMarks=['test','internal']
devServer = False
for m in devMarks:
if host.find(m) >= 0:
devServer = True
if devServer:
#running on the development server, request no indexing
#at all.
disallow = "/"
else:
#running on the production server,
#allow indexing of everything.
disallow = ""
print "\nUser-agent: *\nDisallow: %s\n" % disallow
#Add in prints for global disallows right here.
return printed
Previous Topic
