The Artima Developer Community
Articles | News | Weblogs | Books | Forums
Artima Forums | Articles | Weblogs | Java Answers | News
Sponsored Link •

Java Buzz Forum
Must Read: Google Robots.txt Parsing Weirdness

0 replies on 1 page.

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 0 replies on 1 page
Angsuman Chakraborty

Posts: 19816
Nickname: angsuman
Registered: Dec, 2003

Angsuman Chakraborty is the CEO and Chief Architect at Taragana
Must Read: Google Robots.txt Parsing Weirdness Posted: Jul 7, 2008 12:02 PM
Reply to this message Reply

This post originated from an RSS feed registered with Java Buzz by Angsuman Chakraborty.
Original Post: Must Read: Google Robots.txt Parsing Weirdness
Feed Title: Simple Thoughts - Java and Web Technology Blog
Feed URL: http://tech.gaeatimes.com/index.php/feed/
Feed Description: News and views as a Software Architect and Entrepreneur on Java and Web Software. 		Latest Java Buzz Posts
Latest Java Buzz Posts by Angsuman Chakraborty
Latest Posts From Simple Thoughts - Java and Web Technology Blog

Advertisement
Google search bot (Googlebot) parses robots.txt file to find excluded sections of the site like any good webbots. However unlike the other bots, Google bot behaves differently when you have a section for all bots (*) as well as a section specifically for Googlebot. Let’s say you have a section in robots.txt for all bots beginning [...]

Read: Must Read: Google Robots.txt Parsing Weirdness

Topic: Save the JMS for last Previous Topic   Next Topic Topic: Don’t confuse SaaS with Cloud Computing - Cloud Conference Week, Part 4 from PeopleOverProcess.com
Sponsored Links


Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use