Making the Best Use of the Robots Text File

Tuesday, May 13th, 2008 Posted in Onsite SEO, Tools | No Comments »

Place a robots.txt file on every domain. The first thing a spider looks for when it hits your Web site is a robots.txt file. If it's looking for that file, put it on your site! A robots.txt file will keep ...