User-Agent: Googlebot Allow: / #recommended by Site Pro News article by Mike Banks Valentine of Publish101.com: User-Agent: msnbot Crawl-delay: 10 User-Agent: Teoma Crawl-delay: 10 User-Agent: Slurp Crawl-delay: 10 User-Agent: aipbot Disallow: / User-Agent: BecomeBot Disallow: / User-Agent: psbot Disallow: / # this identifies the wayback machine User-Agent: User-Agent: ia_archiver Disallow: / # the NameProtect.com bot User-Agent: NPBot Disallow: /