User-agent: * Sitemap: http://www.learnclubdance.com/sitemap.xml.gz Sitemap: http://www.learnclubdance.com/urllist.txt.gz Disallow: # User-agent: Googlebot # specific to Google # User-agent: Yahoo! Slurp # specific to Yahoo # Disallow: # allows all robots to crawl entire site # Disallow: /privatefolder/ # denies access to specified directories (in this case /privatefolder/ will be denied) # Disallow: / # keeps all robots out # Sitemap: # tells bot where to find sitemap for the website # Request-rate: 1/5 # maximum rate is one page every 5 seconds # Visit-time: 0600-0845 # only visit between 6:00 AM and 8:45 AM UT (GMT)