User-Agent: * Disallow: /images/ Disallow: /kahin_to* Disallow: /dev/ Allow: / #To prevent only a select bots from any scans #User-agent: crawlers.looksmart.com #Disallow: / #To prenet any images from being scanned #Disallow: *.gif #Disallow: *.jpg #Disallow: *.png #To block main page due to size from bots #Disallow: /index.php #To shut down site completely from bots use # Disallow: / #Block out things that are secure or login oriented #These are based on not using any rewrite rules Disallow: / Disallow: /kahin_to* User-agent: Googlebot-Image Disallow: /