User-agent: * Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /images/ Disallow: /include/ Disallow: /languages/ Disallow: /settings/ Request-rate: 1/5 # maximum rate is one page every 5 seconds Crawl-delay: 10