# For domain: http://a1uk.net # All robots will spider the domain User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow:/ User-agent: PycURL/7.18.2 Disallow:/ User-agent: * Disallow: Crawl-delay: 120 Disallow: /invoices/ Disallow: /notfor/ Disallow: /mypc/ Disallow: /Gp/ Disallow: /personal/ Disallow: /personal2/ Disallow: /secure/ Disallow: /dietguides/dlxcvbnmqwerty/ Disallow: /*/*/.doc