# robots.txt for http://web.org.uk/ # Delay Crawl-delay: 5 User-agent: * Disallow: /stats/ Disallow: /testing/ Disallow: /ssa/ Disallow: /images/ Disallow: /ipc/ Disallow: /maps/ Disallow: /resources/ Disallow: /picasso/high_res_images/