Sitemap: http://www.theportsmouthchorus/sitemap.xml User-agent: Googlebot-Image Disallow: /book/ Disallow: /books/ Disallow: /stock/ Disallow: /cgi-bin/ Disallow: /stock/ # Education bot to help avoid student's plagiarism User-agent: TurnitinBot Disallow: / User-agent: psbot Disallow: / User-agent: Sogou Disallow: / User-agent: Sogou web spider Disallow: / # Internet Archive User-agent: ia_archiver Disallow: / User-agent: * Disallow: /h/ Disallow: /php/ Disallow: /search/ Disallow: /flash/ Disallow: /music-cd/ Disallow: /music-cds/ Disallow: /music/ Disallow: /cgi-bin/ Disallow: /img/ Disallow: /images/ Disallow: /stock/ Disallow: /*.php$ # for a in www.*; do cp www.loanspage.co.uk/web/robots.txt $a/web/robots.txt; done;