# Define access-restrictions for robots/spiders # http://www.robotstxt.org/wc/norobots.html # Add Googlebot-specific syntax extension to exclude forms # that are repeated for each piece of content in the site # the wildcard is only supported by Googlebot # http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling User-agent: Googlebot Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /*contact-info Disallow: /*images User-agent: Linguee Disallow: / User-agent: Slurp Crawl-delay: 50 Disallow: /contact-info User-agent: Exabot Disallow: / User-agent: * Disallow: /contact-info User-agent: Googlebot-Image Disallow: / User-agent: twiceler Disallow: / User-agent: Speedy Disallow: / User-agent: discobot Disallow: / User-agent: sogou+spider Disallow: / User-agent: Yandex Disallow: / User-agent: kalooga Disallow: / User-agent: dotbot Disallow: / User-agent: ScoutJet Disallow: / User-agent: Vagabondo Disallow: /