User-agent: * Disallow: /2HG.css User-agent: * Disallow: /email-addresses/ User-agent: * Disallow: /cgi-bin/ User-agent" * Disallow: /images/ User-agent: * Disallow: /nav3/ User-agent: * Disallow: /midi/ User-agent: abc/Nutch-0.9-dev (abc; http://abc#11.us; abc at abc dot com) Disallow: / User-agent: AboutUsBot Disallow: / User-agent: Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) Disallow: / User-agent: Alexibot Disallow: / User-agent: altavista Disallow: / User-agent: altavista.com/image/randomlink Disallow: / User-agent: altavistaimages.com Disallow: / User-agent: appie 1.1 (www.walhello.com) Disallow: / User-agent: asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) Disallow: / User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) Disallow: / User-agent: Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) Disallow: / User-agent: BecomeBot Disallow: / User-agent: Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) Disallow: / User-agent: Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) Disallow: / User-agent: bot/1.0 (bot; http://; bot@bot.bot) Disallow: / User-agent: Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) Disallow: / User-agent: CatchBot/1.0; +http://www.catchbot.com Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) Disallow: / User-agent: CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) Disallow: / User=agent: CCBot/1.0 (+http://www.commoncrawl.org/bot.html) Disallow: / User-agent: ccubee/3.5 Disallow: / User-agent: Charlotte Disallow: / User-agent: ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) Disallow: / User-agent: ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) Disallow: / User-agent: del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) Disallow: / User-agent: DiamondBot/2.0 Disallow: / User-agent: dragonfly(ebingbong@playstarmusic.com) Disallow: / User-agent: eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at global-opto dot com) Disallow: / User-agent: ejupiter.com Disallow: / User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html) Disallow: / User-agent: envolk Disallow: / User-agent: envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Disallow: / User-agent: Eurobot/Nutch-1.0-dev (1.0) Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: Exabot-Test/1.0 Disallow: / User-agent: exactseek.com Disallow: / User-agent: exooba/exooba crawler (exooba; exooba) Disallow: / User-agent: factbot Disallow: / User-agent: findlinks Disallow: / User-agent: GeonaBot/1.2; http://www.geona.com/ Disallow: / User-agent: Gigabot Disallow: / User-agent: GOFORITBOT ( http://www.goforit.com/about/ ) Disallow: / User-agent: Googlebot-Image/1.0 Disallow: / User-agent: Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org) Disallow: / User-agent: gsa-crawler (Enterprise; S4-K9PMMWZNHQJAS; greg.ryman@bnw.com,aaron.miller@bnw.com) Disallow: / User-agent: GT::WWW/1.026 Disallow: / User-agent: GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Disallow: / User-agent: HouxouCrawler/Nutch-0.9-dev (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.houxou.com/crawler; crawler at houxou dot com) Disallow: / User-agent: i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) Disallow: / User-agent: ia_archiver Disallow: / User-agent: ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: ilial/Nutch-0.9-dev Disallow: / User-agent: IlseBot/1.0 Disallow: / User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) Disallow: / User-agent: !Susie (http://www.sync2it.com/susie) Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.6.0-beta Disallow: / User-agent: Java/1.6.0-rc Disallow: / User-agent: KnowItAll/Nutch-0.9 (Nutch-UW-Crawler; http://cs.washington.edu/homes/mjc/crawler.html; uwcrawler08@gmail.com) Disallow: / User-agent: LapozzBot/1.4 (+http://robot.lapozz.com) Disallow: / User-agent: LinkWalker Disallow: / User-agent: LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) Disallow: / User-agent: MJ12bot Disallow: / User-agent: Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Disallow: / User-agent: Mozilla/4.0 (compatible; DAUMOA-web; +http://ws.daum.net/aboutkr.html) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; Comcast; ZangoToolbar 4.8.3; SpamBlockerUtility 4.8.0; Comcast) Disallow: / User-agent: Mozilla/4.0 (compatible; Spider; Linux) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Disallow: / User-Agent: Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Disallow: / User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) Disallow: / User-agent: Mozilla/4.0 (compatible; NaverBot/1.0; nhnbot@naver.com) Disallow: / User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com) Disallow: / User-agent: Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) Disallow: / User-agent: Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) Disallow: / User-agent: Mozilla/5.0 (compatible; del.icio.us-thumbnails/1.0; FreeBSD) KHTML/4.3.2 (like Gecko) Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/1.15.1-200807172326 +http://www.accelobot.com) Disallow: / User-agent: Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html) Disallow: / User-agent: Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) Disallow: / User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7 Disallow: / User-agent: Mozilla/5.0 (X11; I; Linux 2.6.22-gentoo-r8 x86_64) Gecko/20071115 Firefox/2.0.0.10 Disallow: / User-agent: Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com) Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/1.8.0 +http://wiki.office.aol.com/wiki/SEO) Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com] Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.pubsub.com) Disallow: / User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+) Disallow: / User-agent: Mozilla/5.0 (compatible; nextthing.org/1.0; +http://www.nextthing.org/bot) Disallow: / User-agent: Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) Disallow: / User-agent: Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) Disallow: / User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0/1.0 (bot; http://; bot@bot.com) Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; +http://process4.com) Gecko/20070508 Firefox/1.5.0.12 Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http://www.voila.com/) Disallow: / User-agent: MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) Disallow: / User-agent: msnbot-media Disallow: / User-agent: MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.html Disallow: / User-agent: NaverBot Disallow: / User-agent: NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com) Disallow: / User-Agent: NetResearchServer Disallow: / User-agent: NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) Disallow: / User-agent: NG-Search/0.9.8 (NG-SearchBot; http://www.ng-search.com) Disallow: / User-agent: nicebot Disallow: / User-agent: NimbleCrawler Disallow: / User-agent: noxtrumbot/1.0 (crawler@noxtrum.com) Disallow: / User-agent: nnn/ttt (n) Disallow: / User-agent: Nusearch Spider (www.nusearch.com) Disallow: / User-agent: Nutch Disallow: / User-agent: nutch/Nutch-1.0-dev (nutch) Disallow: / User-agent: Nutch/Nutch-0.8.1 (Nutch; Nutch; Nutch) Disallow: / User-agent: NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) Disallow: / User-agent: NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) Disallow: / User-agent: nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) Disallow: / User-agent: panscient.com Disallow: / User-agent: pathfinder.gr Disallow: / User-agent: PHP version tracker (http://www.nexen.net/phpversion/bot.php) Disallow: / User-agent: Pingdom GIGRIB (http://www.pingdom.com) Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 (+http://www.picsearch.com/bot.html) Disallow: / User-agent: Python-urllib/1.15 Disallow: / User-agent: RGScan/Nutch-0.9 (pchang at riverglassinc) Disallow: / User-agent: SBIder Disallow: / User-agent: sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Disallow: / User-agent: Scooter/3.3 Disallow: / User-agent: Sean L. Scurlock/Nutch-0.9 (BibleGuru Spider; http://www.bibleguru.com ; sean@bibleguru.com) Disallow: / User-agent: ShablastBot 1.0 Disallow: / User-agent: Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) Disallow: / User-agent: Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) Disallow: / User-agent: silk/1.0 Disallow: / User-agent: SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com) Disallow: / User-agent: sogou spider Disallow: / User-agent: Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) Disallow: / User-agent: Sproose Disallow: / User-agent: sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com) Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: szukacz Disallow: / User-agent: Test Spider 0.1 Disallow: / User-agent: Test Spider 0.2 Disallow: / User-agent: TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) Disallow: / User-agent: Twiceler www.cuill.com/robots.html Disallow: / User-agent: VadixBot Disallow: / User-agent: VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager-hc/1.0 Disallow: / User-agent: WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) Disallow: / User-agent: webcollage Disallow: / User-agent: WinHTTP Robot/1.0 Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; I) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; H) Disallow: / User-agent: Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386) Disallow: / User-agent: YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com) Disallow: / User-agent: zedzo.digest/0.1 (http://www.zedzo.com/) Disallow: / User-agent: zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com] Disallow: / User-agent: ZyBorg Disallow: /