Loading

robots.txt User bot

  1. User-agent: AhrefsBot
  2. User-agent: Alexibot
  3. User-agent: AppEngine
  4. User-agent: Aqua_Products
  5. User-agent: archive.org_bot
  6. User-agent: archive
  7. User-agent: asterias
  8. User-agent: b2w/0.1
  9. User-agent: BackDoorBot/1.0
  10. User-agent: BecomeBot
  11. User-agent: BlekkoBot
  12. User-agent: Blexbot
  13. User-agent: BlowFish/1.0
  14. User-agent: Bookmark search tool
  15. User-agent: BotALot
  16. User-agent: BuiltBotTough
  17. User-agent: Bullseye/1.0
  18. User-agent: BunnySlippers
  19. User-agent: CCBot
  20. User-agent: CheeseBot
  21. User-agent: CherryPicker
  22. User-agent: CherryPickerElite/1.0
  23. User-agent: CherryPickerSE/1.0
  24. User-agent: chroot
  25. User-agent: Copernic
  26. User-agent: CopyRightCheck
  27. User-agent: cosmos
  28. User-agent: Crescent
  29. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  30. User-agent: DittoSpyder
  31. User-agent: dotbot
  32. User-agent: dumbot
  33. User-agent: EmailCollector
  34. User-agent: EmailSiphon
  35. User-agent: EmailWolf
  36. User-agent: Enterprise_Search
  37. User-agent: Enterprise_Search/1.0
  38. User-agent: EroCrawler
  39. User-agent: es
  40. User-agent: exabot
  41. User-agent: ExtractorPro
  42. User-agent: FairAd Client
  43. User-agent: Flaming AttackBot
  44. User-agent: Foobot
  45. User-agent: Gaisbot
  46. User-agent: GetRight/4.2
  47. User-agent: gigabot
  48. User-agent: grub
  49. User-agent: grub-client
  50. User-agent: Go-http-client
  51. User-agent: Harvest/1.5
  52. User-agent: Hatena Antenna
  53. User-agent: hloader
  54. User-agent: http://www.SearchEngineWorld.com bot
  55. User-agent: http://www.WebmasterWorld.com bot
  56. User-agent: httplib
  57. User-agent: humanlinks
  58. User-agent: ia_archiver
  59. User-agent: ia_archiver/1.6
  60. User-agent: InfoNaviRobot
  61. User-agent: Iron33/1.0.2
  62. User-agent: JamesBOT
  63. User-agent: JennyBot
  64. User-agent: Jetbot
  65. User-agent: Jetbot/1.0
  66. User-agent: Jorgee
  67. User-agent: Kenjin Spider
  68. User-agent: Keyword Density/0.9
  69. User-agent: larbin
  70. User-agent: LexiBot
  71. User-agent: libWeb/clsHTTP
  72. User-agent: LinkextractorPro
  73. User-agent: LinkpadBot
  74. User-agent: LinkScan/8.1a Unix
  75. User-agent: LinkWalker
  76. User-agent: LNSpiderguy
  77. User-agent: looksmart
  78. User-agent: lwp-trivial
  79. User-agent: lwp-trivial/1.34
  80. User-agent: Mata Hari
  81. User-agent: Megalodon
  82. User-agent: Microsoft URL Control
  83. User-agent: Microsoft URL Control - 5.01.4511
  84. User-agent: Microsoft URL Control - 6.00.8169
  85. User-agent: MIIxpc
  86. User-agent: MIIxpc/4.2
  87. User-agent: Mister PiX
  88. User-agent: MJ12bot
  89. User-agent: moget
  90. User-agent: moget/2.1
  91. User-agent: mozilla
  92. User-agent: Mozilla
  93. User-agent: mozilla/3
  94. User-agent: mozilla/4
  95. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
  96. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
  97. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
  98. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
  99. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
  100. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
  101. User-agent: mozilla/5
  102. User-agent: MSIECrawler
  103. User-agent: naver
  104. User-agent: NerdyBot
  105. User-agent: NetAnts
  106. User-agent: NetMechanic
  107. User-agent: NICErsPRO
  108. User-agent: Nutch
  109. User-agent: Offline Explorer
  110. User-agent: Openbot
  111. User-agent: Openfind
  112. User-agent: Openfind data gathere
  113. User-agent: Oracle Ultra Search
  114. User-agent: PerMan
  115. User-agent: ProPowerBot/2.14
  116. User-agent: ProWebWalker
  117. User-agent: psbot
  118. User-agent: Python-urllib
  119. User-agent: QueryN Metasearch
  120. User-agent: Radiation Retriever 1.1
  121. User-agent: RepoMonkey
  122. User-agent: RepoMonkey Bait & Tackle/v1.01
  123. User-agent: RMA
  124. User-agent: rogerbot
  125. User-agent: scooter
  126. User-agent: Screaming Frog SEO Spider
  127. User-agent: searchpreview
  128. User-agent: SEMrushBot
  129. User-agent: SemrushBot
  130. User-agent: SemrushBot-SA
  131. User-agent: SEOkicks-Robot
  132. User-agent: SiteSnagger
  133. User-agent: sootle
  134. User-agent: SpankBot
  135. User-agent: spanner
  136. User-agent: spbot
  137. User-agent: Stanford
  138. User-agent: Stanford Comp Sci
  139. User-agent: Stanford CompClub
  140. User-agent: Stanford CompSciClub
  141. User-agent: Stanford Spiderboys
  142. User-agent: SurveyBot
  143. User-agent: SurveyBot_IgnoreIP
  144. User-agent: suzuran
  145. User-agent: Szukacz/1.4
  146. User-agent: Szukacz/1.4
  147. User-agent: Teleport
  148. User-agent: TeleportPro
  149. User-agent: Telesoft
  150. User-agent: Teoma
  151. User-agent: The Intraformant
  152. User-agent: TheNomad
  153. User-agent: toCrawl/UrlDispatcher
  154. User-agent: True_Robot
  155. User-agent: True_Robot/1.0
  156. User-agent: turingos
  157. User-agent: Typhoeus
  158. User-agent: URL Control
  159. User-agent: URL_Spider_Pro
  160. User-agent: URLy Warning
  161. User-agent: VCI
  162. User-agent: VCI WebViewer VCI WebViewer Win32
  163. User-agent: Web Image Collector
  164. User-agent: WebAuto
  165. User-agent: WebBandit
  166. User-agent: WebBandit/3.50
  167. User-agent: WebCopier
  168. User-agent: WebEnhancer
  169. User-agent: WebmasterWorld Extractor
  170. User-agent: WebmasterWorldForumBot
  171. User-agent: WebSauger
  172. User-agent: Website Quester
  173. User-agent: Webster Pro
  174. User-agent: WebStripper
  175. User-agent: WebVac
  176. User-agent: WebZip
  177. User-agent: WebZip/4.0
  178. User-agent: Wget
  179. User-agent: Wget/1.5.3
  180. User-agent: Wget/1.6
  181. User-agent: WWW-Collector-E
  182. User-agent: Xenu's
  183. User-agent: Xenu's Link Sleuth 1.1c
  184. User-agent: Zeus
  185. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  186. User-agent: Zeus Link Scout
  187. Disallow: /

Comments