Robots.txt Examples
A robots.txt file is a special text file that is always located in your Web server's root directory. This file contains restrictions for Web Spiders, telling them where they have permission to search. It should be noted that Web Robots are not required to respect robots.txt files, but most well-written Web Spiders follow the rules you define.
User-agent: ASPSeek
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BravoBrian
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Dumbot
Disallow: /
User-agent: EgotoBot
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: Generic
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: Larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: NaverRobot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NexaBot
Disallow: /
User-agent: OWR_Crawler
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gatherer
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: PhpDig
Disallow: /
User-agent: ProPowerBot
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: RMA
Disallow: /
User-agent: RPT-HTTPClient
Disallow: /
User-agent: Radiation Retriever
Disallow: /
User-agent: Reaper
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: ScoutAbout
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: Steeler
Disallow: /
User-agent: Szukacz
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: Teleport Pro
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: Tight****Bot
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: TutorGig
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebCapture
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: Wget
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: dloader(NaverRobot)
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: jyxobot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: moget
Disallow: /
User-agent: obot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: puf
Disallow: /
User-agent: rabaz
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: semanticdiscovery
Disallow: /
User-agent: spanner
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: turingos
Disallow: /
User-agent: vsecrawler
Disallow: /
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BravoBrian
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Dumbot
Disallow: /
User-agent: EgotoBot
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: Generic
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: Larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: NaverRobot
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NexaBot
Disallow: /
User-agent: OWR_Crawler
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gatherer
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: PhpDig
Disallow: /
User-agent: ProPowerBot
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: RMA
Disallow: /
User-agent: RPT-HTTPClient
Disallow: /
User-agent: Radiation Retriever
Disallow: /
User-agent: Reaper
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: ScoutAbout
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: Steeler
Disallow: /
User-agent: Szukacz
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: Teleport Pro
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: Tight****Bot
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: TutorGig
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebCapture
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: Wget
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: dloader(NaverRobot)
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: jyxobot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: moget
Disallow: /
User-agent: obot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: puf
Disallow: /
User-agent: rabaz
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: semanticdiscovery
Disallow: /
User-agent: spanner
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: turingos
Disallow: /
User-agent: vsecrawler
Disallow: /