# Robots.txt for www.alldirectorylinks.com # Last Updated: 2025-12-22 # Total Blocked Bots: 320 User-agent: * Disallow: /login Disallow: /admin Disallow: /dashboard Disallow: /{locale}/login Disallow: /{locale}/admin Disallow: /{locale}/dashboard User-agent: GPTBot Disallow: /blog Disallow: / User-agent: ChatGPT Disallow: /blog Disallow: / User-agent: ChatGPT-User Disallow: /blog Disallow: / User-agent: OAI-SearchBot Disallow: /blog Disallow: / User-agent: OpenAI Disallow: /blog Disallow: / User-agent: ClaudeBot Disallow: /blog Disallow: / User-agent: Claude-Web Disallow: /blog Disallow: / User-agent: anthropic-ai Disallow: /blog Disallow: / User-agent: Google-Extended Disallow: /blog Disallow: / User-agent: GoogleOther Disallow: /blog Disallow: / User-agent: Storebot-Google Disallow: /blog Disallow: / User-agent: Google-InspectionTool Disallow: /blog Disallow: / User-agent: FacebookBot Disallow: /blog Disallow: / User-agent: Meta-ExternalAgent Disallow: /blog Disallow: / User-agent: Meta-ExternalFetcher Disallow: /blog Disallow: / User-agent: facebookexternalhit Disallow: /blog Disallow: / User-agent: facebookplatform Disallow: /blog Disallow: / User-agent: Facebot Disallow: /blog Disallow: / User-agent: Applebot-Extended Disallow: /blog Disallow: / User-agent: Amazonbot Disallow: /blog Disallow: / User-agent: AmazonBot Disallow: /blog Disallow: / User-agent: bingpreview Disallow: /blog Disallow: / User-agent: Bytespider Disallow: /blog Disallow: / User-agent: ByteDance Disallow: /blog Disallow: / User-agent: Bytedance Disallow: /blog Disallow: / User-agent: PerplexityBot Disallow: /blog Disallow: / User-agent: Perplexity Disallow: /blog Disallow: / User-agent: cohere-ai Disallow: /blog Disallow: / User-agent: Cohere Disallow: /blog Disallow: / User-agent: CCBot Disallow: /blog Disallow: / User-agent: commoncrawl Disallow: /blog Disallow: / User-agent: YouBot Disallow: /blog Disallow: / User-agent: Youbot Disallow: /blog Disallow: / User-agent: Neevabot Disallow: /blog Disallow: / User-agent: Neeva Disallow: /blog Disallow: / User-agent: Diffbot Disallow: /blog Disallow: / User-agent: diffbot Disallow: /blog Disallow: / User-agent: omgili Disallow: /blog Disallow: / User-agent: Omgilibot Disallow: /blog Disallow: / User-agent: Omgili Disallow: /blog Disallow: / User-agent: webzio Disallow: /blog Disallow: / User-agent: AI2Bot Disallow: /blog Disallow: / User-agent: ai2bot Disallow: /blog Disallow: / User-agent: HuggingFace Disallow: /blog Disallow: / User-agent: huggingface Disallow: /blog Disallow: / User-agent: StabilityAI Disallow: /blog Disallow: / User-agent: Laion Disallow: /blog Disallow: / User-agent: laion Disallow: /blog Disallow: / User-agent: MistralBot Disallow: /blog Disallow: / User-agent: Mistral Disallow: /blog Disallow: / User-agent: xAI Disallow: /blog Disallow: / User-agent: Grok Disallow: /blog Disallow: / User-agent: InflectionBot Disallow: /blog Disallow: / User-agent: Pi-AI Disallow: /blog Disallow: / User-agent: DeepMind Disallow: /blog Disallow: / User-agent: deepmind Disallow: /blog Disallow: / User-agent: WriteSonic Disallow: /blog Disallow: / User-agent: JasperAI Disallow: /blog Disallow: / User-agent: CopyAI Disallow: /blog Disallow: / User-agent: Grammarly Disallow: /blog Disallow: / User-agent: Wordtune Disallow: /blog Disallow: / User-agent: Quillbot Disallow: /blog Disallow: / User-agent: NotionBot Disallow: /blog Disallow: / User-agent: Runway Disallow: /blog Disallow: / User-agent: Midjourney Disallow: /blog Disallow: / User-agent: DALL-E Disallow: /blog Disallow: / User-agent: dalle Disallow: /blog Disallow: / User-agent: PetalBot Disallow: /blog Disallow: / User-agent: Petalbot Disallow: /blog Disallow: / User-agent: petalsearch Disallow: /blog Disallow: / User-agent: Sogou Disallow: /blog Disallow: / User-agent: sogou Disallow: /blog Disallow: / User-agent: 360Spider Disallow: /blog Disallow: / User-agent: HaoSouSpider Disallow: /blog Disallow: / User-agent: YisouSpider Disallow: /blog Disallow: / User-agent: YisouspiderAds Disallow: /blog Disallow: / User-agent: JDSpider Disallow: /blog Disallow: / User-agent: Qihoo Disallow: /blog Disallow: / User-agent: Baiduboxapp Disallow: /blog Disallow: / User-agent: DuerOS Disallow: /blog Disallow: / User-agent: AlibabaBot Disallow: /blog Disallow: / User-agent: Alibaba Disallow: /blog Disallow: / User-agent: TencentTraveler Disallow: /blog Disallow: / User-agent: Tencent Disallow: /blog Disallow: / User-agent: DataForSeoBot Disallow: /blog Disallow: / User-agent: dataforseo Disallow: /blog Disallow: / User-agent: SEOkicks Disallow: /blog Disallow: / User-agent: seokicks Disallow: /blog Disallow: / User-agent: serpstatbot Disallow: /blog Disallow: / User-agent: Serpstatbot Disallow: /blog Disallow: / User-agent: Seekport Disallow: /blog Disallow: / User-agent: MegaIndex Disallow: /blog Disallow: / User-agent: megaindex Disallow: /blog Disallow: / User-agent: BLEXBot Disallow: /blog Disallow: / User-agent: SeznamBot Disallow: /blog Disallow: / User-agent: linkdex Disallow: /blog Disallow: / User-agent: Linkdex Disallow: /blog Disallow: / User-agent: Exabot Disallow: /blog Disallow: / User-agent: ZoominfoBot Disallow: /blog Disallow: / User-agent: zoominfo Disallow: /blog Disallow: / User-agent: Screaming Frog SEO Spider Disallow: /blog Disallow: / User-agent: archive.org_bot Disallow: /blog Disallow: / User-agent: ia_archiver Disallow: /blog Disallow: / User-agent: Wayback Disallow: /blog Disallow: / User-agent: Turnitin Disallow: /blog Disallow: / User-agent: turnitin Disallow: /blog Disallow: / User-agent: Webrecorder Disallow: /blog Disallow: / User-agent: HeadlessChrome Disallow: /blog Disallow: / User-agent: PhantomJS Disallow: /blog Disallow: / User-agent: Phantom Disallow: /blog Disallow: / User-agent: Selenium Disallow: /blog Disallow: / User-agent: Puppeteer Disallow: /blog Disallow: / User-agent: Playwright Disallow: /blog Disallow: / User-agent: webdriver Disallow: /blog Disallow: / User-agent: scrapy Disallow: /blog Disallow: / User-agent: Scrapy Disallow: /blog Disallow: / User-agent: python-requests Disallow: /blog Disallow: / User-agent: python-urllib Disallow: /blog Disallow: / User-agent: aiohttp Disallow: /blog Disallow: / User-agent: httpx Disallow: /blog Disallow: / User-agent: axios Disallow: /blog Disallow: / User-agent: node-fetch Disallow: /blog Disallow: / User-agent: got Disallow: /blog Disallow: / User-agent: curl Disallow: /blog Disallow: / User-agent: wget Disallow: /blog Disallow: / User-agent: libwww Disallow: /blog Disallow: / User-agent: lwp-trivial Disallow: /blog Disallow: / User-agent: Java Disallow: /blog Disallow: / User-agent: okhttp Disallow: /blog Disallow: / User-agent: Go-http-client Disallow: /blog Disallow: / User-agent: Apache-HttpClient Disallow: /blog Disallow: / User-agent: http_request Disallow: /blog Disallow: / User-agent: HTTP_Request Disallow: /blog Disallow: / User-agent: python-httpx Disallow: /blog Disallow: / User-agent: Ruby Disallow: /blog Disallow: / User-agent: Mechanize Disallow: /blog Disallow: / User-agent: mechanize Disallow: /blog Disallow: / User-agent: Guzzle Disallow: /blog Disallow: / User-agent: GuzzleHttp Disallow: /blog Disallow: / User-agent: SimplePie Disallow: /blog Disallow: / User-agent: libcurl Disallow: /blog Disallow: / User-agent: RestSharp Disallow: /blog Disallow: / User-agent: Faraday Disallow: /blog Disallow: / User-agent: Typhoeus Disallow: /blog Disallow: / User-agent: Pingdom Disallow: /blog Disallow: / User-agent: StatusCake Disallow: /blog Disallow: / User-agent: UptimeRobot Disallow: /blog Disallow: / User-agent: Site24x7 Disallow: /blog Disallow: / User-agent: Datadog Disallow: /blog Disallow: / User-agent: NewRelic Disallow: /blog Disallow: / User-agent: AppDynamics Disallow: /blog Disallow: / User-agent: Dynatrace Disallow: /blog Disallow: / User-agent: Catchpoint Disallow: /blog Disallow: / User-agent: GTmetrix Disallow: /blog Disallow: / User-agent: WebPageTest Disallow: /blog Disallow: / User-agent: PageSpeed Disallow: /blog Disallow: / User-agent: Lighthouse Disallow: /blog Disallow: / User-agent: PTST Disallow: /blog Disallow: / User-agent: Postman Disallow: /blog Disallow: / User-agent: Insomnia Disallow: /blog Disallow: / User-agent: Nessus Disallow: /blog Disallow: / User-agent: Nikto Disallow: /blog Disallow: / User-agent: sqlmap Disallow: /blog Disallow: / User-agent: Acunetix Disallow: /blog Disallow: / User-agent: Nmap Disallow: /blog Disallow: / User-agent: masscan Disallow: /blog Disallow: / User-agent: ZmEu Disallow: /blog Disallow: / User-agent: zgrab Disallow: /blog Disallow: / User-agent: CensysInspect Disallow: /blog Disallow: / User-agent: Censys Disallow: /blog Disallow: / User-agent: Shodan Disallow: /blog Disallow: / User-agent: BinaryEdge Disallow: /blog Disallow: / User-agent: SecurityTrails Disallow: /blog Disallow: / User-agent: BuiltWith Disallow: /blog Disallow: / User-agent: Wappalyzer Disallow: /blog Disallow: / User-agent: WhatWeb Disallow: /blog Disallow: / User-agent: WPScan Disallow: /blog Disallow: / User-agent: Nuclei Disallow: /blog Disallow: / User-agent: Qualys Disallow: /blog Disallow: / User-agent: Tenable Disallow: /blog Disallow: / User-agent: Rapid7 Disallow: /blog Disallow: / User-agent: Burp Disallow: /blog Disallow: / User-agent: OWASP Disallow: /blog Disallow: / User-agent: dirbuster Disallow: /blog Disallow: / User-agent: gobuster Disallow: /blog Disallow: / User-agent: ffuf Disallow: /blog Disallow: / User-agent: feroxbuster Disallow: /blog Disallow: / User-agent: Feedly Disallow: /blog Disallow: / User-agent: Feedspot Disallow: /blog Disallow: / User-agent: FeedFetcher Disallow: /blog Disallow: / User-agent: Fever Disallow: /blog Disallow: / User-agent: NetNewsWire Disallow: /blog Disallow: / User-agent: Embedly Disallow: /blog Disallow: / User-agent: PagePeeker Disallow: /blog Disallow: / User-agent: Thumbshots Disallow: /blog Disallow: / User-agent: SocialRankIOBot Disallow: /blog Disallow: / User-agent: Slackbot Disallow: /blog Disallow: / User-agent: Discordbot Disallow: /blog Disallow: / User-agent: TelegramBot Disallow: /blog Disallow: / User-agent: WhatsApp Disallow: /blog Disallow: / User-agent: Viber Disallow: /blog Disallow: / User-agent: Twitterbot Disallow: /blog Disallow: / User-agent: LinkedInBot Disallow: /blog Disallow: / User-agent: Pinterest Disallow: /blog Disallow: / User-agent: Snapchat Disallow: /blog Disallow: / User-agent: Outlook Disallow: /blog Disallow: / User-agent: Thunderbird Disallow: /blog Disallow: / User-agent: Mail.RU Disallow: /blog Disallow: / User-agent: Gigabot Disallow: /blog Disallow: / User-agent: Grapeshot Disallow: /blog Disallow: / User-agent: Hatena Disallow: /blog Disallow: / User-agent: ICC-Crawler Disallow: /blog Disallow: / User-agent: IonCrawl Disallow: /blog Disallow: / User-agent: LivelapBot Disallow: /blog Disallow: / User-agent: ltx71 Disallow: /blog Disallow: / User-agent: Mindbreeze Disallow: /blog Disallow: / User-agent: newspaper Disallow: /blog Disallow: / User-agent: panscient Disallow: /blog Disallow: / User-agent: Purebot Disallow: /blog Disallow: / User-agent: Qwantify Disallow: /blog Disallow: / User-agent: Seekr Disallow: /blog Disallow: / User-agent: VoilaBot Disallow: /blog Disallow: / User-agent: Zeus Disallow: /blog Disallow: / User-agent: woobot Disallow: /blog Disallow: / User-agent: ZoomBot Disallow: /blog Disallow: / User-agent: VelenPublicWebCrawler Disallow: /blog Disallow: / User-agent: MBCrawler Disallow: /blog Disallow: / User-agent: SurdotlyBot Disallow: /blog Disallow: / User-agent: ljsebot Disallow: /blog Disallow: / User-agent: AppEngine-Google Disallow: /blog Disallow: / User-agent: Barkrowler Disallow: /blog Disallow: / User-agent: BomboraBot Disallow: /blog Disallow: / User-agent: buck Disallow: /blog Disallow: / User-agent: Cincraw Disallow: /blog Disallow: / User-agent: Clickagy Disallow: /blog Disallow: / User-agent: coccocbot Disallow: /blog Disallow: / User-agent: Crowsnest Disallow: /blog Disallow: / User-agent: Daum Disallow: /blog Disallow: / User-agent: Dispatch Disallow: /blog Disallow: / User-agent: Kaspersky Disallow: /blog Disallow: / User-agent: LightspeedSystems Disallow: /blog Disallow: / User-agent: Linguee Disallow: /blog Disallow: / User-agent: MojeekBot Disallow: /blog Disallow: / User-agent: Netcraft Disallow: /blog Disallow: / User-agent: NetSystemsResearch Disallow: /blog Disallow: / User-agent: Nimbostratus Disallow: /blog Disallow: / User-agent: ips-agent Disallow: /blog Disallow: / User-agent: Mediatoolkitbot Disallow: /blog Disallow: / User-agent: PiplBot Disallow: /blog Disallow: / User-agent: scoop.it Disallow: /blog Disallow: / User-agent: Scraperapi Disallow: /blog Disallow: / User-agent: sitecheck.internetseer Disallow: /blog Disallow: / User-agent: BrandVerity Disallow: /blog Disallow: / User-agent: linkfluence Disallow: /blog Disallow: / User-agent: rogerbot Disallow: /blog Disallow: / User-agent: SiteAuditBot Disallow: /blog Disallow: / User-agent: domainstatsbot Disallow: /blog Disallow: / User-agent: DomainsProject Disallow: /blog Disallow: / User-agent: DomainCrawler Disallow: /blog Disallow: / User-agent: integromedb Disallow: /blog Disallow: / User-agent: proximic Disallow: /blog Disallow: / User-agent: NerdyBot Disallow: /blog Disallow: / User-agent: Cliqzbot Disallow: /blog Disallow: / User-agent: adbeat_bot Disallow: /blog Disallow: / User-agent: CriteoBot Disallow: /blog Disallow: / User-agent: SemrushBot-BA Disallow: /blog Disallow: / User-agent: SemrushBot-SA Disallow: /blog Disallow: / User-agent: SemrushBot-SWA Disallow: /blog Disallow: / User-agent: AdsBot Disallow: /blog Disallow: / User-agent: Mediapartners Disallow: /blog Disallow: / User-agent: adidxbot Disallow: /blog Disallow: / User-agent: Pinterestbot Disallow: /blog Disallow: / User-agent: spbot Disallow: /blog Disallow: / User-agent: RSSingBot Disallow: /blog Disallow: / User-agent: Twisted Disallow: /blog Disallow: / User-agent: Jersey Disallow: /blog Disallow: / User-agent: Apache-Ant Disallow: /blog Disallow: / User-agent: Wget Disallow: /blog Disallow: / User-agent: Curl Disallow: /blog Disallow: / User-agent: Python-urllib Disallow: /blog Disallow: / User-agent: HTMLParser Disallow: /blog Disallow: / User-agent: HttpUrlConnection Disallow: /blog Disallow: / User-agent: jakarta Disallow: /blog Disallow: / User-agent: rest-client Disallow: /blog Disallow: / User-agent: YandexMobileBot Disallow: /blog Disallow: / User-agent: YandexAccessibilityBot Disallow: /blog Disallow: / User-agent: YandexDirectDyn Disallow: /blog Disallow: / User-agent: YandexImages Disallow: /blog Disallow: / User-agent: YandexVideo Disallow: /blog Disallow: / User-agent: YandexMedia Disallow: /blog Disallow: / User-agent: YandexBlogs Disallow: /blog Disallow: / User-agent: YandexFavicons Disallow: /blog Disallow: / User-agent: YandexWebmaster Disallow: /blog Disallow: / User-agent: YandexPagechecker Disallow: /blog Disallow: / User-agent: YandexImageResizer Disallow: /blog Disallow: / User-agent: YandexAdNet Disallow: /blog Disallow: / User-agent: YandexDirect Disallow: /blog Disallow: / User-agent: YaDirectFetcher Disallow: /blog Disallow: / User-agent: YandexCalendar Disallow: /blog Disallow: / User-agent: YandexSitelinks Disallow: /blog Disallow: / User-agent: YandexMetrika Disallow: /blog Disallow: / User-agent: YandexNews Disallow: /blog Disallow: / User-agent: YandexCatalog Disallow: /blog Disallow: / User-agent: YandexAntivirus Disallow: /blog Disallow: / User-agent: YandexMarket Disallow: /blog Disallow: / User-agent: YandexVertis Disallow: /blog Disallow: / User-agent: YandexForDomain Disallow: /blog Disallow: / User-agent: YandexSpravBot Disallow: /blog Disallow: / User-agent: YandexSearchShop Disallow: /blog Disallow: / User-agent: YandexMedianaBot Disallow: /blog Disallow: / User-agent: YandexOntoDBAPI Disallow: /blog Disallow: / User-agent: YandexOntoDB Disallow: /blog Disallow: / User-agent: YandexTurbo Disallow: /blog Disallow: / User-agent: YandexVerticals Disallow: /blog Disallow: / User-agent: SamsungBrowser Disallow: /blog Disallow: / User-agent: mj12 Disallow: /blog Disallow: / User-agent: MJ12bot Disallow: /blog Disallow: / User-agent: httpclient Disallow: /blog Disallow: / User-agent: phpcrawl Disallow: /blog Disallow: / User-agent: seoscanners Disallow: /blog Disallow: / Sitemap: https://www.alldirectorylinks.com/sitemap/sitemap.xml