# crawler.sh

> AI-optimized mirror of crawler.sh containing 50 pages totalling 20,344 words of clean markdown content, structured data, and semantic HTML. Original source: https://crawler.sh/. Last updated: 2026-04-30T23:42:29.586Z.

Each page is available as HTML (with JSON-LD structured data) and Markdown (text-only, ideal for LLMs and RAG).

## Homepage

- [Crawl, Analyze & Export Any Website in Seconds](/site-root.html): Crawl any website in seconds. Find SEO issues, extract clean content, and export to JSON, Sitemap XML, or CSV — from the terminal or a desktop app. (299 words)

## Articles & Blog Posts

- [Product Roadmap](/roadmap/index.html): See what's coming next for crawler.sh. View planned features, improvements, and the development roadmap for the CLI and desktop app. (327 words)
- [docs/cli/reference/index.html](/docs/cli/reference/index.html) (1 words)
- [blog/technical-seo-audits/index.html](/blog/technical-seo-audits/index.html) (1 words)
- [Why Preference Data Is the Bottleneck](/blog/challenges-collecting-preference-data-rlhf/index.html): The hardest problems in RLHF data pipelines - from annotator disagreement and label noise to scaling preference collection and keeping training data fresh. (1,866 words)
- [CLI Features - Commands and Crawl Engine](/docs/cli/features/index.html): Crawl engine behavior, subcommands, SEO analysis, content extraction, and the .crawl file format (1,259 words)
- [blog/eeat-checklist-guideline/index.html](/blog/eeat-checklist-guideline/index.html) (1 words)
- [Crawler CLI Overview](/docs/cli/index.html): A fast, concurrent web crawler CLI with SEO analysis (966 words)
- [Desktop Features - Dashboard and SEO Analysis](/docs/desktop/features/index.html): Dashboard cards, SEO analysis, content extraction, and UI/UX (821 words)
- [What Is Answer Engine Optimization?](/blog/answer-engine-optimization/index.html): Learn what Answer Engine Optimization is, why it matters, and how to make your content visible to AI search engines like ChatGPT and Perplexity. (1,471 words)
- [Why MLOps Teams Need a Web Crawler](/blog/best-web-crawler-for-mlops/index.html): Why crawler.sh is the best web crawler for MLOps pipelines. Fast Rust-powered crawling, clean content extraction, JSON export, and CI/CD automation for ML teams. (1,401 words)
- [SEO Glossary](/glossary/index.html): Definitions of common SEO and web crawling terms. Learn about crawl budget, meta descriptions, canonical URLs, redirect chains, and more. (369 words)
- [What’s New in v0.7.0](/changelog/v070-javascript-rendering/index.html): Crawl JavaScript-heavy websites with built-in JS rendering. Auto-detects client-side frameworks and renders pages without a browser. (609 words)
- [Broken links](/glossary/broken-links/index.html): Outgoing links and images that return 404 or fail to connect hurt user trust and waste crawl budget (400 words)
- [Download crawler.sh](/download/index.html): Download the crawler.sh desktop app for macOS and Linux, or install the CLI with a single command. Crawl any website in seconds — no account required. (159 words)
- [What’s New in v0.6.5](/changelog/v065-single-page-fetch/index.html): New crawler fetch command to grab a single URL with smart, path-based filenames. Skip the full crawl when you only need one page. (406 words)
- [Click-through rate](/glossary/click-through-rate/index.html): Click-through rate (CTR) is the percentage of users who click on a search result after seeing it, measuring how compelling your listing is. (301 words)
- [Desktop Reference - Settings and Export Formats](/docs/desktop/reference/index.html): Settings, page data model, crawl events, and export formats (481 words)
- [Crawler Desktop Overview](/docs/desktop/index.html): Desktop app for crawling websites with an interactive dashboard (511 words)
- [Canonical URL](/glossary/canonical-url/index.html): A canonical URL is an HTML element that tells search engines which version of a page is the preferred one for indexing. (273 words)
- [Duplicate descriptions](/glossary/duplicate-descriptions/index.html): Pages sharing the same meta description compete for clicks in search results (304 words)
- [Crawl budget](/glossary/crawl-budget/index.html): Crawl budget is the number of pages a search engine will crawl on your site within a given timeframe. (278 words)
- [Installation Guide for CLI & Desktop App](/docs/installation/index.html): How to install and uninstall the crawler.sh CLI tool and desktop app (233 words)
- [Frequently Asked Questions](/faq/index.html): Find answers to common questions about crawler.sh installation, configuration, output formats, concurrency settings, and troubleshooting. (115 words)
- [Overview - Getting Started with crawler.sh](/docs/index.html): A fast, concurrent web crawler with CLI and desktop interfaces (392 words)
- [What’s New in v0.6.2](/changelog/v062-orphan-page-detection/index.html): New SEO check detects orphan pages with zero incoming internal links. The 24th automated check, available in CLI and desktop. (479 words)
- [The Problem](/blog/how-to-force-google-to-update-your-favicon/index.html): Changed your favicon but Google still shows the old one? Here is a simple trick using the Google Favicon API to force a refresh. (610 words)
- [What’s New in v0.6.7](/changelog/v067-client-side-redirect-detection/index.html): Crawler now detects client-side redirects including meta refresh tags and Next.js NEXT_REDIRECT patterns, capturing them as part of the redirect chain. (295 words)
- [What’s New in v0.2.2](/changelog/v022-robots-txt-checker/index.html): Crawler now checks for robots.txt during every crawl and reports whether the site has one, including it in SEO audit results. (383 words)
- [What’s New in v0.6.0](/changelog/v060-broken-link-checking/index.html): Detect broken outgoing links and images across your site. The 23rd automated SEO check, available in CLI and desktop. (439 words)
- [What’s New in v0.6.6](/changelog/v066-seo-redirect-fix/index.html): Redirected pages are no longer flagged as SEO issues. Pages with redirect chains are excluded from SEO analysis in both CLI and desktop. (275 words)
- [What’s New in v0.5.0](/changelog/v050-keyword-extraction/index.html): Automatic keyword extraction from crawled pages using RAKE, with XSS/injection sanitization for safer content handling. (374 words)
- [What’s New in v0.6.3](/changelog/v063-orphan-detection-fix/index.html): Fixed false positives in orphan page detection caused by URL normalization mismatch. (206 words)
- [Crawler Cloud - Hosted Crawling API](/docs/cloud/index.html): Cloud-based crawling API with scheduled crawls, webhooks, and a web dashboard. Coming soon. (142 words)
- [What’s New in v0.2.4](/changelog/v024-link-discovery/index.html): Fixed an issue where "discovered from" link tracking was not recording the source page correctly, improving redirect audits and crawl analysis. (351 words)
- [What’s New in v0.2.0](/changelog/v020-desktop-app/index.html): Crawler is now available as a desktop app for macOS. Everything you can do in the CLI - crawl, analyze, export - now with a visual interface. (267 words)
- [What’s New in v0.4.0](/changelog/v040-better-content-extraction/index.html): Significantly improved content extraction accuracy with richer article metadata including site name, lead image, and publish dates. (284 words)
- [What’s New in v0.6.1](/changelog/v061-broken-link-header-fix/index.html): Fixed broken link checking to properly handle HTTP header responses, plus improved Content card details in the desktop app. (220 words)
- [What’s New in v0.1.6](/changelog/v016-crawl-history/index.html): Browse past crawls by site in the desktop app, download Crawler from a dedicated page, and explore new documentation. (329 words)
- [Talk to Crawler.sh](/contact/index.html): Get in touch with the crawler.sh team. Report bugs, request features, or ask questions about the web crawler CLI and desktop app. (114 words)
- [What’s New in v0.5.1](/changelog/v051-h1-seo-checks/index.html): 6 new H1-related SEO checks bring the total to 22 automated checks across CLI and desktop. (321 words)
- [Blog Posts & Articles](/blog/index.html): Read the latest articles about web crawling, SEO analysis, content extraction, and crawler.sh product updates and tutorials. (259 words)
- [What’s New in v0.3.0](/changelog/v030-retry-backoff/index.html): Crawler now automatically retries failed requests with exponential backoff and uses browser-like TLS fingerprints for better compatibility with protected sites. (248 words)
- [What’s New in v0.2.5](/changelog/v025-content-fixes/index.html): Improved content extraction accuracy, fixed edge cases in markdown conversion, and added an SEO glossary to the documentation. (209 words)
- [v0.7.0: JavaScript Rendering](/console/index.html): Manage your crawler.sh account, view your plan, and update your profile. (17 words)
- [What’s New in v0.1.8](/changelog/v018-subscriptions/index.html): Introducing Pro subscriptions with 10,000 page crawl limits and Content Archive - download all extracted page content as a Markdown ZIP. (264 words)
- [What’s New in v0.2.3](/changelog/v023-custom-user-agent/index.html): You can now set a custom User-Agent header for your crawls. Override the default in CLI with a flag or in the desktop app via Settings. (251 words)
- [Introducing Crawler](/changelog/v010-initial-release/index.html): The first public release of Crawler - a fast, local web crawler with CLI interface, SEO analysis, and multiple export formats. (269 words)
- [What’s New in v0.1.7](/changelog/v017-llms-txt-seo/index.html): Crawler now detects and downloads llms.txt files, validates their format, and lets you export SEO audit results as CSV. (258 words)
- [Every website deserves a thorough audit](/about/index.html): Learn about crawler.sh, the web crawling tool built in Rust. Meet the team and discover our mission to make web crawling fast and simple. (236 words)

## Resources

- [Full Page Index](/index.html): Browse all cached pages with rich metadata
- [About This Cache](/content/about.html): Methodology, technical details, and usage guidelines
- [XML Sitemap](/content/sitemap.xml): Machine-readable sitemap for crawler discovery
- [Robots.txt](/content/robots.txt): Crawler directives