Overview - Getting Started with crawler.sh
What is crawler.sh?
crawler.sh is a fast, concurrent web crawler. It provides a CLI tool, the desktop app, and an upcoming cloud API for crawling websites, extracting content, analyzing SEO issues, and exporting results in multiple formats.
Key capabilities:
- Concurrent crawling with configurable depth, page limits, and delay
- JavaScript rendering
- Content extraction
- SEO analysis
- Multiple output formats
- Real-time progress
Products
Crawler CLI
Command-line interface with four subcommands: crawl, info, export, and seo. Features progress bars, color-coded output, and auto-generated filenames.
Crawler Desktop
Desktop application with an interactive dashboard. Eight cards for live feed, SEO analysis, status charts, content browsing, and exports.
Crawler Cloud
Hosted crawling API with scheduled crawls, webhooks, and a web dashboard. Coming soon.
Quick Start
Install the CLI
curl -fsSL https://install.crawler.sh | sh
```
Downloads the correct binary for your platform and adds it to your PATH.
2. ### Run your first crawl
crawler crawl https://example.com
```
This crawls the site (up to 100 pages by default) and saves results to example-com.crawl.
Inspect the results
crawler info example-com.crawl
```
View summary statistics: page count, status code distribution, and response times.
4. ### Analyze SEO issues
crawler seo example-com.crawl
```
Run 23 automated SEO checks and see issues grouped by category.
Download the app
Download the macOS DMG (universal binary - works on both Apple Silicon and Intel).
Install
Open the .dmg file and drag Crawler to your Applications folder.
Start a crawl
Launch Crawler, enter a URL in the top input bar, configure settings, and click Start Crawl.
Explore results
Browse the dashboard cards: Live Feed, SEO Issues, Content, and Downloads.