CLI Features - Commands and Crawl Engine

Crawl Engine

The crawler uses BFS (breadth-first search) traversal starting from the given URL. Pages at depth 0 are crawled first, then depth 1, and so on up to max_depth. Concurrency is controlled by the configured limit - requests never exceed the set number of concurrent connections.

Domain Constraint

Only links on the same host as the start URL are followed. Cross-domain links are discovered but not crawled. This keeps the crawl focused on a single site.

URL Normalization

Before enqueuing, URLs are normalized:

• Fragments (#section) are stripped
• Trailing slashes are removed
• Previously seen URLs are skipped

Link Filtering

The following link types are ignored:

• javascript: URLs
• mailto: links
• tel: links
• Anchor-only links (#)

JavaScript Rendering

crawler.sh includes a built-in JavaScript rendering engine that executes inline scripts and produces the final DOM. Pages built with client-side frameworks are crawled with their full rendered content - no external browser required.

Three rendering modes:

In Auto mode, the engine profiles the first few pages looking for JavaScript framework markers, empty body shells, script-to-text ratios, and bot protection signatures. If it detects that pages need rendering, it enables the JS engine automatically for the rest of the crawl. Resource limits (CPU timeout, memory cap, output size) are enforced to prevent runaway scripts from affecting the crawl.

Subcommands

The CLI provides five subcommands:

crawl - Crawl a website

crawler crawl <URL> [OPTIONS]

Crawls a website starting from the given URL, staying within the same domain. Outputs crawled page data in the specified format. The https:// prefix is added automatically if omitted.

# Basic crawl

crawler crawl https://example.com

# Crawl with custom limits

crawler crawl https://example.com -p 500 -d 5 -c 10

# Output as JSON

crawler crawl https://example.com -f json -o site.json

# Generate a sitemap

crawler crawl https://example.com -f sitemap

# Fast crawl without content extraction

crawler crawl https://example.com --no-extract --delay 50

fetch - Fetch a single URL

crawler fetch <URL> [OPTIONS]

Fetches a single page without crawling the entire site. The output filename is derived from the URL path, so each page gets a unique, readable file.

# Fetch a single page

crawler fetch https://example.com/about/team

# Creates: example-com-about-team.crawl

# Fetch the homepage

crawler fetch https://example.com/

# Creates: example-com-index.crawl

# Output as JSON

crawler fetch --format json https://example.com/pricing

# Skip content extraction

crawler fetch --no-extract https://example.com/about

# Custom output path

crawler fetch -o page.crawl https://example.com/about

The output format is identical to crawler crawl, so fetched pages work with info, seo, and export commands.

info - Inspect a .crawl file

crawler info <FILE>

Analyzes a .crawl (NDJSON) file and displays summary statistics: domain, page count, file size, status code distribution, and response time stats.

crawler info example-com.crawl

export - Convert a .crawl file

crawler export <FILE> --format <fmt> [OPTIONS]

Converts a .crawl file to another format. When --output is omitted, the path is derived from the input filename (e.g., example-com.json or example-com-sitemap.xml).

# Export to JSON

crawler export example-com.crawl -f json

# Export as sitemap XML

crawler export example-com.crawl -f sitemap -o sitemap.xml

seo - Analyze SEO issues

crawler seo <FILE> [OPTIONS]

Analyzes a .crawl file for SEO issues across 23 check categories. Only 2xx HTML pages are analyzed.

# Display SEO analysis in terminal

crawler seo example-com.crawl

# Export issues as CSV

crawler seo example-com.crawl --export csv

# Export issues as TXT with custom path

crawler seo example-com.crawl --export txt -o report.txt

SEO Analysis

The SEO analyzer runs on successful HTML pages - pages with 2xx HTTP status codes and text/html content type. Non-HTML resources (images, CSS, JS) and error pages are excluded.

Duplicate detection is canonical-aware: pages pointing to the same <link rel="canonical"> URL are grouped together before checking for duplicate titles or descriptions. This prevents false positives across canonicalized page groups.

Check Categories

Export Formats

• CSV - Two columns: Issue Type and URL. One row per affected URL, with values double-quote escaped.
• TXT - Sections grouped by issue type. Each section shows the issue title with count, followed by indented URLs.

Content Extraction

When content extraction is enabled (the default), the crawler extracts the main article content from each HTML page and converts it to clean Markdown. The extraction process is similar to browser reader modes - it strips navigation, sidebars, footers, and scripts, keeping only the primary content.

The result includes:

• Markdown text - clean article content
• Word count - total word count
• Author byline - when available
• Excerpt - article summary

Disable extraction with --no-extract for faster crawling when you only need URLs and metadata.

Broken Link Checking

By default, crawler crawl checks every outgoing link and image src on crawled pages for broken URLs. Internal links are already verified by the crawl itself, so only external URLs are checked.

How it works

• Sends a HEAD request to each external URL (falls back to GET on 405)
• Results are cached per URL so duplicate links across pages are only checked once
• Runs concurrently alongside the crawl and does not significantly slow it down

What counts as broken

• Anchors (<a> links) - 404, 410, 5xx, or connection failures
• Images (<img> sources) - any 4xx or 5xx response, or connection failures

Disabling link checks

Use --no-check-outgoing to skip outgoing link verification for faster crawls:

crawler crawl --no-check-outgoing https://example.com

Results in SEO reports

Broken links appear in crawler seo reports (check #23) grouped by broken URL with the source pages listed.

The .crawl File Format

The default output is NDJSON (Newline-Delimited JSON) with a .crawl extension. Each line is a standalone JSON object.

The first line is a metadata header:

{

"_meta": true,

"url": "https://example.com",

"max_pages": 100,

"max_depth": 10,

"concurrency": 5

}

Subsequent lines are crawled page records:

{

"url": "https://example.com/",

"status_code": 200,

"content_type": "text/html",

"title": "Example",

"meta_description": "...",

"canonical_url": "https://example.com/",

"links_found": 42,

"depth": 0,

"response_time_ms": 120,

"word_count": 350

}

Page records include optional SEO-relevant fields when available: meta_robots, x_robots_tag, rel_next, rel_prev, word_count, markdown, byline, excerpt, outgoing_link_errors.

This format is streamable - pages are written as they are crawled, so the file is always in a valid state even if the crawl is interrupted.

Polite Crawling

The delay_ms parameter adds a delay between requests to avoid overloading the target server. The CLI defaults to 200ms.