v0.7.0: JavaScript Rendering

March 11, 2026

What’s New in v0.6.5

Single Page Fetch

The new crawler fetch command grabs a single URL without crawling the entire site. It is the fastest way to capture one page when you do not need a full site crawl.

crawler fetch https://example.com/about/team

This creates example-com-about-team.crawl - a predictable, path-based filename instead of the domain-only name that crawler crawl produces.

Smart Filenames

The fetch command generates filenames from the URL path so every page gets a unique, readable output file:

URL Output file
https://example.com/ example-com-index.crawl
https://example.com/about/team example-com-about-team.crawl
https://example.com/blog/my-post.html example-com-blog-my-post.crawl
https://example.com/a/b?q=1 example-com-a-b.crawl

Path segments are joined with hyphens. File extensions like .html and .php are stripped. Query parameters and fragments are ignored. Long paths are truncated at a clean hyphen boundary.

CLI Usage

Basic fetch:

crawler fetch https://example.com/pricing

Override the output filename:

crawler fetch -o pricing-page.crawl https://example.com/pricing

Fetch as JSON or sitemap format:

crawler fetch --format json https://example.com/pricing

Disable content extraction for a smaller, faster result:

crawler fetch --no-extract https://example.com/pricing

All Options

crawler fetch [OPTIONS] <URL>

Arguments:

<URL>  The URL to fetch

Options:

-o, --output <PATH>     Output file path

-f, --format <FORMAT>   Output format: ndjson, json, sitemap [default: ndjson]

--no-extract        Disable content extraction

--user-agent <UA>   Custom User-Agent header

-v, --verbose           Enable verbose logging

-q, --quiet             Suppress all output except errors

When to Use Fetch vs Crawl

Use crawler fetch when you need a single page - checking one URL, grabbing content from a specific article, or testing how the crawler handles a particular page. Use crawler crawl when you need to analyze an entire site.

The output format is identical. Files produced by fetch work with all existing commands - crawler info, crawler seo, and crawler export.

Who Benefits

  • Content teams can quickly capture individual pages for review without waiting for a full site crawl
  • Developers can test and debug single-page crawl behavior with a lightweight command
  • SEO professionals can spot-check specific pages and run targeted crawler seo analysis on them

Wrap-up

A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.

If that sounds like the kind of tooling you want to use — try Crawler.

Crawler runs locally on your machine. Use the CLI or the desktop app — your workflow, your terms.