v0.7.0: JavaScript Rendering

March 3, 2026

## What’s New in v0.2.3

Some websites block or rate-limit requests from non-browser User-Agents, returning 403 or 429 responses. Previously, Crawler always identified itself as `crawler.sh/0.1` with no way to change it.

You can now set a custom User-Agent string for any crawl. This is useful when:

- A site blocks the default crawler User-Agent
- You want to test how your site responds to specific bots
- You need to identify your crawl traffic in server logs with a custom string

When no custom value is set, crawls continue to use the default `crawler.sh/0.1` User-Agent.

## CLI Usage

Pass the `--user-agent` flag to the `crawl` command:

```
crawler crawl https://example.com --user-agent "Mozilla/5.0 (compatible; MyCrawler/1.0)"
```

You can combine it with any other flags as usual:

```
crawler crawl https://example.com \
  --user-agent "Mozilla/5.0" \
  --max-pages 500 \
  --concurrency 10
```

## Desktop App

Open the **Settings** card and enter your custom User-Agent in the **User-Agent** text field. Leave it empty to use the default. The setting persists across sessions.

## Wrap-up

A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.

If that sounds like the kind of tooling you want to use — [try Crawler](/content/download/index.html).

Crawler runs locally on your machine. Use the CLI or the desktop app — your workflow, your terms.