v0.7.0: JavaScript Rendering

February 26, 2026

## What’s New in v0.1.7

### llms.txt Detection & Download

Crawler now checks whether crawled sites have an `llms.txt` file - the emerging standard for telling LLMs what content to use from your site. During a crawl, Crawler will:

- Detect if `llms.txt` exists at the site root
- Report it in the site info summary
- Let you download it directly

This is useful for auditing whether your site is prepared for AI-powered search and content consumption.

### SEO Audit CSV Export

The `seo` command can now export results as CSV in addition to the existing TXT format. This makes it easy to import audit results into spreadsheets or other analysis tools.

```
crawler seo site.crawl --format csv -o audit.csv
```

### Copy SEO Audit to Clipboard

In the desktop app, you can now copy the full SEO audit to your clipboard with one click - useful for sharing results in tickets, docs, or messages.

### Other Improvements

- Sitemap and robots.txt parsing improvements
- CDN-specific URLs (like `cdn-cgi`) are now ignored during crawls

## Wrap-up

A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.

If that sounds like the kind of tooling you want to use — [try Crawler](/content/download/index.html).

Crawler runs locally on your machine. Use the CLI or the desktop app — your workflow, your terms.