v0.7.0: JavaScript Rendering

March 2, 2026

## What’s New in v0.2.2

### Robots.txt Detection

Every crawl now automatically checks whether the target site has a `robots.txt` file. The result appears in the site info summary and as a site-level SEO check.

If no `robots.txt` is found, the SEO audit flags it as an issue. Search engines rely on this file to understand which pages should and shouldn’t be indexed. A missing `robots.txt` does not block crawling, but it signals to search engines that the site owner has not configured crawl directives - and it means you are missing an opportunity to point crawlers toward your sitemap.

### Why Robots.txt Matters

The `robots.txt` file sits at the root of every website and serves as the first point of communication between a site and search engine crawlers. It tells bots which directories or pages to skip, which crawl rate to use, and where to find the XML sitemap. Sites without one may see search engines wasting crawl budget on low-value pages like admin panels, staging URLs, or duplicate filtered views. Adding a well-structured `robots.txt` is one of the simplest and most impactful technical SEO improvements you can make.

### How It Works

At the start of each crawl, Crawler fetches `<site>/robots.txt` and reports:

- Whether the file exists  
- Whether a sitemap is referenced in it  
- The number of sitemap URLs discovered

This information shows up in:

- The **SiteInfo** event in CLI output  
- The **SEO Issues** card in the desktop app  
- CSV and TXT exports from the `seo` command

If the `robots.txt` references a sitemap, Crawler parses it and adds any discovered URLs to the crawl queue. This means you get broader coverage even on sites with deep or orphaned pages that might not be reachable through internal links alone.

## Wrap-up

A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.