Crawler.sh is a fast, local-first desktop and CLI tool for comprehensive website analysis. It crawls entire sites in seconds, extracts clean Markdown content, performs automated SEO audits with 16 checks, and exports data in multiple formats like JSON, Sitemap XML, and CSV.
Paid
$99/year
How to use Crawler.sh?
Download the desktop app or use the CLI. Enter a website URL to start a crawl. Configure concurrency and depth. The tool will rapidly scan the site, analyze each page for SEO issues, extract readable content as Markdown, and visualize HTTP status codes. Export the results for reporting, archiving, or feeding into other systems.
Crawler.sh 's Core Features
Blazing-fast website crawling with configurable concurrency, depth limits, and polite delays between requests.
Automatic extraction of main article content from any page, converted into clean, structured Markdown format with metadata.
Comprehensive SEO analysis performing 16 automated checks per page for issues like missing titles, duplicate meta descriptions, and thin content.
Flexible output formats including real-time NDJSON streaming, JSON arrays, W3C-compliant Sitemap XML, and human-readable CSV/TXT reports.
Local-first operation ensuring data privacy and speed, as all processing happens on your own machine without cloud dependency.
Visual dashboard in the Pro app with interactive cards and real-time feed for monitoring crawl progress and status.
Crawler.sh 's Use Cases
SEO specialists can audit client websites for technical issues, generate accurate sitemaps, and monitor for broken links before they impact search rankings.
Content managers and archivists can backup or migrate website content by extracting clean, readable Markdown from pages for use in other tools or systems.
Web developers can integrate crawl data into pipelines, analyze site structure, and export JSON/XML for automated processing and monitoring.
Digital agencies can run regular site health checks for multiple clients, exporting standardized SEO reports in CSV format for easy sharing and action.
Researchers and analysts can systematically collect data from websites for studies, converting page content into a consistent, analyzable Markdown format.
Crawler.sh 's Pricing
CLI Tool
$99 / year
Command-line interface with crawl, info, export, and SEO subcommands. Outputs NDJSON, JSON, Sitemap XML, Markdown, and SEO reports.
Pro Desktop App
$99 / year
Visual dashboard with interactive cards, real-time crawl feed, SEO issues panel, and content archive export. One subscription covers both CLI and desktop.
