Tools
Tools for crawling and site audits
A search engine can only rank a page it has found and stored. The tools here look at that from two sides. Some report what a search engine has seen on your site. Others crawl the site themselves, the way a search engine would, and list what they find, such as broken links, redirect chains, missing titles and pages blocked from the index.
This matters most for Crawling and Indexing, where the question is whether a page was found and kept. It also matters for Technical SEO audits, and for a Site migration, where a crawl before and after a move shows what changed.
The search engines' own consoles are free and are the natural first stop, because they show your site from the search engine's side rather than a guess at it.
Free tools
-
Google Search Console
Shows how Google Search sees a site. It has reports on indexing, sitemaps and rich results, a URL Inspection tool for single pages, and email alerts when Google finds issues. The site owner has to verify ownership first.
It reports on Google only, and only for sites you can verify as yours. It does not crawl a site on demand the way an audit tool does.
-
Bing Webmaster Tools
Microsoft's counterpart for Bing. Site owners verify a site, submit sitemaps and see clicks, impressions and average position on Bing. A site already verified in Google Search Console can be imported, which saves repeating verification. It also supports IndexNow, a way to tell participating search engines about new or changed URLs.
It reports on Bing only. Importing from Search Console proves ownership; it does not mean Bing will crawl or index the pages.
-
Screaming Frog SEO Spider (free version)
A desktop crawler for Windows, macOS and Linux. It finds broken links and server errors, checks redirects, page titles, meta descriptions and indexing directives, spots duplicate content and can generate XML sitemaps.
The free version caps how many URLs one crawl can cover and leaves out some advanced features and settings. A paid licence from the same maker lifts the cap.
Paid tools
-
Sitebulb
A website crawler for technical SEO audits, sold as a desktop application and as a cloud version that runs in the browser. It groups issues, explains each one and produces reports.
The maker offers a free trial with no card needed.
-
Semrush Site Audit
Part of the Semrush toolset. It lists site errors with tips for fixing each one, gives a site health score, ranks issues by severity and can send reports by email or export them.
Site Audit comes with a Semrush subscription, which has a short free trial. The maker does not describe a free plan for it.
-
SE Ranking website audit
A site audit inside the SE Ranking platform, which also covers rank tracking and keyword research. It checks crawlability, indexability, status codes and robots directives, groups the issues by priority and can re-run the audit on a schedule.
The maker offers a free trial with no card needed, and does not describe a free plan for the audit.
How to choose
Search Console comes first for almost any site, since it shows what Google reports about pages you own. Bing Webmaster Tools is worth adding if visitors reach you through Bing.
Add a crawler when you want to check a whole site yourself, for example before a migration or during a first audit. On a small site, the free version of Screaming Frog shows what a crawl report looks like. The paid crawlers and the audits inside larger toolsets suit bigger sites or regular reporting. Any of them will list problems; deciding which ones matter is still a human job.
Videos to watch first
Official source
How to read the Indexing Report
Sells in this field
Google Search Console Tutorial (2026): The Complete Guide
Facts checked 11 November 2026