Complete website crawl analysis
Follow eligible internal links and sitemap entries to build a complete inventory of important public pages, their status, depth, indexability, connections, and the site-wide patterns that deserve attention.
Website-wide evidence
Replace assumptions with a crawl inventory that shows what exists, what can be reached, how deeply it is buried, and where the website sends visitors and crawlers.
The report clearly shows the chosen page coverage limit and elapsed time.
AI visibility overview
Six-platform readiness and evidence
ChatGPT
Google
Gemini
Perplexity
Claude
GrokCombine eligible internal-link discovery with sitemap declarations to find pages hidden beyond the main navigation.
/website-crawl-audit
Current direction
Depth 1–2
Recommended direction
Discover every important page: Homepage and primary navigation
See successful pages, redirects, restricted or intentionally excluded URLs, non-HTML resources, and unavailable destinations clearly.
Whole-site crawl
Depth 0
Discover every important page
Depth 1
Understand every URL state
Depth 1
Homepage and primary navigation
Service and product pages
Location pages
Identify repeated metadata, templates, status issues, depth problems, and link behavior across groups of pages.
Side-by-side competitor evidence
Visibility, content and commercial coverage
How it works
Choose the public website you want Rankworth to explore and set an optional page coverage limit.
Eligible same-site HTML links and sitemap entries are followed while every response and exclusion is recorded.
Filter pages by status, depth, indexability, sitemap presence, template patterns, and the actions that matter most.
Transparent coverage
The report shows full crawl coverage, records restricted, intentionally excluded, malformed, oversized, unavailable, and non-HTML URLs separately, and states when the chosen page or time setting is reached.
Crawl coverage
Depth 0
Discovered
Depth 1
Crawled HTML
Depth 1
Page discovery
Response status
Page depth
Patterns across the site
A shared template can reveal opportunities across dozens of URLs, from title differentiation and direct links to canonical alignment and richer content. Rankworth groups each pattern into one actionable recommendation.
Side-by-side competitor evidence
Visibility, content and commercial coverage
Complete analysis
Understand discovery source, response status, redirect destination, page depth, indexability, sitemap coverage, and internal connections across the site.
Eligible internal links, sitemap entries, source pages, and the first path used to discover each URL.
Successful pages, redirects, access restrictions, server responses, unavailable URLs, and non-HTML resources.
How many internal steps separate a page from strong entry points and primary navigation paths.
Noindex directives, canonicals, robots blocks, and conflicting signals that shape index eligibility.
Alignment between declared pages, navigation paths, discovered URLs, and the website's sitemaps.
Repeated metadata, templates, duplicate content, broken links, and issues shared by groups of pages.
Exact recommendation
Every recommendation connects a real page and an observed condition to a specific change, expected benefit, effort, confidence, and practical verification step.
Example report action
Evidence and implementation in one view
What we found
Twenty-four location pages can connect directly to the current service URL through one template update.
Supporting evidence
Every page returns 200, and the shared link currently passes through two redirects before reaching the current service page.
Recommended change
Update the location-page template to link directly to the final service URL and confirm the same component is used across all affected pages.
How to verify it
Re-crawl the location group and confirm all 24 pages link directly to a 200-status destination in a single step.
Frequently asked questions
The crawler follows eligible same-site HTML links and, when enabled, valid sitemap entries. It records discovery sources, status codes, redirects, blocks, exclusions, and crawl depth for the URLs it encounters.
The analysis uses a configurable page coverage setting within the selected plan's allowance. Each report shows that setting, elapsed time, and final coverage so your team can interpret the crawl accurately.
Yes. The standard analysis follows the website's configured robots rules and records restricted URLs separately with a clear status in the report.
PDFs, images, archives, media, and other non-HTML resources may be discovered and classified separately, while page-level HTML analysis focuses on supported HTML documents.
Available report and export options depend on the selected plan. The on-screen report provides page-level results, filtering, statuses, and prioritized actions so teams can work through the crawl evidence directly.
Grow your Google & ChatGPT visibility
Enter your domain to crawl the complete website and receive a transparent inventory with page-by-page findings and priorities.
