How to Crawl Your Website

Discover your entire website structure and extract valuable SEO metadata automatically with ViSitemap's powerful crawler.

Starting a New Crawl

To begin, navigate to the Crawl Dashboard and click on "New Crawl". You will be prompted to configure your crawl settings.

Base URL

The homepage of the website you want to crawl (e.g., https://example.com). The crawler will only stay within this domain.

Max Pages

The maximum number of pages to discover. Free plans are limited to 50 pages, while Pro plans allow up to 10,000.

Max Depth

How many clicks away from the homepage the crawler should go. Setting this to 3 means it will find pages up to 3 levels deep.

Configuring Filters

You can fine-tune what the crawler discovers by using Include and Exclude patterns.

Include Patterns

Only crawl URLs containing these strings. Great for focusing on specific sections like /blog or /products.

Exclude Patterns

Skip URLs containing these strings. Use this to ignore /admin areas, ?tags=, or other dynamic parameters.

Understanding Crawl Status

Running- The crawler is currently visiting pages and extracting data.

Paused- The crawl has been manually paused. You can resume it at any time.

Completed- All target pages have been discovered and analyzed.

Pro Tip: Headless State

Our crawler runs in the background using Inngest. You don't need to keep your browser tab open once the crawl starts. You can close the window and come back later to see the results or receive an email notification when it's finished.

Start Your First Crawl