webcrawler
New. MCP support for Claude, Cursor & VS Code

Crawl any website.
Audit every page.

Paste a URL. Get every page on the domain with its title, status code, response time, and indexability. Free, no signup.

Used by SEO folks, developers, and content teams.

How it works

Three steps to a full site audit

Paste a URL. Watch it scan. Get the report. Nothing to install.

01

Paste any URL

Enter a URL and hit start. The crawler does a breadth-first walk of the whole root domain, with no depth limit.

02

Watch live progress

Watch the crawl in real time. You get a bookmark link, so you can close the tab and come back later.

03

Get your report

Read the results in the browser or export the whole crawl as a CSV. You can also get an email when it finishes.

Features

What you get

One free crawler. No signup, no page limits, nothing to install.

Full domain discovery

Breadth-first crawl scoped to the root domain. Every internal page gets found, with no depth limit.

SEO data per page

Page titles, meta descriptions, H1 tags, and word counts, in one view.

HTTP status codes

Spot 404s, 301 redirects, 500s, and any other status code across the site.

Response time tracking

See how fast each page responds. Find slow pages that hurt user experience and search rankings.

Indexability detection

Detect noindex tags, canonical URLs, and robots directives so you know exactly what search engines see.

CSV export via email

Download your full crawl report as a spreadsheet. The CSV file is generated and delivered to your inbox.

Bookmark & resume

Every crawl gets a unique tracking link. Close your browser, come back later, your results are waiting.

robots.txt compliant

The crawler respects robots.txt rules. Pages disallowed by the site owner are never crawled.

No signup required

No account, no credit card, no trial. Paste a URL and go.

Who it's for

Built for anyone who audits websites

SEO professionals

Run a full site audit in minutes. Find broken pages, missing titles, duplicate meta descriptions, and indexation problems from one crawl.

Web developers

Debug site migrations, verify redirects, and catch 500s before your users do. Test any staging or production URL.

Content managers

Find orphan pages, spot content that isn't indexed, and export a full URL inventory as CSV.

Agency owners

Run crawl reports for client sites. Share tracking links, export CSVs, and deliver findings without a paid tool.

FAQ

Frequently asked questions

Is this web crawler really free?
Yes. No signup, no credit card, no hidden fees. Paste a URL and start.
How many pages can the crawler discover?
There is no hard page limit. The crawler uses a breadth-first approach scoped to the root domain and will continue discovering pages until the entire site is mapped.
Do I need to create an account?
No. You can start a crawl immediately without signing up. You only provide an email if you want to be notified when a large crawl finishes.
How long does a crawl take?
Small sites (under 100 pages) usually finish in under a minute. Larger sites may take several minutes. You get a tracking link so you can close the tab and come back later.
What data does the crawler collect for each page?
URL, page title, meta description, HTTP status code, response time, content type, word count, and indexability status (noindex, canonical).
Can I export the results?
Yes. After a crawl completes you can export the full report as a CSV spreadsheet. The file is generated and delivered to your email.
Does the crawler respect robots.txt?
Yes. WebCrawler.buzz respects robots.txt directives and will not crawl pages that are disallowed by the site owner.
Can I crawl password-protected or private sites?
No. The crawler only accesses publicly available pages. It cannot log in or bypass authentication.

Ready to crawl a website?

Start your free crawl now. No signup, no page limits, no credit card.

Crawl a website free