Connect WebCrawler.buzz to Claude, Cursor, and VS Code with One URL (MCP)

Model Context Protocol (MCP) is quickly becoming the standard way for AI assistants to talk to external tools. WebCrawler.buzz now ships a public MCP server at https://webcrawler.buzz/mcp, no auth, no API key, no rate-limit gymnastics for legitimate use. Connect it once in your AI client of choice and your assistant gains seven crawling tools it can call on your behalf.

What Is MCP and Why Should You Care?

MCP (Model Context Protocol) is an open standard that lets AI clients like Claude, Cursor, VS Code Copilot, and Codex discover and call external tools in a typed, safe way. Instead of every product inventing its own plugin system, MCP gives one common protocol, so a single server URL works everywhere.

For you, the practical upside is simple: no browser tabs to juggle, no copying URLs back and forth, no downloading CSVs manually. Just tell your AI 'crawl example.com and summarize what you find' and it happens inline.

The Endpoint

There is exactly one thing to remember:

https://webcrawler.buzz/mcp

GET that URL in a browser and you get the docs page. POST to it and you speak MCP over Streamable HTTP. Public. Stateless. No key required.

Connect It in 30 Seconds, Claude Desktop

Open Claude Desktop, go to Settings, click Connectors, then Add custom connector. Name it 'webcrawler.buzz' and paste https://webcrawler.buzz/mcp as the URL. Save. Done. Open a new chat and you should see the webcrawler-buzz connector with seven tools ready to use.

Try a prompt like 'Use webcrawler.buzz to crawl https://example.com and tell me every unique page title you find.' Claude will call start_crawl, poll progress, fetch results, and summarize, all on its own.

Cursor Config

Add this to ~/.cursor/mcp.json:

{ "mcpServers": { "webcrawler-buzz": { "type": "http", "url": "https://webcrawler.buzz/mcp" } } }

Restart Cursor. The seven tools appear in the tool picker.

VS Code Copilot Chat Config

Add this to .vscode/mcp.json in your workspace:

{ "servers": { "webcrawler-buzz": { "type": "http", "url": "https://webcrawler.buzz/mcp" } } }

Reload the window. Copilot Chat picks it up automatically.

The Seven Tools You Get

start_crawl, kick off a crawl by URL, returns a job_id immediately.

decide_duplicate, when a domain was already crawled, choose to reuse the results or start fresh.

get_crawl_progress, poll status without blocking.

get_crawl_results, paginated page results with URL, title, meta description, HTTP status, response time, link counts, indexability.

set_notify_email, for long crawls, register the user's email so they get pinged when it finishes and can close the chat safely.

request_csv_export, generate a downloadable CSV of the results. Email is optional; the download URL is always returned.

get_csv_export, poll for the export and receive the download URL when ready.

How Long Crawls Are Handled Without Boring You

AI chats have short attention spans, an LLM will not sit and wait an hour for a big crawl to finish. So the MCP server tells the assistant to poll at most three times over about 30 seconds, and if the crawl is still running, offer you two graceful exits: either provide your email so we can notify you when it is done, or come back to the chat later with the job_id and pick up where you left off. Either way, you never have to sit and stare at a spinner.

For Very Large Crawls, Prefer the CSV Path

If a crawl finds tens of thousands of pages, iterating through them page-by-page in chat is neither efficient nor useful, no assistant can meaningfully summarize 50,000 URLs. For crawls that big, ask the assistant to request a CSV export. You will get a single download link, and you can open the file in a spreadsheet or feed it into another tool for deeper analysis.

Conclusion

MCP turns webcrawler.buzz from a website you have to visit into a capability your favorite AI assistant just has. Connect it once and forget it is there, the next time you ask 'what pages does this site have?' your assistant will just know. Free, public, no signup. Paste the URL and start crawling from wherever you already work.