🤖 AI Integration

WebCrawler.buzz: MCP Server

Public Model Context Protocol server. No auth, no API key. Plug the URL below into any MCP-compatible client and your AI can crawl any website for you.

Endpoint

https://webcrawler.buzz/mcp

Transport: Streamable HTTP · Rate limit: 300 req / 15 min / IP

Install snippets

Claude Desktop / Claude.ai

Settings → Connectors → Add custom connector. Paste:

Name: webcrawler.buzz
URL:  https://webcrawler.buzz/mcp

Cursor

Add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "webcrawler-buzz": {
      "type": "http",
      "url": "https://webcrawler.buzz/mcp"
    }
  }
}

VS Code (GitHub Copilot Chat)

Add to .vscode/mcp.json in your workspace:

{
  "servers": {
    "webcrawler-buzz": {
      "type": "http",
      "url": "https://webcrawler.buzz/mcp"
    }
  }
}

Codex & other clients

Any client that supports remote Streamable-HTTP MCP servers works with the same URL.

Tools exposed

Tool	Purpose
`start_crawl`	Kick off a crawl of a URL. Returns `job_id` immediately.
`decide_duplicate`	Choose `re-crawl` or `use-existing` when a domain was previously crawled.
`get_crawl_progress`	Poll progress. Non-blocking.
`get_crawl_results`	Fetch paginated page results (URL, title, meta, status, etc.).
`set_notify_email`	Register email so the user gets a notification when a slow crawl finishes.
`request_csv_export`	Generate CSV of results. Email is optional, download URL is always returned.
`get_csv_export`	Poll export status; returns download URL when ready.

Try it in Claude

After connecting, try prompts like:

"Crawl https://example.com and give me a summary of every page."
"Export the crawl for job xxxxx as CSV and give me the download link."
"Notify me at [email protected] when the crawl for job xxxxx finishes."