webcrawler
Model Context Protocol

Crawl any site
from your coding agent.

WebCrawler.buzz ships an MCP server. Trigger crawls and read results from Claude Code, Codex, Cursor, GitHub Copilot, Claude Desktop, and any other MCP-compatible client.

Install About MCP
Claude Code Codex Cursor GitHub Copilot Claude Desktop VS Code Windsurf Zed
Setup

Point your client at the endpoint

The MCP server runs at https://webcrawler.buzz/mcp over streamable HTTP. No install, no API key. Add it once and every agent that speaks MCP can use it.

Claude Desktop / Claude Code

Add to claude_desktop_config.json (or ~/.claude/mcp.json for Claude Code):

{
  "mcpServers": {
    "webcrawler": {
      "url": "https://webcrawler.buzz/mcp"
    }
  }
}

Cursor / VS Code / Copilot

Drop into your workspace .cursor/mcp.json, .vscode/mcp.json, or GitHub Copilot's MCP settings:

{
  "servers": {
    "webcrawler": {
      "type": "http",
      "url": "https://webcrawler.buzz/mcp"
    }
  }
}

Codex / CLI agents

For OpenAI Codex, Continue.dev, and other CLI-based MCP clients, register the HTTP endpoint:

codex mcp add webcrawler \
  --url https://webcrawler.buzz/mcp

Windsurf / Zed / others

Any client that supports MCP over streamable HTTP will work. Point it at the same endpoint:

https://webcrawler.buzz/mcp
Tools

Seven tools your agent can call

A full lifecycle: start a crawl, watch progress, get results, register an email, and export a CSV.

start_crawl

Start crawling a URL. Non-blocking. Returns a job_id, root domain, and a tracking URL. If the domain was crawled before, returns already_crawled.

decide_duplicate

When a domain was crawled recently, choose use-existing to reuse the old results or re-crawl to start a fresh run.

get_crawl_progress

Poll a running crawl. Returns pages found, pages queued, status, and a completion percentage. Polling capped at 3 attempts.

get_crawl_results

Fetch the discovered pages with SEO data. Paginated: page, limit up to 500.

set_notify_email

Register the user's email so they get notified when a large crawl finishes. Agent must ask permission first. Rate-limited.

request_csv_export

Kick off a CSV export of the full crawl. Email is optional. Returns an export_id to poll.

get_csv_export

Poll for CSV status. When it's ready, returns a download_url your agent (or the user) can hit directly.

Ready to hook it up?

Add the endpoint to your client and ask your agent to crawl any site.

Try the web version