https://webcrawler.buzz

REST API

Every button on this site is backed by a public endpoint. Start crawls, watch progress, page through results, and pull CSV exports, all programmatically. No API key required.

POST /api/crawl

Start a new crawl. Validates the URL, creates a job, pushes it to the queue, and returns a tracking URL immediately. Non-blocking.

Request body

{
  "url": "https://devhasibulislam.vercel.app"
}

Response, new crawl (201)

{
  "status": "started",
  "job_id": "9c8b4d20-1a5e-4f2c-b7a1-3e8d9c0f5a11",
  "tracking_url": "https://webcrawler.buzz/results?job_id=9c8b4d20-1a5e-4f2c-b7a1-3e8d9c0f5a11",
  "message": "Crawl started! Use tracking_url to follow progress."
}

Response, domain already crawled (200)

{
  "status": "exists",
  "job_id": "5f2a1c8d-9e3b-4a72-8f1c-2b6d4e0a9c73",
  "domain": "devhasibulislam.vercel.app",
  "total_pages": 87,
  "tracking_url": "https://webcrawler.buzz/results?job_id=5f2a1c8d-9e3b-4a72-8f1c-2b6d4e0a9c73",
  "prompt": "Send action 're-crawl' or 'use-existing' to POST /api/crawl/decide"
}

POST /api/crawl/decide

When a domain was crawled recently, decide whether to re-crawl or reuse the existing results.

Request body

{
  "domain": "devhasibulislam.vercel.app",
  "action": "re-crawl"
}

action is one of re-crawl or use-existing.

GET /api/crawl/:job_id/progress

Lightweight polling. Returns the current status and completion percentage.

Response

{
  "job_id": "9c8b4d20-1a5e-4f2c-b7a1-3e8d9c0f5a11",
  "status": "running",
  "total_pages_found": 34,
  "total_pages_queued": 128,
  "progress_percent": 27
}

GET /api/crawl/:job_id

Paginated crawl results. Every row is one discovered page with its SEO data.

Query parameters

Param	Type	Default	Description
`page`	integer	1	Page number, 1-based.
`limit`	integer	50	Results per page. One of 50, 100, 500, 1000.

POST /api/crawl/:job_id/notify

Request body

{
  "email": "[email protected]"
}

POST /api/crawl/:job_id/export

Kick off a CSV export of the full crawl. Email is optional; if you provide one, we send the download link when it's ready.

Request body

{
  "email": "[email protected]"
}

Returns an export_id you can poll on the endpoint below.

GET /api/crawl/:job_id/export/status

Poll for the CSV export status. When it's ready, the response includes a signed download_url valid for 5 minutes.

GET /api/health

System health, including Postgres and Redis connectivity. Returns 200 when everything is healthy, 503 when a dependency is down.

Response

{
  "status": "ok",
  "uptime": 42,
  "version": "2.3.8",
  "services": {
    "database": { "status": "ok", "latency_ms": 4 },
    "redis":    { "status": "ok" }
  }
}

GET /api/stats

Public usage statistics: completed jobs, active jobs, failed jobs, and a running page count. Handy for a status widget.

Rate limits

POST endpoints are limited to 200 requests per 15 minutes per IP. GET endpoints (progress polling, results, health, stats) are unlimited. If you hit the limit, you get a 429 with a Retry-After header.