Spiders

Create, list, update, and delete your web spiders.

GET /api/v1/spiders

List all spiders for the authenticated user. Supports filtering by name and pagination via limit.

Parameters

Name	Type	Description
name	string	Filter spiders by name (case-insensitive partial match)
limit	number	Max results to return (default: 200)

Example Request

cURL

"text">-green-400">curl https://scrapethis.app/api/v1/spiders?limit=10 \
  "text-cyan-400">-H "Authorization: Bearer YOUR_API_KEY"

Response

200 OK

[
  {
    "id": "uuid-here",
    "name": "my-spider",
    "repo_path": "spiders/my_spider.py",
    "description": "Scrapes product listings",
    "settings": {},
    "repo_id": "repo-uuid",
    "user_id": "user-uuid",
    "created_at": "2025-01-15T10: 30: 00Z"
  }
]

Error Responses

401 Invalid or missing API key

POST /api/v1/spiders

Create a new spider.

Request Body

TypeScript

{
  name: "text-purple-400">string;        // required
  repo_path: "text-purple-400">string;   // required — path to spider file in repo
  description?: "text-purple-400">string;
  settings?: object;
  repo_id?: "text-purple-400">string;
}

Example Request

cURL

"text">-green-400">curl "text-cyan-400">-X POST https://scrapethis.app/api/v1/spiders \
  "text-cyan-400">-H "Authorization: Bearer YOUR_API_KEY" \
  "text-cyan-400">-H "Content">-Type: application/json" \
  "text-cyan-400">-d '{
    "name": "product">-spider",
    "repo_path": "spiders/products.py",
    "description": "Scrapes product data"
  }'

Response

200 OK

{
  "id": "new-uuid",
  "name": "product-spider",
  "repo_path": "spiders/products.py",
  "description": "Scrapes product data",
  "settings": {},
  "repo_id": null,
  "user_id": "user-uuid",
  "created_at": "2025-01-15T10: 30: 00Z"
}

Error Responses

400 Missing name or repo_path

401 Invalid or missing API key

GET /api/v1/spiders/{id}

Get a single spider by ID.

Example Request

cURL

"text">-green-400">curl https://scrapethis.app/api/v1/spiders/SPIDER_ID \
  "text-cyan-400">-H "Authorization: Bearer YOUR_API_KEY"

Response

200 OK

{
  "id": "spider-uuid",
  "name": "my-spider",
  "repo_path": "spiders/my_spider.py",
  "description": "Scrapes product listings",
  "settings": {},
  "repo_id": "repo-uuid",
  "user_id": "user-uuid",
  "created_at": "2025-01-15T10: 30: 00Z"
}

Error Responses

404 Spider not found

PATCH /api/v1/spiders/{id}

Update a spider. Send only the fields you want to change.

Request Body

TypeScript

{
  name?: "text-purple-400">string;
  repo_path?: "text-purple-400">string;
  description?: "text-purple-400">string;
  settings?: object;
}

Example Request

cURL

"text">-green-400">curl "text-cyan-400">-X PATCH https://scrapethis.app/api/v1/spiders/SPIDER_ID \
  "text-cyan-400">-H "Authorization: Bearer YOUR_API_KEY" \
  "text-cyan-400">-H "Content">-Type: application/json" \
  "text-cyan-400">-d '{"description": "Updated description"}'

Response

200 OK

{
  "id": "spider-uuid",
  "name": "my-spider",
  "repo_path": "spiders/my_spider.py",
  "description": "Updated description",
  "settings": {},
  "repo_id": "repo-uuid",
  "user_id": "user-uuid",
  "created_at": "2025-01-15T10: 30: 00Z"
}

Error Responses

404 Spider not found

DELETE /api/v1/spiders/{id}

Delete a spider by ID.

Example Request

cURL

"text">-green-400">curl "text-cyan-400">-X DELETE https://scrapethis.app/api/v1/spiders/SPIDER_ID \
  "text-cyan-400">-H "Authorization: Bearer YOUR_API_KEY"

Response

200 OK

{
  "ok": true
}

Error Responses

404 Spider not found