Loading page
Loading page

Developer tools · Zyte Scrapy Cloud alternative
Manage, schedule and watch your web scrapers from one web UI, whatever language or framework they use.
Crawlab is a distributed web crawler management platform written in Go. You upload spiders written in almost any language, run them from a web UI, schedule them with cron, and read logs and results in one place.
A full Crawlab setup has a master node, worker nodes, MongoDB and a distributed file system. The master hands tasks to workers over gRPC. This template deploys the master node with MongoDB, which is enough to start.
It suits data engineers, researchers and growth teams who run many scrapers. It helps most when spiders are scattered across servers and nobody can see which ones failed last night.
Hosted scraping platforms charge by compute time or by request. Your own Crawlab runs as many spiders as your server can handle, and the code and results stay on your own server.
The trade-off is that you run the platform and its spiders. You change the default login, back up MongoDB, and add workers when one server is not enough. Scraping also comes with legal and site-policy questions, which are yours to check. The project's disclaimer covers its stance.
Containers: crawlabteam/crawlab:latest, mongo:4.2
Step 1
Choose the Starter plan. It costs $20 a month and runs up to 3 apps.
Step 2
Open App templates in the dashboard, choose Crawlab (Master), and click Deploy. We set up the containers, secrets and domain.
Step 3
Wait a few minutes for the first deploy, then open your new URL and finish the Crawlab (Master) setup screen.