Scrapy Update: Better Broad Crawl Performance

When crawling the web, there’s always a speed limit. A spider can’t fetch faster than the host willing to send the pages. Page serving takes some amount of resources — CPU, disk, network bandwidth, etc. These resources cost money. Unrestricted serving and extensive crawling are the worst combinations. Such a combination could bring applications to halt and deny service to users. Taking all this into account, limiting serving capacity is natural.

This article explains which Scrapy settings help you honor these limits and how to achieve better performance during broad crawls in the presence of these limits.

Problem statement

For every entity, there is a slot in a Downloader. The number of requests sent to each entity simultaneously is limited by CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP options. Which one to choose depends on the selected way to differentiate between entities. Such requests can be called active or running. Requests enqueued into Downloader and not sent to the host are called inactive. Every slot has a queue for inactive requests. After finishing an active request and before running one from an inactive queue, Scrapy waits for DOWNLOAD_DELAY seconds.

This approach requires tuning to keep a balance between performance and limits honoring. A less tuning-sensitive one is implemented in the AutoThrottle extension. Documentation provides a very good background and description of how it works.

Possible approaches

After presenting the concept of entities to a Scheduler, there are different strategies to select one for the next request.

Round-robin algorithm can be used for request scheduling:

  • store all entities in FIFO queue Q
  • when the next request should be scheduled pop one entity E1 (from the top)
  • issue a request to E1
  • push E1 back into Q (to the bottom)

This approach provides an equal flow of requests. For it to work well, every crawled entity should serve pages with the same speed. In the real world, it is not the case. Different hosts have different rendering times and network latencies are also different.

Let’s look at what’s happening in such a situation.

In the end, the target is to keep the Downloader’s queue of inactive requests as short as possible. Generic Computer Science algorithm doesn’t work well in real-life conditions. A more task-specific approach needs to be presented.

Instead of using a model, real information can be used. Downloader knows the length of every queue. Providing such knowledge to a Scheduler solves the problem.

Experimental results

Based on numbers it was decided to not preserve round-robin implementation inside Scrapy’s codebase, but you can still find it in the commit history.

Originally published at

Hi, we’re Zyte, the central point of entry for all your web data needs.