scrapy.downloadermiddlewares.robotstxt

module documentation
(source)

This is a middleware to respect robots.txt policies. To activate it you must enable this middleware and enable the ROBOTSTXT_OBEY setting.

Class RobotsTxtMiddleware Undocumented
Variable logger Undocumented
logger = (source)

Undocumented