scrapy.downloadermiddlewares

package documentation
(source)

Undocumented

Module ajaxcrawl No module docstring; 0/2 variable, 1/1 function, 1/1 class documented
Module cookies No module docstring; 0/2 variable, 0/1 function, 1/1 class documented
Module decompression This module implements the DecompressionMiddleware which tries to recognise and extract the potentially compressed responses that may arrive.
Module defaultheaders DefaultHeaders downloader middleware
Module downloadtimeout Download timeout middleware
Module httpauth HTTP basic auth downloader middleware
Module httpcache Undocumented
Module httpcompression No module docstring; 0/1 constant, 1/1 class documented
Module httpproxy Undocumented
Module redirect No module docstring; 0/1 variable, 0/1 function, 1/3 class documented
Module retry An extension to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error.
Module robotstxt This is a middleware to respect robots.txt policies. To activate it you must enable this middleware and enable the ROBOTSTXT_OBEY setting.
Module stats Undocumented
Module useragent Set User-Agent header per spider or use a default value from settings