scrapy.core.spidermw.SpiderMiddlewareManager

class documentation

class SpiderMiddlewareManager(MiddlewareManager): (source)

View In Hierarchy

Undocumented

Method __init__ Undocumented
Method process_start_requests Undocumented
Method scrape_response Undocumented
Class Variable component_name Undocumented
Instance Variable downgrade_warning_done Undocumented
Class Method _get_mwlist_from_settings Undocumented
Static Method _get_async_method_pair Undocumented
Method _add_middleware Undocumented
Method _evaluate_iterable Undocumented
Async Method _process_callback_output Undocumented
Method _process_spider_exception Undocumented
Method _process_spider_input Undocumented
Method _process_spider_output Undocumented

Inherited from MiddlewareManager:

Class Method from_crawler Undocumented
Class Method from_settings Undocumented
Method close_spider Undocumented
Method open_spider Undocumented
Instance Variable methods Undocumented
Instance Variable middlewares Undocumented
Method _process_chain Undocumented
Method _process_parallel Undocumented
def __init__(self, *middlewares): (source)
overrides scrapy.middleware.MiddlewareManager.__init__

Undocumented

def process_start_requests(self, start_requests, spider: Spider) -> Deferred: (source)

Undocumented

def scrape_response(self, scrape_func: ScrapeFunc, response: Response, request: Request, spider: Spider) -> Deferred: (source)

Undocumented

component_name: str = (source)
overrides scrapy.middleware.MiddlewareManager.component_name

Undocumented

downgrade_warning_done: bool = (source)

Undocumented

@classmethod
def _get_mwlist_from_settings(cls, settings): (source)
overrides scrapy.middleware.MiddlewareManager._get_mwlist_from_settings

Undocumented

@staticmethod
def _get_async_method_pair(mw: Any, methodname: str) -> Union[None, Callable, Tuple[Callable, Callable]]: (source)

Undocumented

def _add_middleware(self, mw): (source)
overrides scrapy.middleware.MiddlewareManager._add_middleware

Undocumented

def _evaluate_iterable(self, response: Response, spider: Spider, iterable: Union[Iterable, AsyncIterable], exception_processor_index: int, recover_to: Union[MutableChain, MutableAsyncChain]) -> Union[Generator, AsyncGenerator]: (source)

Undocumented

async def _process_callback_output(self, response: Response, spider: Spider, result: Union[Iterable, AsyncIterable]) -> Union[MutableChain, MutableAsyncChain]: (source)

Undocumented

def _process_spider_exception(self, response: Response, spider: Spider, _failure: Failure, start_index: int = 0) -> Union[Failure, MutableChain]: (source)

Undocumented

def _process_spider_input(self, scrape_func: ScrapeFunc, response: Response, request: Request, spider: Spider) -> Any: (source)

Undocumented

@inlineCallbacks
def _process_spider_output(self, response: Response, spider: Spider, result: Union[Iterable, AsyncIterable], start_index: int = 0) -> Deferred: (source)

Undocumented