scrapy.spidermiddlewares.offsite.OffsiteMiddleware

class documentation

class OffsiteMiddleware: (source)

View In Hierarchy

Undocumented

Class Method from_crawler Undocumented
Method __init__ Undocumented
Method get_host_regex Override this method to implement a different offsite policy
Method process_spider_output Undocumented
Async Method process_spider_output_async Undocumented
Method should_follow Undocumented
Method spider_opened Undocumented
Instance Variable domains_seen Undocumented
Instance Variable host_regex Undocumented
Instance Variable stats Undocumented
Method _filter Undocumented
@classmethod
def from_crawler(cls, crawler): (source)

Undocumented

def __init__(self, stats): (source)

Undocumented

def get_host_regex(self, spider): (source)

Override this method to implement a different offsite policy

def process_spider_output(self, response, result, spider): (source)

Undocumented

async def process_spider_output_async(self, response, result, spider): (source)

Undocumented

def should_follow(self, request, spider): (source)

Undocumented

def spider_opened(self, spider): (source)

Undocumented

domains_seen = (source)

Undocumented

host_regex = (source)

Undocumented

stats = (source)

Undocumented

def _filter(self, request, spider) -> bool: (source)

Undocumented