scrapy.spidermiddlewares.referer.RefererMiddleware

class documentation

class RefererMiddleware: (source)

View In Hierarchy

Undocumented

Class Method from_crawler Undocumented
Method __init__ Undocumented
Method policy Determine Referrer-Policy to use from a parent Response (or URL), and a Request to be sent.
Method process_spider_output Undocumented
Async Method process_spider_output_async Undocumented
Method request_scheduled Undocumented
Instance Variable default_policy Undocumented
Method _set_referer Undocumented
@classmethod
def from_crawler(cls, crawler): (source)

Undocumented

def __init__(self, settings=None): (source)

Undocumented

def policy(self, resp_or_url, request): (source)

Determine Referrer-Policy to use from a parent Response (or URL), and a Request to be sent. - if a valid policy is set in Request meta, it is used. - if the policy is set in meta but is wrong (e.g. a typo error), the policy from settings is used - if the policy is not set in Request meta, but there is a Referrer-policy header in the parent response, it is used if valid - otherwise, the policy from settings is used.

def process_spider_output(self, response, result, spider): (source)

Undocumented

async def process_spider_output_async(self, response, result, spider): (source)

Undocumented

def request_scheduled(self, request, spider): (source)

Undocumented

default_policy = (source)

Undocumented

def _set_referer(self, r, response): (source)

Undocumented