scrapy.robotstxt.RerpRobotParser

class documentation

class RerpRobotParser(RobotParser): (source)

Undocumented

Class Method	`from_crawler`	Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend.
Method	`__init__`	Undocumented
Method	`allowed`	Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``.
Instance Variable	`rp`	Undocumented
Instance Variable	`spider`	Undocumented

@classmethod
def from_crawler(cls, crawler, robotstxt_body): (source) ¶

overrides scrapy.robotstxt.RobotParser.from_crawler

Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend. :param crawler: crawler which made the request :type crawler: :class:`~scrapy.crawler.Crawler` instance :param robotstxt_body: content of a robots.txt_ file. :type robotstxt_body: bytes

def __init__(self, robotstxt_body, spider): (source) ¶

Undocumented

def allowed(self, url, user_agent): (source) ¶

overrides scrapy.robotstxt.RobotParser.allowed

Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``. :param url: Absolute URL :type url: str :param user_agent: User agent :type user_agent: str

rp = (source) ¶

Undocumented

spider = (source) ¶

Undocumented