class documentation
class PythonRobotParser(RobotParser): (source)
Undocumented
Class Method | from |
Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend. |
Method | __init__ |
Undocumented |
Method | allowed |
Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``. |
Instance Variable | rp |
Undocumented |
Instance Variable | spider |
Undocumented |
Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend. :param crawler: crawler which made the request :type crawler: :class:`~scrapy.crawler.Crawler` instance :param robotstxt_body: content of a robots.txt_ file. :type robotstxt_body: bytes
overrides
scrapy.robotstxt.RobotParser.allowed
Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``. :param url: Absolute URL :type url: str :param user_agent: User agent :type user_agent: str