class documentation
class RobotParser: (source)
Known subclasses: scrapy.robotstxt.ProtegoRobotParser
, scrapy.robotstxt.PythonRobotParser
, scrapy.robotstxt.ReppyRobotParser
, scrapy.robotstxt.RerpRobotParser
Undocumented
Class Method | from |
Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend. |
Method | allowed |
Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``. |
overridden in
scrapy.robotstxt.ProtegoRobotParser
, scrapy.robotstxt.PythonRobotParser
, scrapy.robotstxt.ReppyRobotParser
, scrapy.robotstxt.RerpRobotParser
Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend. :param crawler: crawler which made the request :type crawler: :class:`~scrapy.crawler.Crawler` instance :param robotstxt_body: content of a robots.txt_ file. :type robotstxt_body: bytes
overridden in
scrapy.robotstxt.ProtegoRobotParser
, scrapy.robotstxt.PythonRobotParser
, scrapy.robotstxt.ReppyRobotParser
, scrapy.robotstxt.RerpRobotParser
Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``. :param url: Absolute URL :type url: str :param user_agent: User agent :type user_agent: str