scrapy.Request

class documentation

class Request(object_ref): (source)

Known subclasses: scrapy.FormRequest, scrapy.http.request.json_request.JsonRequest, scrapy.http.request.rpc.XmlRpcRequest

View In Hierarchy

Represents an HTTP request, which is usually generated in a Spider and executed by the Downloader, thus generating a :class:`Response`.

Class Method	`from_curl`	Create a Request object from a string containing a `cURL <https://curl.haxx.se/>`_ command. It populates the HTTP method, the URL, the headers, the cookies and the body. It accepts the same arguments as the :class:`Request` class, taking preference and overriding the values of the same arguments contained in the cURL command.
Method	`__init__`	Undocumented
Method	`__repr__`	Undocumented
Method	`copy`	Undocumented
Method	`replace`	Create a new Request with the same attributes except for those given new values
Method	`to_dict`	Return a dictionary containing the Request's data.
Class Variable	`attributes`	A tuple of :class:`str` objects containing the name of all public attributes of the class that are also keyword parameters of the ``__init__`` method.
Class Variable	`body`	Undocumented
Class Variable	`url`	Undocumented
Instance Variable	`callback`	Undocumented
Instance Variable	`cookies`	Undocumented
Instance Variable	`dont_filter`	Undocumented
Instance Variable	`errback`	Undocumented
Instance Variable	`flags`	Undocumented
Instance Variable	`headers`	Undocumented
Instance Variable	`method`	Undocumented
Instance Variable	`priority`	Undocumented
Property	`cb_kwargs`	Undocumented
Property	`encoding`	Undocumented
Property	`meta`	Undocumented
Method	`_get_body`	Undocumented
Method	`_get_url`	Undocumented
Method	`_set_body`	Undocumented
Method	`_set_url`	Undocumented
Instance Variable	`_body`	Undocumented
Instance Variable	`_cb_kwargs`	Undocumented
Instance Variable	`_encoding`	Undocumented
Instance Variable	`_meta`	Undocumented
Instance Variable	`_url`	Undocumented

Inherited from object_ref:

Method	`__new__`	Undocumented
Class Variable	`__slots__`	Undocumented

@classmethod
def from_curl(cls: Type[RequestTypeVar], curl_command: str, ignore_unknown_options: bool = True, **kwargs) -> RequestTypeVar: (source) ¶

Create a Request object from a string containing a `cURL <https://curl.haxx.se/>`_ command. It populates the HTTP method, the URL, the headers, the cookies and the body. It accepts the same arguments as the :class:`Request` class, taking preference and overriding the values of the same arguments contained in the cURL command. Unrecognized options are ignored by default. To raise an error when finding unknown options call this method by passing ``ignore_unknown_options=False``. .. caution:: Using :meth:`from_curl` from :class:`~scrapy.http.Request` subclasses, such as :class:`~scrapy.http.JSONRequest`, or :class:`~scrapy.http.XmlRpcRequest`, as well as having :ref:`downloader middlewares <topics-downloader-middleware>` and :ref:`spider middlewares <topics-spider-middleware>` enabled, such as :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`, :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`, or :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`, may modify the :class:`~scrapy.http.Request` object. To translate a cURL command into a Scrapy request, you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.

def __init__(self, url: str, callback: Optional[Callable] = None, method: str = 'GET', headers: Optional[dict] = None, body: Optional[Union[bytes, str]] = None, cookies: Optional[Union[dict, List[dict]]] = None, meta: Optional[dict] = None, encoding: str = 'utf-8', priority: int = 0, dont_filter: bool = False, errback: Optional[Callable] = None, flags: Optional[List[str]] = None, cb_kwargs: Optional[dict] = None): (source) ¶

overridden in scrapy.FormRequest, scrapy.http.request.json_request.JsonRequest, scrapy.http.request.rpc.XmlRpcRequest

Undocumented

def __repr__(self) -> str: (source) ¶

Undocumented

def copy(self) -> Request: (source) ¶

Undocumented

def replace(self, *args, **kwargs) -> Request: (source) ¶

overridden in scrapy.http.request.json_request.JsonRequest

Create a new Request with the same attributes except for those given new values

def to_dict(self, *, spider: Optional[scrapy.Spider] = None) -> dict: (source) ¶

Return a dictionary containing the Request's data. Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object. If a spider is given, this method will try to find out the name of the spider methods used as callback and errback and include them in the output dict, raising an exception if they cannot be found.

attributes: Tuple[str, ...] = (source) ¶

overridden in scrapy.http.request.json_request.JsonRequest

A tuple of :class:`str` objects containing the name of all public attributes of the class that are also keyword parameters of the ``__init__`` method. Currently used by :meth:`Request.replace`, :meth:`Request.to_dict` and :func:`~scrapy.utils.request.request_from_dict`.

body = (source) ¶

Undocumented

url = (source) ¶

Undocumented

callback = (source) ¶

Undocumented

cookies = (source) ¶

Undocumented

dont_filter = (source) ¶

Undocumented

errback = (source) ¶

Undocumented