scrapy.utils.response

module documentation

(source)

This module provides some useful functions for working with scrapy.http.Response objects

Function	`get_base_url`	Return the base url of the given response, joined with the response url
Function	`get_meta_refresh`	Parse the http-equiv refresh parameter from the given response
Function	`open_in_browser`	Open the given response in a local web browser, populating the <base> tag for external links to work
Function	`response_httprepr`	Return raw HTTP representation (as bytes) of the given response. This is provided only for reference, since it's not the exact stream of bytes that was received (that's not exposed by Twisted).
Function	`response_status_message`	Return status code plus status text descriptive message
Variable	`_baseurl_cache`	Undocumented
Variable	`_metaref_cache`	Undocumented

def get_base_url(response: scrapy.http.response.text.TextResponse) -> str: (source) ¶

Return the base url of the given response, joined with the response url

def get_meta_refresh(response: scrapy.http.response.text.TextResponse, ignore_tags: Optional[Iterable[str]] = ('script', 'noscript')) -> Union[Tuple[None, None], Tuple[float, str]]: (source) ¶

Parse the http-equiv refresh parameter from the given response

def open_in_browser(response: Union[scrapy.http.response.html.HtmlResponse, scrapy.http.response.text.TextResponse], _openfunc: Callable[[str], Any] = webbrowser.open) -> Any: (source) ¶

Open the given response in a local web browser, populating the <base> tag for external links to work

@deprecated
def response_httprepr(response: Response) -> bytes: (source) ¶

Return raw HTTP representation (as bytes) of the given response. This is provided only for reference, since it's not the exact stream of bytes that was received (that's not exposed by Twisted).

def response_status_message(status: Union[bytes, float, int, str]) -> str: (source) ¶

Return status code plus status text descriptive message

_baseurl_cache: WeakKeyDictionary[Response, str] = (source) ¶

Undocumented

_metaref_cache: WeakKeyDictionary[Response, Union[Tuple[None, None], Tuple[float, str]]] = (source) ¶

Undocumented