scrapy.utils.python

module documentation
(source)

This module contains essential stuff that should've come with Python itself ;)

Class MutableAsyncChain Similar to MutableChain but for async iterables
Class MutableChain Thin wrapper around itertools.chain, allowing to add iterables "in-place"
Function binary_is_text Returns ``True`` if the given ``data`` argument (a ``bytes`` object) does not contain unprintable control characters.
Function equal_attributes Compare two objects attributes
Function flatten flatten(sequence) -> list
Function garbage_collect Undocumented
Function get_func_args Return the argument name list of a callable
Function get_spec Returns (args, kwargs) tuple for a function >>> import re >>> get_spec(re.match) (['pattern', 'string'], {'flags': 0})
Function global_object_name Return full name of a global object.
Function iflatten iflatten(sequence) -> iterator
Function is_listlike >>> is_listlike("foo") False >>> is_listlike(5) False >>> is_listlike(b"foo") False >>> is_listlike([b"foo"]) True >>> is_listlike((b"foo",)) True >>> is_listlike({}) True >>> is_listlike(set()) True >>> is_listlike((x for x in range(3))) True >>> is_listlike(range(5)) True...
Function memoizemethod_noargs Decorator to cache the result of a method (without arguments) using a weak reference to its object
Function re_rsearch This function does a reverse search in a text using a regular expression given in the attribute 'pattern'. Since the re module does not provide this functionality, we have to find for the expression into chunks of text extracted from the end (for the sake of efficiency)...
Function to_bytes Return the binary representation of ``text``. If ``text`` is already a bytes object, return it as-is.
Function to_unicode Return the unicode representation of a bytes object ``text``. If ``text`` is already an unicode object, return it as-is.
Function unique efficient function to uniquify a list preserving item order
Function without_none_values Return a copy of ``iterable`` with all ``None`` entries removed.
Async Function _async_chain Undocumented
Constant _BINARYCHARS Undocumented
def binary_is_text(data): (source)

Returns ``True`` if the given ``data`` argument (a ``bytes`` object) does not contain unprintable control characters.

def equal_attributes(obj1, obj2, attributes): (source)

Compare two objects attributes

def flatten(x): (source)

flatten(sequence) -> list Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables). Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten(["foo", "bar"]) ['foo', 'bar'] >>> flatten(["foo", ["baz", 42], "bar"]) ['foo', 'baz', 42, 'bar']

def garbage_collect(): (source)

Undocumented

def get_func_args(func, stripself=False): (source)

Return the argument name list of a callable

def get_spec(func): (source)

Returns (args, kwargs) tuple for a function >>> import re >>> get_spec(re.match) (['pattern', 'string'], {'flags': 0}) >>> class Test: ... def __call__(self, val): ... pass ... def method(self, val, flags=0): ... pass >>> get_spec(Test) (['self', 'val'], {}) >>> get_spec(Test.method) (['self', 'val'], {'flags': 0}) >>> get_spec(Test().method) (['self', 'val'], {'flags': 0})

def global_object_name(obj): (source)

Return full name of a global object. >>> from scrapy import Request >>> global_object_name(Request) 'scrapy.http.request.Request'

def iflatten(x): (source)

iflatten(sequence) -> iterator Similar to ``.flatten()``, but returns iterator instead

def is_listlike(x): (source)

>>> is_listlike("foo") False >>> is_listlike(5) False >>> is_listlike(b"foo") False >>> is_listlike([b"foo"]) True >>> is_listlike((b"foo",)) True >>> is_listlike({}) True >>> is_listlike(set()) True >>> is_listlike((x for x in range(3))) True >>> is_listlike(range(5)) True

def memoizemethod_noargs(method): (source)

Decorator to cache the result of a method (without arguments) using a weak reference to its object

def re_rsearch(pattern, text, chunk_size=1024): (source)

This function does a reverse search in a text using a regular expression given in the attribute 'pattern'. Since the re module does not provide this functionality, we have to find for the expression into chunks of text extracted from the end (for the sake of efficiency). At first, a chunk of 'chunk_size' kilobytes is extracted from the end, and searched for the pattern. If the pattern is not found, another chunk is extracted, and another search is performed. This process continues until a match is found, or until the whole file is read. In case the pattern wasn't found, None is returned, otherwise it returns a tuple containing the start position of the match, and the ending (regarding the entire text).

def to_bytes(text, encoding=None, errors='strict'): (source)

Return the binary representation of ``text``. If ``text`` is already a bytes object, return it as-is.

def to_unicode(text, encoding=None, errors='strict'): (source)

Return the unicode representation of a bytes object ``text``. If ``text`` is already an unicode object, return it as-is.

def unique(list_, key=(lambda x: x)): (source)

efficient function to uniquify a list preserving item order

def without_none_values(iterable): (source)

Return a copy of ``iterable`` with all ``None`` entries removed. If ``iterable`` is a mapping, return a dictionary where all pairs that have value ``None`` have been removed.

async def _async_chain(*iterables: Union[Iterable, AsyncIterable]) -> AsyncGenerator: (source)

Undocumented

_BINARYCHARS = (source)

Undocumented

Value
{to_bytes(chr(i)) for i in range(32)}-set([b'\x00', b'\t', b'\n', b'\r'])