scrapy.pqueues.DownloaderAwarePriorityQueue

class documentation

class DownloaderAwarePriorityQueue: (source)

View In Hierarchy

PriorityQueue which takes Downloader activity into account: domains (slots) with the least amount of active downloads are dequeued first.

Class Method from_crawler Undocumented
Method __contains__ Undocumented
Method __init__ Undocumented
Method __len__ Undocumented
Method close Undocumented
Method peek Returns the next object to be returned by :meth:`pop`, but without removing it from the queue.
Method pop Undocumented
Method pqfactory Undocumented
Method push Undocumented
Instance Variable crawler Undocumented
Instance Variable downstream_queue_cls Undocumented
Instance Variable key Undocumented
Instance Variable pqueues Undocumented
Instance Variable _downloader_interface Undocumented
@classmethod
def from_crawler(cls, crawler, downstream_queue_cls, key, startprios=()): (source)

Undocumented

def __contains__(self, slot): (source)

Undocumented

def __init__(self, crawler, downstream_queue_cls, key, slot_startprios=()): (source)

Undocumented

def __len__(self): (source)

Undocumented

def close(self): (source)

Undocumented

def peek(self): (source)

Returns the next object to be returned by :meth:`pop`, but without removing it from the queue. Raises :exc:`NotImplementedError` if the underlying queue class does not implement a ``peek`` method, which is optional for queues.

def pop(self): (source)

Undocumented

def pqfactory(self, slot, startprios=()): (source)

Undocumented

def push(self, request): (source)

Undocumented

crawler = (source)

Undocumented

downstream_queue_cls = (source)

Undocumented

key = (source)

Undocumented

pqueues: dict = (source)

Undocumented

_downloader_interface = (source)

Undocumented