scrapy.utils.sitemap

module documentation

(source)

Module for processing Sitemaps. Note: The main purpose of this module is to provide support for the SitemapSpider, its API is subject to change without notice.

Class	`Sitemap`	Class to parse Sitemap (type=urlset) and Sitemap Index (type=sitemapindex) files
Function	`sitemap_urls_from_robots`	Return an iterator over all sitemap urls contained in the given robots.txt file

def sitemap_urls_from_robots(robots_text, base_url=None): (source) ¶

Return an iterator over all sitemap urls contained in the given robots.txt file