RoborankSitemap URL Extractor
Give us a sitemap URL or just a domain. We find the sitemap, follow any index files, and hand back the full list of URLs with their last modified dates.
URLs found
0
Unique
0
With lastmod
0
Duplicates
0
Child sitemaps
This was a sitemap index, so we followed each child and merged the results.
URLs
Free, no signup. We read up to 2,000 URLs across up to 25 child sitemaps per run.
A list of URLs is the start. The work is knowing which ones matter.
Roborank pulls your Search Console data next to your sitemap, finds the pages that are close to ranking or quietly decaying, and fixes them.
See how Roborank worksWhat you get a URL list for
A sitemap is the closest thing to an official list of what a site thinks its pages are. Pulling it out as a flat list is the first step in a lot of ordinary SEO work, and it is tedious to do by hand once a site has more than a few hundred pages.
- Auditing a site you just took over. The URL list tells you how the site is actually structured, which is often not what anybody says it is.
- Comparing the sitemap to what is indexed. Pages in the sitemap that Search Console has never indexed are the first place to look when traffic is lower than it should be.
- Planning a migration. You need the full before list to build the redirect map, and the sitemap is the cleanest source for it.
- Sizing a competitor. How many pages they have, how they are grouped, and what they have published recently, all from a file they publish deliberately.
- Feeding another tool. Most crawlers, rank trackers and indexing APIs take a plain list of URLs.
Sitemap index files
Sites past a few thousand pages usually split their sitemap. A sitemap index is a sitemap that lists other sitemaps, and WordPress, Yoast, Shopify and most other platforms produce one by default, often split by post type. This tool follows the index automatically and merges the children into a single list, so you do not have to open each one.
The limits are 50,000 URLs and 50MB uncompressed per file, which is where the splitting comes from. We read up to 25 child sitemaps and 2,000 URLs in one run, which covers most sites.
Problems the extracted list makes obvious
URLs listed twice
Usually the same page appearing in two child sitemaps. Harmless in small numbers, and a sign the sitemap is generated by two systems that do not know about each other.
http URLs in an https sitemap
Every one of those is a redirect the crawler has to follow. The sitemap should list final URLs only.
Missing lastmod dates
Google uses lastmod as a hint about what to recrawl, as long as it is accurate. No dates means no hint. Dates that are always today means the hint gets ignored.
More URLs than you expected
Usually tag archives, paginated series or filter combinations that were never meant to be in there. Those dilute what the sitemap is telling Google to look at.
Finding a sitemap when you only have the domain
Enter the domain and we check robots.txt first, since the Sitemap: directive is the authoritative answer, then fall back to the common paths: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and a couple of others. If the site has a sitemap at a non-standard path and does not declare it in robots.txt, paste the sitemap URL directly.
What a sitemap should and should not contain
Canonical, indexable URLs that return 200. Nothing else. Not redirects, not 404s, not pages carrying a noindex tag, not URLs canonicalised to somewhere else. A sitemap full of URLs Google then finds it cannot index is a credibility problem: the whole file starts being trusted less, including the parts that were right.
Frequently asked questions
How do I extract all URLs from a sitemap?
Paste the sitemap URL, or just the domain, into the box above. We fetch it, follow any sitemap index files, and give you the full list to filter, copy or download as CSV or plain text.
Does this work with sitemap index files?
Yes. When the file is an index, we fetch each child sitemap and merge the results into one list, up to 25 children and 2,000 URLs per run.
What if I only know the domain?
Enter the domain. We read robots.txt for a Sitemap directive first, then try the usual paths like /sitemap.xml and /wp-sitemap.xml. If the site keeps its sitemap somewhere unusual and does not declare it, paste the sitemap URL directly.
How many URLs can a sitemap contain?
The spec caps a single sitemap file at 50,000 URLs and 50MB uncompressed. Past that, sites split into several files and list them in a sitemap index.
Should every page be in the sitemap?
Every canonical, indexable page that returns 200. Leave out redirects, error pages, noindexed pages and anything canonicalised elsewhere. A sitemap full of URLs that cannot be indexed makes the whole file less trusted.
Does lastmod actually matter?
It does, as long as it is accurate. Google uses it to decide what to recrawl. A lastmod that updates on every deploy while the content stays the same trains Google to ignore it.
Can I extract URLs from a competitor sitemap?
Yes. A sitemap is a public file published deliberately for crawlers, so reading one is ordinary competitive research.
Is this sitemap extractor free?
Yes, with no signup. There is a daily limit per visitor so the tool stays available to everyone.