How to Find All Existing and Archived URLs on an internet site

How to Find All Existing and Archived URLs on an internet site

Blog Article

There are plenty of good reasons you could will need to uncover all the URLs on a website, but your correct goal will determine what you’re looking for. For example, you may want to:

Identify each individual indexed URL to research problems like cannibalization or index bloat
Accumulate existing and historic URLs Google has seen, especially for web page migrations
Uncover all 404 URLs to Get better from submit-migration glitches
In Every circumstance, a single Instrument gained’t Supply you with every thing you may need. Regretably, Google Look for Console isn’t exhaustive, and a “site:example.com” look for is proscribed and challenging to extract details from.

During this write-up, I’ll stroll you through some tools to make your URL listing and just before deduplicating the info utilizing a spreadsheet or Jupyter Notebook, determined by your site’s sizing.

Previous sitemaps and crawl exports
If you’re looking for URLs that disappeared through the Stay site just lately, there’s a chance an individual with your crew might have saved a sitemap file or possibly a crawl export before the improvements were being made. In the event you haven’t previously, look for these documents; they could typically give what you may need. But, in case you’re looking through this, you probably didn't get so Fortunate.

Archive.org
Archive.org
Archive.org is an invaluable Resource for SEO jobs, funded by donations. Should you hunt for a site and select the “URLs” solution, you'll be able to obtain around ten,000 listed URLs.

Even so, Here are a few limitations:

URL limit: You may only retrieve approximately web designer kuala lumpur ten,000 URLs, which happens to be inadequate for bigger web-sites.
Good quality: Numerous URLs may very well be malformed or reference resource information (e.g., visuals or scripts).
No export choice: There isn’t a created-in solution to export the record.
To bypass The shortage of an export button, utilize a browser scraping plugin like Dataminer.io. However, these limits signify Archive.org may not present a complete Alternative for much larger web sites. Also, Archive.org doesn’t suggest irrespective of whether Google indexed a URL—but if Archive.org identified it, there’s a fantastic chance Google did, as well.

Moz Pro
While you could normally utilize a backlink index to search out external web-sites linking to you, these tools also find URLs on your site in the procedure.


The way to utilize it:
Export your inbound one-way links in Moz Professional to secure a brief and straightforward listing of focus on URLs out of your web site. In case you’re handling an enormous Web site, think about using the Moz API to export information over and above what’s manageable in Excel or Google Sheets.

It’s vital that you Take note that Moz Pro doesn’t affirm if URLs are indexed or identified by Google. However, considering the fact that most web sites use a similar robots.txt rules to Moz’s bots as they do to Google’s, this method typically will work perfectly as a proxy for Googlebot’s discoverability.

Google Search Console
Google Look for Console offers several valuable sources for creating your list of URLs.

Links experiences:


Just like Moz Pro, the Links part gives exportable lists of target URLs. Regretably, these exports are capped at 1,000 URLs Each individual. You may apply filters for precise pages, but considering the fact that filters don’t use on the export, you may ought to rely on browser scraping resources—restricted to five hundred filtered URLs at any given time. Not suitable.

Efficiency → Search Results:


This export provides you with a listing of web pages obtaining research impressions. Though the export is proscribed, You may use Google Look for Console API for much larger datasets. You will also find absolutely free Google Sheets plugins that simplify pulling a lot more considerable knowledge.

Indexing → Pages report:


This portion offers exports filtered by problem sort, even though these are generally also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is a superb source for amassing URLs, by using a generous Restrict of a hundred,000 URLs.


Better still, you are able to utilize filters to generate various URL lists, proficiently surpassing the 100k limit. One example is, in order to export only blog site URLs, adhere to these methods:

Move one: Add a segment on the report

Move two: Simply click “Make a new phase.”


Step three: Determine the phase which has a narrower URL sample, which include URLs that contains /site/


Notice: URLs found in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they provide beneficial insights.

Server log data files
Server or CDN log documents are Potentially the last word tool at your disposal. These logs capture an exhaustive listing of each URL route queried by customers, Googlebot, or other bots during the recorded time period.

Things to consider:

Details size: Log data files is usually significant, lots of web-sites only retain the last two months of data.
Complexity: Analyzing log documents may be challenging, but a variety of tools can be found to simplify the method.
Blend, and great luck
When you finally’ve collected URLs from all these resources, it’s time to combine them. If your site is sufficiently small, use Excel or, for larger sized datasets, instruments like Google Sheets or Jupyter Notebook. Assure all URLs are continually formatted, then deduplicate the list.

And voilà—you now have a comprehensive listing of existing, old, and archived URLs. Fantastic luck!