Skip to main content
Internet Archive's 25th Anniversary Logo

Web Crawls

The Web Archive of the Internet Archive started in late 1996, is made available through the Wayback Machine, and some collections are available in bulk to researchers. Many pages are archived by the Internet Archive for other contributors including partners of Archive-IT, and Save Page Now users. Other captures are donated to the Internet Archive by other partners such as Alexa Internet.



rss RSS

18,119,915
RESULTS


Show sorted alphabetically

Show sorted alphabetically

SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Internet Archive Web Crawls
Internet Archive Web Crawls
collection
1,611,979
ITEMS
37.4B
VIEWS
collection

eye 37.4B

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine .
Topic: webwidecrawl
Worldwide Web Crawls
Worldwide Web Crawls
collection
627,441
ITEMS
15.5B
VIEWS
collection

eye 15.5B

Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
Alexa Crawls
Alexa Crawls
collection
226,900
ITEMS
14.1B
VIEWS
collection

eye 14.1B

Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.
Topics: web crawl, Alexa
Live Web Proxy Crawls
Live Web Proxy Crawls
collection
108,912
ITEMS
9.2B
VIEWS
collection

eye 9.2B

Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Survey Crawls
Survey Crawls
collection
100,903
ITEMS
9.4B
VIEWS
collection

eye 9.4B

Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
Save Page Now
Save Page Now
collection
114,914
ITEMS
1.6B
VIEWS
collection

eye 1.6B

Archive-It Digital Collection
Archive-It Digital Collection
collection
9,557,108
ITEMS
4.8B
VIEWS
collection

eye 4.8B

Archive-It is a subscription web archiving service of the Internet Archive that helps organizations harvest, build, and preserve collections of digital content. Partners create domain specific collections of web captures that can be searched on Archive It . Content is hosted and stored at the Internet Archive data centers. Archive-It works with more than 400 partner organizations in 48 U.S. states and 16 countries worldwide including: College and University Libraries State Archives, Libraries,...
Topic: Colleges, Universities, Libraries, Archives, NGOs, Museums
Archive-It Partners
Archive-It Partners
collection
9,551,355
ITEMS
4.7B
VIEWS
collection

eye 4.7B

Archive-It is the leading web archiving service for collecting and accessing cultural heritage on the web and is a service of Internet Archive used by libraries, archives, governments, non-profits, and other organizations to build collections of web materials.
Topic: TK
Archive Team
Archive Team
collection
2,962,842
ITEMS
3.4B
VIEWS
collection

eye 3.4B

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history. History is littered with hundreds of conflicts over the future of a community, group, location or...
Focused Crawls
Focused Crawls
collection
381,423
ITEMS
3.9B
VIEWS
by Internet Archive
collection

eye 3.9B

Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.
Topic: webcrawl
Fix Broken Links Web Crawls
Fix Broken Links Web Crawls
collection
129,480
ITEMS
3.2B
VIEWS
collection

eye 3.2B

These crawls are part of an effort to archive pages as they are created and archive the pages that they refer to. That way, as the pages that are referenced are changed or taken from the web, a link to the version that was live when the page was written will be preserved. Then the Internet Archive hopes that references to these archived pages will be put in place of a link that would be otherwise be broken, or a companion link to allow people to see what was originally intended by a page's...
Top Domains
collection
194,158
ITEMS
2.5B
VIEWS
collection

eye 2.5B

A daily collection of thousands of the most popular web sites according to Alexa.com's top sites rankings .
Topics: daily, popular sites, Alexa
Wiki Collections
Wiki Collections
collection
2,736,272
ITEMS
1.8B
VIEWS
collection

eye 1.8B

Collections of Wiki data
Topics: crawls, data, wiki
Wikipedia Outlinks
Wikipedia Outlinks
collection
85,661
ITEMS
1.8B
VIEWS
collection

eye 1.8B

Crawl of outlinks from wikipedia.org . These files are currently not publicly accessible. from Wikipedia : Wikipedia is a multilingual, web-based, free-content encyclopedia project operated by the Wikimedia Foundation and based on an openly editable model. The name "Wikipedia" is a portmanteau of the words wiki (a technology for creating collaborative websites, from the Hawaiian word wiki, meaning "quick") and encyclopedia. Wikipedia's articles provide links to guide the...
collection

eye 1.6B

Wide17 was seeded with the "Total Domains" list of 256,796,456 URLs provided by  Domains Index   on June 26th, and crawled with max-hops set to "3" and de-duplication set "on".   
ArchiveBot: The Archive Team Crowdsourced Crawler
ArchiveBot: The Archive Team Crowdsourced Crawler
collection
27,512
ITEMS
1.9B
VIEWS
collection

eye 1.9B

ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites). To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel...
Topics: archiveteam, archivebot, webcrawl, robot, love
perma_cc
perma_cc
collection
2,369,989
ITEMS
852.8M
VIEWS
collection

eye 852.8M

collection

eye 2.1B

The seed for Wide00014 was: - Slash pages from every domain on the web: -- a list of domains using Survey crawl seeds -- a list of domains using Wide00012 web graph -- a list of domains using Wide00013 web graph - Top ranked pages (up to a max of 100) from every linked-to domain using the Wide00012 inter-domain navigational link graph -- a ranking of all URLs that have more than one incoming inter-domain link (rank was determined by number of incoming links using Wide00012 inter domain links)...
Custom Crawl Services
Custom Crawl Services
collection
143,284
ITEMS
1.7B
VIEWS
by Internet Archive
collection

eye 1.7B

Large-scale web harvests and national domain crawls performed for National Libraries, National Archives, preservation partners, research initiatives, and as part of special projects and custom crawling and research services.
Topic: ccs
Common Crawl
Common Crawl
collection
22,656
ITEMS
380.1M
VIEWS
collection

eye 380.1M

Web crawl data from Common Crawl.
Wikipedia Eventstream
collection
52,462
ITEMS
426.2M
VIEWS
collection

eye 426.2M

Wikipedia Near Real Time (from IRC)
Wikipedia Near Real Time (from IRC)
collection
18,249
ITEMS
1.5B
VIEWS
collection

eye 1.5B

This is a collection of web page captures from links added to, or changed on, Wikipedia pages. The idea is to bring a reliability to Wikipedia outlinks so that if the pages referenced by Wikipedia articles are changed, or go away, a reader can permanently find what was originally referred to. This is part of the Internet Archive's attempt to rid the web of broken links .
Topics: Wikipedia, Wikimedia
Twitter Outlinks
collection
72,427
ITEMS
730.4M
VIEWS
collection

eye 730.4M

This is a Collection of URLs (and Outlinked URLs) extracted from a random feed of 1% of all Tweets.
Survey Crawl Number 8
collection
8,758
ITEMS
984.7M
VIEWS
collection

eye 984.7M

Top News
collection
135,952
ITEMS
1B
VIEWS
collection

eye 1B

A daily collection of hundreds of the world's top news sites.
Topics: daily, news
Mark Graham's Archive-IT Collections
Mark Graham's Archive-IT Collections
collection
6,084,346
ITEMS
380.6M
VIEWS
collection

eye 380.6M

This is a set of web collections curated by Mark Graham using the Archive-IT service of the Internet Archive. They include web captures of the ISKME.org website as well as captures from sites hosted by IGC.org. These web captures are available to the general public. For more information about this collection please feel free to contact Mark via Send Mail
Topic: Mark Graham, ISKME, IGC.org
collection

eye 1.8B

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection

eye 1.3B

Web wide crawl.
GDELT
GDELT
collection
57,656
ITEMS
1B
VIEWS
collection

eye 1B

A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
collection

eye 1.1B

Web wide crawl number 16 The seed list for Wide00016 was made from the join of the top 1 million domains from CISCO and the top 1 million domains from Alexa.
collection

eye 1.1B

The seeds for this crawl came from: 251 million Domains that had at least one link from a different domain in the Wayback Machine, across all time ~ 300 million Domains that we had in the Wayback, across all time 55,945,067 Domains from https://archive.org/details/wide00016 This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) The WARC files associated with this crawl are not currently available to the general public.
alexa_2007
alexa_2007
collection
7,636
ITEMS
1.4B
VIEWS
collection

eye 1.4B

this data is currently not publicly accessible.
Wide Crawl Number 12 - started March, 14th 2015
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
1.2B
VIEWS
collection

eye 1.2B

Web wide crawl with initial seedlist and crawler configuration from January 2015.
Archive Team: URLs
collection
58,020
ITEMS
156.9M
VIEWS
by Archive Team
collection

eye 156.9M

Wide Crawl started April 2013
Wide Crawl started April 2013
collection
25,035
ITEMS
1.2B
VIEWS
collection

eye 1.2B

Web wide crawl with initial seedlist and crawler configuration from April 2013.
Wide Crawl started June 2014
Wide Crawl started June 2014
collection
45,341
ITEMS
1.1B
VIEWS
collection

eye 1.1B

Web wide crawl with initial seedlist and crawler configuration from June 2014.
web-group-internal
collection
33,241
ITEMS
1.2B
VIEWS
collection

eye 1.2B

miscellaneous data
Topic: brad tofel
collection

eye 1.3B

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wikipedia Outlinks March 2016
Wikipedia Outlinks March 2016
collection
27,594
ITEMS
780.9M
VIEWS
collection

eye 780.9M

Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
GDELT Project
collection
30,441
ITEMS
215.1M
VIEWS
collection

eye 215.1M

Wordpress Blogs and the Pages They Link To
Wordpress Blogs and the Pages They Link To
collection
51,509
ITEMS
660.9M
VIEWS
collection

eye 660.9M

This is a collection of pages and embedded objects from WordPress blogs and the external pages they link to. Captures of these pages are made on a continuous basis seeded from a feed of new or changed pages hosted by Wordpress.com or by Wordpress pages hosted by sites running a properly configured Jetpack wordpress plugin.
Topics: Wordpress.com, blogs, jetpack
Survey Crawl Number 7
Survey Crawl Number 7
collection
6,605
ITEMS
767.4M
VIEWS
collection

eye 767.4M

This "Survey" crawl was started on Feb. 24, 2018. This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) Survey 7 is based on a seed list of 339,249,218 URLs which is all the URLs in the Wayback Machine that we saw a 200 response code from in 2017 based on a query we ran on Feb. 1st, 2018.   The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl Number 13
Wide Crawl Number 13
collection
46,050
ITEMS
861.1M
VIEWS
collection

eye 861.1M

Web Wide Crawl Number 13
collection

eye 973.9M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Outlinks From Tweets
collection
23,845
ITEMS
195.2M
VIEWS
collection

eye 195.2M

Wayback Indexes
Wayback Indexes
collection
554
ITEMS
1B
VIEWS
collection

eye 1B

Wayback indexes. This data is currently not publicly accessible.
Wide Crawl started August 2013
Wide Crawl started August 2013
collection
21,932
ITEMS
807.2M
VIEWS
collection

eye 807.2M

Web wide crawl with initial seedlist and crawler configuration from August 2013.
alexa_2006
alexa_2006
collection
6,506
ITEMS
826.7M
VIEWS
collection

eye 826.7M

this data is currently not publicly accessible.
Archive Team: The News Roundup
Archive Team: The News Roundup
collection
38,892
ITEMS
633.3M
VIEWS
by Archive Team
collection

eye 633.3M

Archive Team now searches many, many news sites, including extensive worldwide and obscure sources, to capture unique news stories for history.
collection

eye 704.9M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl started January 2012
Wide Crawl started January 2012
collection
30,373
ITEMS
703.1M
VIEWS
collection

eye 703.1M

Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
National Library of Australia Crawls
National Library of Australia Crawls
collection
46,986
ITEMS
436.5M
VIEWS
collection

eye 436.5M

Crawls performed by Internet Archive on behalf of the National Library of Australia. This data is currently not publicly accessible.
alexa_web_2009
alexa_web_2009
collection
3,080
ITEMS
551.4M
VIEWS
collection

eye 551.4M

this data is currently not publicly accessible.
alexa_web_2010
alexa_web_2010
collection
2,994
ITEMS
516.9M
VIEWS
collection

eye 516.9M

this data is currently not publicly accessible.
Internet Archive Research Publication Crawls
Internet Archive Research Publication Crawls
collection
20,355
ITEMS
42.3M
VIEWS
by Internet Archive Web Group
collection

eye 42.3M

A series of open web crawls targeting journal articles, technical memos, essays, datasets, and other research publications. This collection contains WARC and CDX files that end up in Wayback ( https://web.archive.org ). See also bibliographic metadata corpuses at  https://archive.org/details/ia_biblio_metadata
collection

eye 607.4M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl started April 2012
Wide Crawl started April 2012
collection
39,279
ITEMS
615.7M
VIEWS
collection

eye 615.7M

Web wide crawl with initial seedlist and crawler configuration from April 2012.
Wide Crawl started February 2014
Wide Crawl started February 2014
collection
9,806
ITEMS
515.4M
VIEWS
collection

eye 515.4M

Web wide crawl with initial seedlist and crawler configuration from February 2014.
Library of Congress
collection
507,495
ITEMS
107.5M
VIEWS
collection

eye 107.5M

Archive-It Partner 126: Library of Congress
Wide Crawl started September 2012
Wide Crawl started September 2012
collection
22,423
ITEMS
452.5M
VIEWS
collection

eye 452.5M

Web wide crawl with initial seedlist and crawler configuration from September 2012.
Media Cloud
Media Cloud
collection
17,499
ITEMS
155.6M
VIEWS
collection

eye 155.6M

.com survey started January 2011
.com survey started January 2011
collection
2,535
ITEMS
461.1M
VIEWS
collection

eye 461.1M

Survey crawl of .com domains started January 2011.
Topic: webcrawl
Wide Crawl Started January 2013
Wide Crawl Started January 2013
collection
15,157
ITEMS
457.8M
VIEWS
collection

eye 457.8M

Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Wide Crawl started October 2011
Wide Crawl started October 2011
collection
12,648
ITEMS
434.6M
VIEWS
collection

eye 434.6M

Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
Wide Crawl started October 2010
Wide Crawl started October 2010
collection
15,839
ITEMS
473.5M
VIEWS
collection

eye 473.5M

Web wide crawl with initial seedlist and crawler configuration from October 2010
collection

eye 334M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Host Screen Captures
Host Screen Captures
collection
17,413
ITEMS
144.4M
VIEWS
collection

eye 144.4M

Screen captures of hosts discovered during wide crawls. This data is currently not publicly accessible.
Survey Crawl Number 9
collection
561
ITEMS
167.1M
VIEWS
collection

eye 167.1M

Wide Crawl started March 2011
Wide Crawl started March 2011
collection
8,528
ITEMS
401.6M
VIEWS
collection

eye 401.6M

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
51_crawl
51_crawl
collection
1,138
ITEMS
349.2M
VIEWS
collection

eye 349.2M

this data is currently not publicly accessible.
Internet Memory Foundation
Internet Memory Foundation
collection
1,918
ITEMS
215.3M
VIEWS
collection

eye 215.3M

Data crawled on behalf of Internet Memory Foundation . This data is currently not publicly accessible. from Wikipedia : The Internet Memory Foundation (formerly the European Archive Foundation) is a non profit foundation whose purpose is archiving web content, it supports projects and research which include the preservation and protection of multimedia content. Its archives form a digital library of cultural content.
Live Web Proxy Crawls
web

eye 24.4M

favorite 3

comment 0

International News Crawls
International News Crawls
collection
8,238
ITEMS
218.5M
VIEWS
collection

eye 218.5M

Crawls of International News Sites
Political TV Ad Archive
Political TV Ad Archive
collection
83,152
ITEMS
184.9M
VIEWS
collection

eye 184.9M

Archive-It Partner 1067: The Political TV Ad Archive, a project of the Internet Archive, collects political TV ads and social media sites in key 2016 primary election states, unlocking the metadata underneath and highlighting quality journalism to provide journalists, civic organizations, academics, and the general public with reliable information on who is trying to influence them & how.
Wikipedia Outlinks February 2012
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
317.2M
VIEWS
collection

eye 317.2M

Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.