Outer Web | Explore

/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.

You are here		data.commoncrawl.org Common Crawl Index Table
\|	\|	commoncrawl.org Common Crawl - Blog - Index to WARC Files and URLs in Columnar Format	1.0 parsecs away Travel
\|	\|	We're happy to announce the release of an index to WARC files and URLs in a columnar format. The columnar format (we use Apache Parquet) allows to efficiently query or process the index and saves time and computing resources. Especially, if only few columns are accessed, recent big data tools will run impressively fast.	1.0 parsecs away Travel
\|	\|	digitalpebble.blogspot.com DigitalPebble's Blog: Need billions of web pages? Don't bother crawling...	1.3 parsecs away Travel
\|	\|	How big did you say? I am often contacted by prospective clients to help them crawl the web on a very large scale or find questions such...	1.3 parsecs away Travel
\|	\|	skeptric.com skeptric - Extracing Text, Metadata and Data from Common Crawl	1.6 parsecs away Travel
\|	\|	[AI summary] The article explains how to access and process different data formats (WET, WAT, and WARC) from the Common Crawl open web dataset using Python libraries and Spark.	1.6 parsecs away Travel
\|	\|	commoncrawl.org Common Crawl - Blog - September/October 2023 crawl archive now available	1.8 parsecs away Travel
\|		The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 - October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.	1.8 parsecs away Travel