Outer Web | Explore

/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.

You are here		skeptric.com skeptric - Read Common Crawl Parquet Metadata with Python
\|	\|	freeman.vc Parsing Common Crawl in a day for $60 - Pierce.dev	5.8 parsecs away Travel
\|	\|	In addition to forming a bulk of the foundation of modern language models, there's a ton of other data buried within Common Crawl. Incoming and external links to websites, referral codes, leaked data. If it's public on the Internet, there's a good chance CC has it somewhere within its index. Here we parse all of common crawl in a day, on the cheap.	5.8 parsecs away Travel
\|	\|	data.commoncrawl.org Common Crawl Index Table	2.0 parsecs away Travel
\|	\|	[AI summary] A technical reference document providing access methods and schema details for the Common Crawl internet archive index.	2.0 parsecs away Travel
\|	\|	avilpage.com Common Crawl On Laptop - Extracting Subset Of Data \| Avil Page	1.5 parsecs away Travel
\|	\|	How to process entire common crawl data set from your local machine.	1.5 parsecs away Travel
\|	\|	skeptric.com skeptric - Extracing Text, Metadata and Data from Common Crawl	9.5 parsecs away Travel
\|		[AI summary] The article explains how to access and process different data formats (WET, WAT, and WARC) from the Common Crawl open web dataset using Python libraries and Spark.	9.5 parsecs away Travel