/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.
You are here

 commoncrawl.org
Common Crawl - Blog - Index to WARC Files and URLs in Columnar Format
| | data.commoncrawl.org
Common Crawl Index Table
0.5 parsecs away

Travel
| | [AI summary] A technical reference document providing access methods and schema details for the Common Crawl internet archive index.
| | skeptric.com
skeptric - Read Common Crawl Parquet Metadata with Python
2.3 parsecs away

Travel
| | [AI summary] The author demonstrates how to efficiently query Common Crawl's Apache Parquet web crawl data using Python libraries like PyArrow, fastparquet, and asyncio.
| | avilpage.com
Common Crawl On Laptop - Extracting Subset Of Data | Avil Page
1.2 parsecs away

Travel
| | How to process entire common crawl data set from your local machine.
| | skeptric.com
skeptric - Extracing Text, Metadata and Data from Common Crawl
4.4 parsecs away

Travel
| [AI summary] The article explains how to access and process different data formats (WET, WAT, and WARC) from the Common Crawl open web dataset using Python libraries and Spark.