Repositories list

• robotstxt-experiments

  Public

  commoncrawl/robotstxt-experiments’s past year of commit activity
  How is the Robots Exclusion Protocol (robots.txt) used in the WWW? This projects tries to get some insights mining Common Crawl's robots.txt captures of the yea…

  robots-txtrobotstxt

  robots-txtrobotstxt

  Jupyter Notebook

  •

  MIT License

  •00 forks•00 stars•00 issues•00 pull requests•Updated Apr 17, 2026Apr 17, 2026

• cc-index-table

  Public

  commoncrawl/cc-index-table’s past year of commit activity
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3

  sqlsparkcolumnar-storageaws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •1515 forks•128128 stars•66 issues•33 pull requests•Updated Apr 17, 2026Apr 17, 2026

• nutch

  Public
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2

  javabig-datahadoopweb-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k1.3k forks•4141 stars•1010 issues•33 pull requests•Updated Apr 17, 2026Apr 17, 2026

• news-crawl

  Public

  commoncrawl/news-crawl’s past year of commit activity
  News crawling with StormCrawler - stores content as WARC

  crawlernewsweb-crawler+ 5

  crawlernewsweb-crawlerapache-stormwarccommoncrawlcommon-crawlstorm-crawler

  Java

  •

  Apache License 2.0

  •3939 forks•365365 stars•1414 issues•33 pull requests•Updated Apr 14, 2026Apr 14, 2026

• cc-quick-scripts

  Public
  Scripts to verify Common Crawl segments and WARC/WET/WAT files

  internal-tools

  internal-tools

  Python

  •

  MIT License

  •44 forks•22 stars•00 issues•00 pull requests•Updated Apr 14, 2026Apr 14, 2026

• cc-crawl-statistics

  Public

  commoncrawl/cc-crawl-statistics’s past year of commit activity
  Statistics of Common Crawl monthly archives mined from URL index files

  statisticscommoncrawlcommon-crawl

  statisticscommoncrawlcommon-crawl

  Python

  •

  Apache License 2.0

  •1616 forks•218218 stars•33 issues•11 pull request•Updated Apr 14, 2026Apr 14, 2026

• cc-host-index

  Public
  Tools for working with the host index

  Python

  •33 forks•1313 stars•11 issue•11 pull request•Updated Apr 14, 2026Apr 14, 2026

• whirlwind-python

  Public

  commoncrawl/whirlwind-python’s past year of commit activity
  A whirlwind tour of Common Crawl's data using Python

  pythontutorialarchive+ 1

  pythontutorialarchivewarc

  Python

  •

  Apache License 2.0

  •99 forks•4141 stars•00 issues•11 pull request•Updated Apr 13, 2026Apr 13, 2026

• cc-index-annotations

  Public

  commoncrawl/cc-index-annotations’s past year of commit activity
  Example code to join an annotation to a host or url index

  Python

  •00 forks•11 star•00 issues•00 pull requests•Updated Apr 13, 2026Apr 13, 2026

• cc-downloader

  Public

  commoncrawl/cc-downloader’s past year of commit activity
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •55 forks•7676 stars•99 issues•11 pull request•Updated Apr 11, 2026Apr 11, 2026

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •9090 forks•22 stars•00 issues•22 pull requests•Updated Apr 7, 2026Apr 7, 2026

• whirlwind-java

  Public
  A whirlwind tour of Common Crawl's data using Java

  Java

  •

  Apache License 2.0

  •00 forks•33 stars•22 issues•11 pull request•Updated Apr 1, 2026Apr 1, 2026

• cc-webgraph-statistics

  Public
  Statistics of Common Crawl monthly Web Graphs

  JavaScript

  •

  Apache License 2.0

  •11 fork•55 stars•00 issues•11 pull request•Updated Mar 31, 2026Mar 31, 2026

• eot2020-host-index

  Public
  Tools to work with the preliminary End of Term Archive host index

  Python

  •00 forks•00 stars•00 issues•00 pull requests•Updated Mar 30, 2026Mar 30, 2026

• cc-webgraph

  Public
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3

  pagerankwebgraphcommoncrawlcommon-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •66 forks•108108 stars•77 issues•00 pull requests•Updated Mar 26, 2026Mar 26, 2026

• cc-pyspark

  Public
  Process Common Crawl data with Python and Spark

  sparkpysparksparksql+ 5

  sparkpysparksparksqlwetcommoncrawlcommon-crawlwarc-fileswat-files

  Python

  •

  MIT License

  •9494 forks•454454 stars•44 issues•22 pull requests•Updated Mar 26, 2026Mar 26, 2026

• cdx_toolkit

  Public
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  pythonwarcweb-archiving+ 4

  pythonwarcweb-archivingcdxweb-archivescommoncrawlcdx-api

  Python

  •

  Apache License 2.0

  •3434 forks•202202 stars•66 issues•44 pull requests•Updated Mar 23, 2026Mar 23, 2026

• ipv6-analysis

  Public
  A survey of IPv6 support among the 100k top ranked hosts on the web

  Python

  •

  Apache License 2.0

  •00 forks•11 star•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• warcio-s3

  Public
  Streaming WARC/ARC library for fast web archive IO

  Python

  •

  Apache License 2.0

  •6868 forks•00 stars•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• cc-citations

  Public
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  bibtexbibliographyopendata

  Jupyter Notebook

  •44 forks•2828 stars•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• cc-nutch-example

  Public
  Apache Nutch example project to archive content in WARC files

  Shell

  •

  Apache License 2.0

  •22 forks•33 stars•00 issues•00 pull requests•Updated Mar 17, 2026Mar 17, 2026

• awesome-low-resource-languages

  Public

  commoncrawl/awesome-low-resource-languages’s past year of commit activity
  This list provides resources useful for documenting, conserving, developing, preserving, or working with endangered and low resource languages.

  00 forks•00 stars•00 issues•00 pull requests•Updated Mar 16, 2026Mar 16, 2026

• cc-web-graph-neo4j

  Public
  Instructions and code for using the Common Crawl Web Graph in Neo4j format

  Shell

  •00 forks•22 stars•00 issues•00 pull requests•Updated Mar 16, 2026Mar 16, 2026

• cc-warc-examples

  Public
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1

  javahadoopmapreducecommoncrawl

  Java

  •

  MIT License

  •4545 forks•3838 stars•00 issues•00 pull requests•Updated Mar 12, 2026Mar 12, 2026

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •7676 forks•1111 stars•44 issues•11 pull request•Updated Mar 11, 2026Mar 11, 2026

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •22 forks•1717 stars•33 issues•11 pull request•Updated Feb 26, 2026Feb 26, 2026

• cc-citations-paper-explorer

  Public
  A visual paper explorer based on cc-citations. https://huggingface.co/spaces/commoncrawl/cc-citations

  embeddingsldacommoncrawl+ 1

  embeddingsldacommoncrawlscincl

  Python

  •

  Apache License 2.0

  •00 forks•00 stars•00 issues•00 pull requests•Updated Feb 19, 2026Feb 19, 2026

• presentations

  Public
  A collection of public presentations from the Common Crawl Foundation

  11 fork•1111 stars•00 issues•00 pull requests•Updated Feb 2, 2026Feb 2, 2026

• webarchive-indexing

  Public
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

  cc-index

  cc-index

  Python

  •

  MIT License

  •1212 forks•66 stars•00 issues•00 pull requests•Updated Jan 27, 2026Jan 27, 2026

• cc-mrjob

  Public archive
  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

  pythonhadoopmap-reduce+ 1

  pythonhadoopmap-reducecommoncrawl

  Python

  •

  MIT License

  •7878 forks•168168 stars•22 issues•22 pull requests•Updated Jan 27, 2026Jan 27, 2026