Repositories list

• warc2zip

  Public

  commoncrawl/warc2zip’s past year of commit activity
  Convert WARC files into zip archives that are good for you.

  zipwarc

  zipwarc

  Python

  •11 fork in commoncrawl/warc2zip•44 stars in commoncrawl/warc2zip•33 issues in commoncrawl/warc2zip•33 pull requests in commoncrawl/warc2zip•Updated Oct 18, 2026Oct 18, 2026

• cc-downloader

  Public

  commoncrawl/cc-downloader’s past year of commit activity
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •99 forks in commoncrawl/cc-downloader•9797 stars in commoncrawl/cc-downloader•1010 issues in commoncrawl/cc-downloader•22 pull requests in commoncrawl/cc-downloader•Updated Oct 18, 2026Oct 18, 2026

• cc-citations

  Public

  commoncrawl/cc-citations’s past year of commit activity
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  bibtexbibliographyopendata

  Jupyter Notebook

  •44 forks in commoncrawl/cc-citations•3131 stars in commoncrawl/cc-citations•00 issues in commoncrawl/cc-citations•11 pull request in commoncrawl/cc-citations•Updated Oct 18, 2026Oct 18, 2026

• news-crawl

  Public

  commoncrawl/news-crawl’s past year of commit activity
  News crawling with StormCrawler - stores content as WARC

  crawlernewsweb-crawler+ 5

  crawlernewsweb-crawlerapache-stormwarccommoncrawlcommon-crawlstorm-crawler

  Java

  •

  Apache License 2.0

  •4242 forks in commoncrawl/news-crawl•380380 stars in commoncrawl/news-crawl•1212 issues in commoncrawl/news-crawl•66 pull requests in commoncrawl/news-crawl•Updated Oct 17, 2026Oct 17, 2026

• cc-crawl-statistics

  Public

  commoncrawl/cc-crawl-statistics’s past year of commit activity
  Statistics of Common Crawl monthly archives mined from URL index files

  statisticscommoncrawlcommon-crawl

  statisticscommoncrawlcommon-crawl

  Python

  •

  Apache License 2.0

  •1919 forks in commoncrawl/cc-crawl-statistics•233233 stars in commoncrawl/cc-crawl-statistics•22 issues in commoncrawl/cc-crawl-statistics•11 pull request in commoncrawl/cc-crawl-statistics•Updated Oct 17, 2026Oct 17, 2026

• cdx_toolkit

  Public

  commoncrawl/cdx_toolkit’s past year of commit activity
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  pythonwarcweb-archiving+ 4

  pythonwarcweb-archivingcdxweb-archivescommoncrawlcdx-api

  Python

  •

  Apache License 2.0

  •3636 forks in commoncrawl/cdx_toolkit•218218 stars in commoncrawl/cdx_toolkit•55 issues in commoncrawl/cdx_toolkit•55 pull requests in commoncrawl/cdx_toolkit•Updated Oct 16, 2026Oct 16, 2026

• cc-index-table

  Public

  commoncrawl/cc-index-table’s past year of commit activity
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3

  sqlsparkcolumnar-storageaws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •1717 forks in commoncrawl/cc-index-table•132132 stars in commoncrawl/cc-index-table•55 issues in commoncrawl/cc-index-table•77 pull requests in commoncrawl/cc-index-table•Updated Oct 15, 2026Oct 15, 2026

• cc-webgraph

  Public

  commoncrawl/cc-webgraph’s past year of commit activity
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3

  pagerankwebgraphcommoncrawlcommon-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •77 forks in commoncrawl/cc-webgraph•116116 stars in commoncrawl/cc-webgraph•77 issues in commoncrawl/cc-webgraph•33 pull requests in commoncrawl/cc-webgraph•Updated Oct 15, 2026Oct 15, 2026

• commonlid-eval

  Public

  commoncrawl/commonlid-eval’s past year of commit activity
  Benchmark for language identification by the CommonLID project

  Python

  •

  Apache License 2.0

  •22 forks in commoncrawl/commonlid-eval•33 stars in commoncrawl/commonlid-eval•44 issues in commoncrawl/commonlid-eval•22 pull requests in commoncrawl/commonlid-eval•Updated Oct 14, 2026Oct 14, 2026

• langid-eval

  Public

  commoncrawl/langid-eval’s past year of commit activity
  Tuneable fuzzy matching for language identification evaluation

  Python

  •

  MIT License

  •00 forks in commoncrawl/langid-eval•00 stars in commoncrawl/langid-eval•00 issues in commoncrawl/langid-eval•00 pull requests in commoncrawl/langid-eval•Updated Oct 12, 2026Oct 12, 2026

• crawler-commons

  Public

  commoncrawl/crawler-commons’s past year of commit activity
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •9292 forks in commoncrawl/crawler-commons•22 stars in commoncrawl/crawler-commons•00 issues in commoncrawl/crawler-commons•22 pull requests in commoncrawl/crawler-commons•Updated Oct 2, 2026Oct 2, 2026

• cc-warc-examples

  Public

  commoncrawl/cc-warc-examples’s past year of commit activity
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1

  javahadoopmapreducecommoncrawl

  Java

  •

  MIT License

  •4444 forks in commoncrawl/cc-warc-examples•3838 stars in commoncrawl/cc-warc-examples•00 issues in commoncrawl/cc-warc-examples•00 pull requests in commoncrawl/cc-warc-examples•Updated Oct 1, 2026Oct 1, 2026

• ia-hadoop-tools

  Public

  commoncrawl/ia-hadoop-tools’s past year of commit activity
  Web archiving tools on Hadoop

  Java

  •2727 forks in commoncrawl/ia-hadoop-tools•44 stars in commoncrawl/ia-hadoop-tools•22 issues in commoncrawl/ia-hadoop-tools•11 pull request in commoncrawl/ia-hadoop-tools•Updated Oct 1, 2026Oct 1, 2026

• ia-web-commons

  Public

  commoncrawl/ia-web-commons’s past year of commit activity
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •7878 forks in commoncrawl/ia-web-commons•1212 stars in commoncrawl/ia-web-commons•44 issues in commoncrawl/ia-web-commons•11 pull request in commoncrawl/ia-web-commons•Updated Oct 1, 2026Oct 1, 2026

• cc-webgraph-statistics

  Public

  commoncrawl/cc-webgraph-statistics’s past year of commit activity
  Statistics of Common Crawl monthly Web Graphs

  JavaScript

  •

  Apache License 2.0

  •11 fork in commoncrawl/cc-webgraph-statistics•88 stars in commoncrawl/cc-webgraph-statistics•00 issues in commoncrawl/cc-webgraph-statistics•00 pull requests in commoncrawl/cc-webgraph-statistics•Updated Sep 22, 2026Sep 22, 2026

• cc-pyspark

  Public

  commoncrawl/cc-pyspark’s past year of commit activity
  Process Common Crawl data with Python and Spark

  sparkpysparksparksql+ 5

  sparkpysparksparksqlwetcommoncrawlcommon-crawlwarc-fileswat-files

  Python

  •

  MIT License

  •9898 forks in commoncrawl/cc-pyspark•459459 stars in commoncrawl/cc-pyspark•44 issues in commoncrawl/cc-pyspark•22 pull requests in commoncrawl/cc-pyspark•Updated Sep 17, 2026Sep 17, 2026

• cc-notebooks

  Public

  commoncrawl/cc-notebooks’s past year of commit activity
  Various Jupyter notebooks about Common Crawl data

  jupyter-notebookaws-athenacommoncrawl+ 3

  jupyter-notebookaws-athenacommoncrawlcommon-crawlwebarchivingwebgraph-framework

  Jupyter Notebook

  •

  Apache License 2.0

  •1111 forks in commoncrawl/cc-notebooks•6969 stars in commoncrawl/cc-notebooks•00 issues in commoncrawl/cc-notebooks•00 pull requests in commoncrawl/cc-notebooks•Updated Sep 8, 2026Sep 8, 2026

• web-graph-embeddings

  Public

  commoncrawl/web-graph-embeddings’s past year of commit activity
  Graph embedding built using Common Crawl's host-level hyperlink graph

  Python

  •

  Apache License 2.0

  •00 forks in commoncrawl/web-graph-embeddings•22 stars in commoncrawl/web-graph-embeddings•00 issues in commoncrawl/web-graph-embeddings•00 pull requests in commoncrawl/web-graph-embeddings•Updated Sep 7, 2026Sep 7, 2026

• cc-quick-scripts

  Public

  commoncrawl/cc-quick-scripts’s past year of commit activity
  Scripts to verify Common Crawl segments and WARC/WET/WAT files

  internal-tools

  internal-tools

  Python

  •

  MIT License

  •44 forks in commoncrawl/cc-quick-scripts•22 stars in commoncrawl/cc-quick-scripts•00 issues in commoncrawl/cc-quick-scripts•00 pull requests in commoncrawl/cc-quick-scripts•Updated Sep 4, 2026Sep 4, 2026

• cc-warcinfo-index-builder

  Public
  Code to build an index that maps warcinfo-id to crawl / warc

  Python

  •00 forks in commoncrawl/cc-warcinfo-index-builder•00 stars in commoncrawl/cc-warcinfo-index-builder•00 issues in commoncrawl/cc-warcinfo-index-builder•00 pull requests in commoncrawl/cc-warcinfo-index-builder•Updated Aug 31, 2026Aug 31, 2026

• web-languages

  Public

  commoncrawl/web-languages’s past year of commit activity
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

  crawlinglanguage-detectiondataset

  crawlinglanguage-detectiondataset

  101101 forks in commoncrawl/web-languages•7272 stars in commoncrawl/web-languages•55 issues in commoncrawl/web-languages•11 pull request in commoncrawl/web-languages•Updated Aug 24, 2026Aug 24, 2026

• llms-txt-experiments

  Public

  commoncrawl/llms-txt-experiments’s past year of commit activity
  Experiments on the usage of llms.txt in the Common Crawl corpus

  Python

  •

  Apache License 2.0

  •00 forks in commoncrawl/llms-txt-experiments•00 stars in commoncrawl/llms-txt-experiments•00 issues in commoncrawl/llms-txt-experiments•00 pull requests in commoncrawl/llms-txt-experiments•Updated Aug 19, 2026Aug 19, 2026

• cc-index-annotations

  Public

  commoncrawl/cc-index-annotations’s past year of commit activity
  Example code to join an annotation to a host or url index

  Python

  •00 forks in commoncrawl/cc-index-annotations•11 star in commoncrawl/cc-index-annotations•00 issues in commoncrawl/cc-index-annotations•00 pull requests in commoncrawl/cc-index-annotations•Updated Aug 17, 2026Aug 17, 2026

• nutch

  Public

  commoncrawl/nutch’s past year of commit activity
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2

  javabig-datahadoopweb-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k1.3k forks in commoncrawl/nutch•4343 stars in commoncrawl/nutch•88 issues in commoncrawl/nutch•00 pull requests in commoncrawl/nutch•Updated Aug 13, 2026Aug 13, 2026

• language-detection-cld2

  Public

  commoncrawl/language-detection-cld2’s past year of commit activity
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •22 forks in commoncrawl/language-detection-cld2•2020 stars in commoncrawl/language-detection-cld2•33 issues in commoncrawl/language-detection-cld2•11 pull request in commoncrawl/language-detection-cld2•Updated Aug 2, 2026Aug 2, 2026

• webarchive-indexing

  Public

  commoncrawl/webarchive-indexing’s past year of commit activity
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

  cc-index

  cc-index

  Python

  •

  MIT License

  •1313 forks in commoncrawl/webarchive-indexing•66 stars in commoncrawl/webarchive-indexing•00 issues in commoncrawl/webarchive-indexing•00 pull requests in commoncrawl/webarchive-indexing•Updated Jul 18, 2026Jul 18, 2026

• ipv6-analysis

  Public

  commoncrawl/ipv6-analysis’s past year of commit activity
  A survey of IPv6 support among the 100k top ranked hosts on the web

  Python

  •

  Apache License 2.0

  •00 forks in commoncrawl/ipv6-analysis•11 star in commoncrawl/ipv6-analysis•00 issues in commoncrawl/ipv6-analysis•00 pull requests in commoncrawl/ipv6-analysis•Updated Jul 17, 2026Jul 17, 2026

• eot-host-index

  Public

  commoncrawl/eot-host-index’s past year of commit activity
  Tools to work with the preliminary End of Term Archive host index

  Python

  •00 forks in commoncrawl/eot-host-index•00 stars in commoncrawl/eot-host-index•00 issues in commoncrawl/eot-host-index•00 pull requests in commoncrawl/eot-host-index•Updated Jul 13, 2026Jul 13, 2026

• open-data-registry

  Public

  commoncrawl/open-data-registry’s past year of commit activity
  A registry of publicly available datasets on AWS

  Python

  •

  Apache License 2.0

  •1.2k1.2k forks in commoncrawl/open-data-registry•22 stars in commoncrawl/open-data-registry•00 issues in commoncrawl/open-data-registry•00 pull requests in commoncrawl/open-data-registry•Updated Jul 9, 2026Jul 9, 2026

• ccbot-blocking-analysis

  Public

  commoncrawl/ccbot-blocking-analysis’s past year of commit activity
  Python

  •00 forks in commoncrawl/ccbot-blocking-analysis•00 stars in commoncrawl/ccbot-blocking-analysis•00 issues in commoncrawl/ccbot-blocking-analysis•00 pull requests in commoncrawl/ccbot-blocking-analysis•Updated Jun 22, 2026Jun 22, 2026