Feat/Download domains-top-1000.csv from s3

[handecelikkanat]

As part of the task to prepare and publish domains-top-N.csv files as part of the crawl pipeline, some processing that was previously done here is moved to main pipeline.

In particular, main pipeline now takes over:

• Preparing domains-top-1000-extended.csv.gz files, which include the features domain, pages, urls, hosts.
• Shares these files to here, together with other files that include crawl statistics.

cc-crawl-statistics then:

• Read also the new filedomains-top-1000-extended.csv.gz in get_stats.sh
• Pipes size excerpt to plot/domain.py in plot.sh
• In domain.py combined the features from the read file + calculates and combines %pages and %urls features (using the size information from the argument excerpt)
• Publishes the resulting file in domains.html

To-Do just before merging, after cc-crawl-statistics is updated by the crawl operator:

• Pull updated main, rebase this branch on it (main should now shows October in crawlstats.py as latest crawl)
• Run statistics package again, calculate now 1000-values for October crawl
• git mv new files: data/domains-top-1000.csv and data/domains-top-1000.html, showing October values, onto old files data/domains-top-500.csv and `data/domains-top-500.html
• Now PR can be merged.

Decisions, please check:

• Intentionally does not change stats/{crawl}.gz files (ie. does not change crawlstats.py and run_stats_hadoop.py, to ensure compatibility with previous files.
• Intentionally does not remove building of stats/excerpt/domain.json.gz, in case another downstream user needs it.

Validation:

I tested this setup using the Docker container and a test csv file that I upload beforehand to the test bucket.

@sebastian-nagel I left this file there so you too can test if you like using this.

Some settings are needed:

0. If CC-MAIN-2026-39 is the latest crawl in crawlstats.py, comment it out. Use August 2026 crawl (CC-MAIN-2026-34) as latest instead.

1. In get_stats.sh, I had to add a || true at the end of line that reads from AWS each top-domains file, because for test purposes I have only one. This I will not commit to the eventual version because Id need the script to fail on prod.

We can make this a command line argument, but didnt look worthy to bog down the script to me.

Thoughts?

ie.:

     if [ -e ${TOP_DOMAINS_TARGET} ]; then
         echo "  ... top-domains exist"
     elif $ON_AWS; then
-        aws s3 cp ${TOP_DOMAINS_S3}/$crawl/stats/${TOP_DOMAINS_FILE} ${TOP_DOMAINS_TARGET}
+        aws s3 cp ${TOP_DOMAINS_S3}/$crawl/stats/${TOP_DOMAINS_FILE} ${TOP_DOMAINS_TARGET} || true
     else
         curl --silent --fail ${TOP_DOMAINS_URL}/$crawl/stats/${TOP_DOMAINS_FILE} -o ${TOP_DOMAINS_TARGET} || { rm -f ${TOP_DOMAINS_TARGET}; false; }
     fi

2. Then run Docker, TOP_DOMAINS_S3 must be overriden to the test location that crawl-tools test uploaded the temp test files. eg.:

docker run --rm -it \
    -e TOP_DOMAINS_S3=s3://<test-bucket>/<test-prefix> \
    -v ~/.aws:/root/.aws:ro \
    -v $(pwd -P):/app \
    ghcr.io/commoncrawl/cc-crawl-statistics/stats:latest bash

3. Then in Docker shell:

$ echo $TOP_DOMAINS_S3    ### Sanity check
<...>

$ aws s3 ls $TOP_DOMAINS_S3/CC-MAIN-2026-34/stats/domains-top-1000-extended.csv.gz  ### Sanity check
<...> domains-top-1000-extended.csv.gz

$ ./get_stats.sh
# This will skip many failed files, check only for CC-MAIN-2026-34

$ ls stats/top-domains/   
CC-MAIN-2026-34.domains-top-1000-extended.csv.gz

$ ./plot.sh
# Check and compare values with current crawl values at `https://commoncrawl.github.io/cc-crawl-statistics/plots/domains.html`
#    - plots/domains.md
#    - plots/domains-top-1000.csv
#    - plots/domains-top-1000.html

4. In a local shell, can view the generated files using the site.Dockerfile, which will redirect output to http://localhost:4000/

docker build -f site.Dockerfile -t ccsite .
docker run --rm -p 4000:4000 ccsite bundle exec jekyll serve --host 0.0.0.0

[handecelikkanat]

@sebastian-nagel I intend to not touch crawlstats.py and run_stats_hadoop.py for now. They will continue calculating the statistics for top 500 domains (which can be updated to 1000?)

I will update plot/domain.py to use data from my own files rather than from crawl.gz files. Feels like less amount of change in one go.

Let me know if you prefer to update everything all at once. I thought if current system fails, its good to have the old one also running in parallel for another month. Plus I think some other stats calculations use these domain numbers, which I dont want to mess with.

Opinions?

[sebastian-nagel]

Thanks, @handecelikkanat. Looks good.

Don't forget: the top-500 domain files (CSV and HTML) need to be removed from Git and replaced by the top-1000 files. This should be part of the PR.

[handecelikkanat]

Don't forget: the top-500 domain files (CSV and HTML) need to be removed from Git and replaced by the top-1000 files. This should be part of the PR.

• Yes this makes sense, thank you. How is better to handle this? Im always confused which was is clearer for git history. Options: (1) rm 500-files + add 1000-files, (2) or mv (i.e., rename) 500-files to 1000-files.

• I will leave this PR open, it is to be merged at the end of October crawl.
  (I dont know if it is better to mark as Draft in the meanwhile.)

[handecelikkanat]

I am leaving this file (repo-only) uncompressed, so its easier for the github.io users to open.

[handecelikkanat]

@sebastian-nagel Converted internal (extended) file to compressed.

Ready for your review. Intentionally left as "Draft" so I dont accidentally merge before completing the To-Do items after crawl, noted above with checkboxes to tick. Those will create the new plots/domains-top-1000.csv and plots/domains-top-1000.html files to add to the PR.