I realized that since USA rugby went bankrupt in 2020, it might delete it's stats pages. Turns out they did in fact move to Rugby Xplorer which looks like a much a better management system.
But what about those stats?
Well as of writing this you can still access things through links like https://usarugbystats.com/embed/team/696 but who knows how long that'll last. Their link to the list of CMS links was already gone! (https://assets.usarugby.org/docs/club/cms/USA-Rugby-CMS-List-of-Embed-Friendly-URLs.pdf)
Popped onto the wayback machine and luckily found it. It now lives on in this repo
So the wayback machine hasn't saved all of our stats so I wanted to make sure we could preserve them for UTD.
scrapy startproject utd_rugby_stats .scrapy genspider season https://usarugbystats.com/embed/club/696/season/scrapy crawl season| url (https://usarugbystats.com) | maps to |
|---|---|
| /api/stats/club/pts/696/2015-2016 | points |
| /api/stats/club/tr/696/2015-2016 | Tries |
| /api/stats/club/cv/696/2015-2016 | conversions |
| /api/stats/club/pk/696/2015-2016 | penalty kicks |
| /api/stats/club/dg/696/2015-2016 | drop goals |
| /api/stats/club/started/696/2015-2016 | Started |
| /api/stats/club/played/696/2015-2016 | Games played |
| /api/stats/club/yc/696/2015-2016 | Yellow cards |
| /api/stats/club/rc/696/2015-2016 | Red Cards |
I think I'm going to have to pull the competitions manually/ scrapy through it
Looks like you can just replace embedded with api and it'll spit back json
https://usarugbystats.com/api/competition/1409/match/29329
But it might just be easier to parse the tables...
Looks like the teams urls has:
- Schedule
- Players
- Standings
- Stats
- Save the overall table
- Follow the link for each game
- Save the match stream table
- Save the roster
- Download the table
- Follow the link to each player(https://usarugbystats.com/player/140195) and download
- https://usarugbystats.com/api/player/140195
- Player Statistics
- Team Statistics
- Match History
Download the table
This is where the API will be most beneficial
https://usarugbystats.com/api/stats/club/pts/696/2015-2016
2021-2022 2019-2020 2018-2019 2017-2018 2016-2017 2015-2016 2014-2015
| url (https://usarugbystats.com) | maps to |
|---|---|
| /api/stats/club/pts/696/2015-2016 | points |
| /api/stats/club/tr/696/2015-2016 | Tries |
| /api/stats/club/cv/696/2015-2016 | conversions |
| /api/stats/club/pk/696/2015-2016 | penalty kicks |
| /api/stats/club/dg/696/2015-2016 | drop goals |
| /api/stats/club/started/696/2015-2016 | Started |
| /api/stats/club/played/696/2015-2016 | Games played |
| /api/stats/club/yc/696/2015-2016 | Yellow cards |
| /api/stats/club/rc/696/2015-2016 | Red Cards |
https://usarugbystats.com/api/team/696/season/2021-2022
https://usarugbystats.com/embed/team/696/season/2021-2022 https://usarugbystats.com/embed/team/696/season/2021-2022/schedule https://usarugbystats.com/embed/team/696/season/2021-2022/players https://usarugbystats.com/embed/team/696/season/2021-2022/standings https://usarugbystats.com/embed/team/696/season/2021-2022/stats