feat: add py3langid to the benchmark

[adbar]

Add py3langid

Wrapper for py3langid 0.4.0
(pure Python + numpy, byte n-gram naive Bayes with a word table, 139 labels).

• requires_preprocessing = False: the model normalizes and tokenizes raw text itself.
• Returns LIDPrediction(label, posterior); the score is the normalized
  probability of the top label. Predictions below 0.25 abstain but keep
  their score.
• zxx (not a language) and und abstain.
• Output map for codes the generic 639-1 → 639-3 conformation would not
  reach: ar→arb, sw→swh, az→azj, or→ory (the registers the model is trained
  on, as in FLORES/OpenLID) and bcl→bik (macrolanguage).

Training-data disclosure

• The training corpus was decontaminated against CommonLID by exact line
  match; residual overlap is zero.
• No UDHR text is in the corpus. google/smol was never used for training.
• commonlid and flores_dev were used for evaluation purposes only.
• Training sources: CC-100, Wikipedia, Tatoeba, Leipzig corpora, plus
  Glot500, GlotCC and GlotSparse for thin classes.

See also https://huggingface.co/datasets/commoncrawl/commonlid-results/discussions/6

[malteos]

Hi @adbar

thanks for your contribution. Just a few minor comments that should be quickly to fix. Please also resolve the merge conflict so we can run the CI.

Best,
Malte

[adbar]

@malteos Thanks for the review, the PR should now be OK.