Ace-CEFR: Best Model for Conversational Text Difficulty

This is a public demo project built with the Glite Autonomous Research Framework. It keeps the research trail, task folders, logs, datasets, model artifacts, and generated dashboards so readers can inspect how an ARF project evolves from setup through experiments and reporting.

Goal

Find the best model and the lowest Mean Squared Error on the Ace-CEFR benchmark — 890 English conversational passages labeled on the CEFR 1–6 scale. The project aims to beat the published BERT + PaLM ensemble (0.33 MSE) with a cleaner methodology that avoids test-set tuning, and to understand which architectural and feature choices drive the remaining headroom below human agreement (0.75 MSE for a new expert rater).

Research Questions

1. Which transformer backbone (BERT-base, DeBERTa-v3, ModernBERT, XLM-R) achieves the lowest test MSE on Ace-CEFR under a standardized two-stage fine-tuning recipe?
2. How does pseudo-label corpus size (1k, 10k, 100k) affect transfer performance in the two-stage recipe, and where does it saturate?
3. Do concept-dictionary and lexical features (per-concept CEFR mass, idiomaticity, phrasal-verb count) add predictive signal above word-length and sentence-length surface features?
4. Can the project reach or beat 0.33 MSE on a properly stratified split without tuning ensemble weights on held-out test samples?
5. How does per-source MSE (internal, authored, LLM-generated, learner, web) differ, and does the headline score mask systematic weakness on naturalistic conversation?

Success Criteria

• Reproduce the published BERT-base two-stage fine-tune baseline on Ace-CEFR within 0.05 MSE of the paper's 0.37 value, using a documented pipeline that runs from raw dataset to reported number.
• Achieve test MSE at or below 0.33 on a properly stratified split without tuning any weights on test samples, with paired bootstrap 90% confidence intervals reported.
• Produce at least one head-to-head comparison of transformer backbones with a statistical test that reports whether observed MSE differences are significant given the 445-passage test size.
• Publish per-source MSE breakdown so readers can distinguish performance on synthetic versus naturalistic text.
• Report inference latency for every final model on both CPU and API deployment paths.

Current Phase

Demo snapshot complete through the first 35 research tasks. The project has downloaded and inspected Ace-CEFR, reproduced linear and BERT baselines, evaluated curriculum-trained BERT, DeBERTa-v3, ModernBERT, CatBoost, embedding, Llama LoRA, and feature-injection variants, and materialized the aggregated dashboard in overview/. Three queued follow-ups (t0036, t0037, t0038) are documented in tasks/ as planned next steps but were not executed in this snapshot.

Results Dashboard

See overview/README.md for the latest aggregated results, metrics, and task status.

Getting Started

New contributors:

1. Clone the repo.
2. Install Git LFS, then run git lfs pull to fetch the large published artifacts.
3. Run uv sync to install dependencies.
4. Run /setup-project in Claude Code or Codex to complete one-time onboarding (safety acknowledgement, environment validation, and project scaffolding).

This repository intentionally contains large artifacts. A normal checkout plus LFS download is several GB because model weights, dataset files, paper PDFs, predictions, and experiment caches are part of the demo.

Daily Workflow

• /create-task — scaffold a new task folder from a suggestion or idea.
• /execute-task <task_id> — run a task through all mandatory stages and open its PR.
• /human-brainstorm — review project state and pick follow-up tasks from suggestions.json.
• Regenerate the aggregated dashboard with uv run python -u arf/scripts/overview/materialize.py; the output lands in overview/ and is committed alongside the change that produced it.

Key Rules

• Every CLI call is wrapped in uv run python -m arf.scripts.utils.run_with_logs <cmd> so logs are captured.
• Tasks only modify files inside their own folder. The only top-level files a task may touch are pyproject.toml, uv.lock, ruff.toml, and .gitignore.
• Every task stage and every action is a separate, well-described commit.
• Completed task folders are immutable. Fix mistakes via correction files in a new task, never by editing past folders.
• Read through aggregators, never walk task folders directly. Raw globs miss the corrections overlay. This is the single hardest rule to follow.
• Metrics must be registered in meta/metrics/ before a task reports them. Unregistered metrics fail verification.
• No human intervention for anything a script can resolve (merge conflicts, log generation, etc.). Genuine blockers produce a file in the task's intervention/ folder.

Project Structure

arf/            Framework code: scripts, skills, specifications, styleguide, docs, tests
meta/           Project metadata: asset_types/, categories/, metrics/, task_types/
tasks/          One folder per research task (created by the create-task skill)
overview/       Materialized aggregator dashboard (regenerated, committed)
project/        Project-level files: description.md, budget.json
.claude/        Claude Code config (settings.json, rules/, skills/ symlinks)
.codex/         Codex CLI config (agents/, skills/ symlinks)
CLAUDE.md       Project overview loaded at Claude Code session start
pyproject.toml  Python deps and tooling config
doctor.py       Environment validation script
ruff.toml       Lint config
LICENSE         Apache License 2.0
NOTICE          Copyright notice
CITATION.cff    Academic citation metadata

Key files for new readers: project/description.md (full project brief), project/budget.json (budget and authorized services), CLAUDE.md (agent context loaded at session start), arf/README.md (framework architecture and glossary).

Categories

• cefr-assessment — CEFR Assessment
• concept-dictionary — Concept Dictionary
• error-analysis — Error Analysis
• evaluation-methodology — Evaluation Methodology
• llm-few-shot — LLM Few-Shot
• pseudo-labeling — Pseudo-Labeling
• text-readability — Text Readability
• transformer-models — Transformer Models

Metrics

• latency_inference_seconds_per_passage — Inference latency per passage (seconds)
• mae_test — MAE on Ace-CEFR test (unitless)
• mse_test — MSE on Ace-CEFR test (unitless)
• mse_test_ci_high_90 — MSE test 90% CI high (unitless)
• mse_test_ci_low_90 — MSE test 90% CI low (unitless)
• qwk_test — Quadratic-weighted kappa test (unitless)
• spearman_test — Spearman correlation on test (unitless)
• training_time_seconds — Training wall-clock seconds (seconds)

Task Types

All task types below are generic ARF template types; this project has not added any project-specific task types.

• answer-question — Answer Question
• baseline-evaluation — Baseline Evaluation
• brainstorming — Brainstorming
• build-model — Build/Train Model
• code-reproduction — Code Reproduction
• comparative-analysis — Comparative Analysis
• correction — Correction
• data-analysis — Data Analysis
• deduplication — Deduplication
• download-dataset — Download Dataset
• download-paper — Download Paper
• experiment-run — Experiment Run
• feature-engineering — Feature Engineering
• infrastructure-setup — Infrastructure Setup
• internet-research — Internet Research
• literature-survey — Literature Survey
• write-library — Write Library

Budget and Services

Total budget: 500.00 USD. Available services: openai_api, anthropic_api, vast_ai. Full budget and service authorization in project/budget.json.

Artifacts

The repo intentionally publishes research artifacts rather than replacing them with stubs. See ARTIFACT_LICENSES.md for the generated inventory of datasets, paper PDFs, model weights, predictions, libraries, and their recorded license/access metadata.

Documentation

• Autonomy and Safety
• Tutorials
• Reference
• Python style guide
• Markdown style guide
• Agent instructions style guide

License

Original code and documentation are released under the Apache License 2.0. Copyright 2024-2026 Glite Tech Ltd. See NOTICE for attribution details.

Research artifacts retain their own licenses and access terms. Some are non-commercial, mixed, research-only, proprietary, or not recorded in upstream metadata; review ARTIFACT_LICENSES.md before reusing artifacts outside this demo.