diff --git a/docs/source/guide/dashboard_data_quality.md b/docs/source/guide/dashboard_data_quality.md new file mode 100644 index 000000000000..f4b09aa598cf --- /dev/null +++ b/docs/source/guide/dashboard_data_quality.md @@ -0,0 +1,76 @@ +--- +title: Data quality dashboard +short: Data quality dashboard +tier: enterprise +type: guide +order: 0 +order_enterprise: 71 +meta_title: Data quality dashboard +meta_description: Use the Data quality dashboard to analyze agreement, confusion, and dimension-level quality for a project. +section: "Project Management" +parent: "dashboards" +parent_enterprise: "dashboards" +date: 2026-06-29 00:00:00 +--- + +> Investigate project data quality with agreement metrics, dimension-level comparisons, and confusion analysis. + +The **Data quality** dashboard helps you understand how consistently tasks are labeled and where annotations or labels disagree. Use it to find ambiguous tasks, confusing labels, and dimensions that need clearer instructions or review. + +For velocity and time spent, see the [Throughput dashboard](dashboard_throughput). For member-level performance, see the [Members dashboard](dashboard_members). + +## Access the dashboard + +From the project, open the **Dashboard** tab and select **Data quality** from the dashboard navigation. + +The Data quality dashboard has two tabs: + +- **Agreement analysis**: Review agreement metrics and confusion across dimensions. +- **Label distribution**: Compare annotation and prediction label counts by dimension. + +## Agreement analysis + +Use the **Agreement analysis** tab to understand where annotators, models, and ground truth annotations align or disagree. + +### Average Task Agreement + +The **Average Task Agreement** KPI card shows the average agreement score for tasks in the project. + +Agreement is most useful when tasks have multiple annotations or a comparison source such as ground truth. Tasks with only one annotation do not provide an annotator-to-annotator agreement signal. + +### Overall Agreement Charts + +| Chart | Description | +|-------|-------------| +| **Task Agreement Distribution** | Histogram of task-level agreement scores.

More tasks in higher ranges indicate stronger labeling consistency; more tasks in lower ranges suggest ambiguous instructions, difficult examples, or labels that need clarification. Clicking on a bar opens up the tasks tasks in the Data Manager. | +| **Agreement by Dimension** | Agreement score per dimension.

Use it to identify which dimensions are driving disagreement issues. | + +### Ground Truth Agreement + +| Chart | Description | +|-------|-------------| +| **Top Confusion Pairs** | Table of labels with the most number of incorrect values applied according to the ground truth.

Use it to find label pairs that need better examples, clearer definitions, or additional training. | +| **Confusion Matrix** | Heatmap comparing labels or values within a selected dimension.

Use the dimension dropdown to choose which dimension to inspect (default: All dimensions). | + +## Label distribution + +Use the **Label distribution** tab to compare values from annotations and predictions for each dimension. + +Each supported dimension has a card with: + +- **Average Agreement**: Agreement for the dimension when there is enough data to compare annotations or predictions. +- **Label Distribution chart**: A grouped horizontal bar chart comparing values from annotations and predictions. +- **Label Distribution table**: Counts from annotations and predictions for each value, plus a **Total** row. Values in the **From Annotations** and **From Predictions** columns link to the Data Manager with filters applied for the selected dimension and value. + +For more detail about supported dimensions and Label Distribution behavior, see the [Label Distribution dashboard](dashboard_distribution). + +## Interpreting data quality issues + +Low agreement or high confusion does not always mean poor annotator or model performance. It can also indicate: + +- Ambiguous task data. +- Labels that overlap conceptually. +- Missing or unclear labeling instructions. +- A project configuration that requires higher annotation overlap or [additional review strategy](quality). + +Use the Data quality dashboard together with the [Members dashboard](dashboard_members) and Data Manager to decide whether to update instructions, add examples, reassign tasks, or coach specific members. diff --git a/docs/source/guide/dashboard_distribution.md b/docs/source/guide/dashboard_distribution.md index f8bff3deccca..b42c4122d670 100644 --- a/docs/source/guide/dashboard_distribution.md +++ b/docs/source/guide/dashboard_distribution.md @@ -6,16 +6,16 @@ type: guide order: 0 order_enterprise: 175 meta_title: Label distribution dashboard -meta_description: Use the label distribution dashboard to track the distribution of labels in a project. +meta_description: Use the Label Distribution dashboard to compare annotation and prediction label distributions by project dimension. section: "Analytics" -date: 2024-04-04 11:50:41 +date: 2026-06-29 00:00:00 --- -The label distribution dashboard provides a visual representation of the label distribution in a project over time. +The **Analytics > Label Distribution** dashboard shows how annotation and prediction values are distributed across a project. Use it to identify imbalanced labels, compare annotator and model outputs, and drill into the tasks behind a label value. ![Screenshot of Label distribution dashboard](/images/analytics/label-dist.png) -#### Access by user role +## Access by user role | User role | Access restrictions | |-----------|---------------------| @@ -23,33 +23,44 @@ The label distribution dashboard provides a visual representation of the label d | **Manager** | Can view the **Label Distribution** dashboard, but can only filter for and view projects in which they are a member. | | **Reviewer** and **Annotator** | Cannot access the **Label Distribution** dashboard.

When they open **Analytics**, they are redirected to the [Member Performance dashboard](dashboard_annotator), where they can only see their own annotation and review history. | -## Supported control tags +## Label distribution by dimension -The label distribution dashboard supports control tags that have nested child tags. +The dashboard groups results by labeling dimension. A dimension can come from a traditional labeling config control tag or from an Interface output, such as ReactCode or CustomInterface fields. -For example, tags such as ``, ``, `` are supported. +Each supported dimension has a card with: -But tags such as `