Skip to content

docs: Project Dashboard: Updated docs #9784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hakan458 merged 3 commits into from
Jul 1, 2026
+305 −211
Merged

docs: Project Dashboard: Updated docs #9784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e63a269
updated members page
shereezhang Jun 17, 2026
1ab6ac1
update dashboards docs (no screenshots)
Jun 30, 2026
5f58b78
dashboard docs copy edits and re-organization
shereezhang Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions docs/source/guide/dashboard_data_quality.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
title: Data quality dashboard
short: Data quality dashboard
tier: enterprise
type: guide
order: 0
order_enterprise: 71
meta_title: Data quality dashboard
meta_description: Use the Data quality dashboard to analyze agreement, confusion, and dimension-level quality for a project.
section: "Project Management"
parent: "dashboards"
parent_enterprise: "dashboards"
date: 2026-06-29 00:00:00
---

> Investigate project data quality with agreement metrics, dimension-level comparisons, and confusion analysis.

The **Data quality** dashboard helps you understand how consistently tasks are labeled and where annotations or labels disagree. Use it to find ambiguous tasks, confusing labels, and dimensions that need clearer instructions or review.

For velocity and time spent, see the [Throughput dashboard](dashboard_throughput). For member-level performance, see the [Members dashboard](dashboard_members).

## Access the dashboard

From the project, open the **Dashboard** tab and select **Data quality** from the dashboard navigation.

The Data quality dashboard has two tabs:

- **Agreement analysis**: Review agreement metrics and confusion across dimensions.
- **Label distribution**: Compare annotation and prediction label counts by dimension.

## Agreement analysis

Use the **Agreement analysis** tab to understand where annotators, models, and ground truth annotations align or disagree.

### Average Task Agreement

The **Average Task Agreement** KPI card shows the average agreement score for tasks in the project.

Agreement is most useful when tasks have multiple annotations or a comparison source such as ground truth. Tasks with only one annotation do not provide an annotator-to-annotator agreement signal.

### Overall Agreement Charts

| Chart | Description |
|-------|-------------|
| **Task Agreement Distribution** | Histogram of task-level agreement scores.<br /><br /> More tasks in higher ranges indicate stronger labeling consistency; more tasks in lower ranges suggest ambiguous instructions, difficult examples, or labels that need clarification. Clicking on a bar opens up the tasks tasks in the Data Manager. |
| **Agreement by Dimension** | Agreement score per dimension.<br /><br /> Use it to identify which dimensions are driving disagreement issues. |

### Ground Truth Agreement

| Chart | Description |
|-------|-------------|
| **Top Confusion Pairs** | Table of labels with the most number of incorrect values applied according to the ground truth.<br /><br />Use it to find label pairs that need better examples, clearer definitions, or additional training. |
| **Confusion Matrix** | Heatmap comparing labels or values within a selected dimension. <br /><br />Use the dimension dropdown to choose which dimension to inspect (default: All dimensions). |

## Label distribution

Use the **Label distribution** tab to compare values from annotations and predictions for each dimension.

Each supported dimension has a card with:

- **Average Agreement**: Agreement for the dimension when there is enough data to compare annotations or predictions.
- **Label Distribution chart**: A grouped horizontal bar chart comparing values from annotations and predictions.
- **Label Distribution table**: Counts from annotations and predictions for each value, plus a **Total** row. Values in the **From Annotations** and **From Predictions** columns link to the Data Manager with filters applied for the selected dimension and value.

For more detail about supported dimensions and Label Distribution behavior, see the [Label Distribution dashboard](dashboard_distribution).

## Interpreting data quality issues

Low agreement or high confusion does not always mean poor annotator or model performance. It can also indicate:

- Ambiguous task data.
- Labels that overlap conceptually.
- Missing or unclear labeling instructions.
- A project configuration that requires higher annotation overlap or [additional review strategy](quality).

Use the Data quality dashboard together with the [Members dashboard](dashboard_members) and Data Manager to decide whether to update instructions, add examples, reassign tasks, or coach specific members.
49 changes: 30 additions & 19 deletions docs/source/guide/dashboard_distribution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,50 +6,61 @@ type: guide
order: 0
order_enterprise: 175
meta_title: Label distribution dashboard
meta_description: Use the label distribution dashboard to track the distribution of labels in a project.
meta_description: Use the Label Distribution dashboard to compare annotation and prediction label distributions by project dimension.
section: "Analytics"
date: 2024-04-04 11:50:41
date: 2026-06-29 00:00:00
---

The label distribution dashboard provides a visual representation of the label distribution in a project over time.
The **Analytics > Label Distribution** dashboard shows how annotation and prediction values are distributed across a project. Use it to identify imbalanced labels, compare annotator and model outputs, and drill into the tasks behind a label value.

![Screenshot of Label distribution dashboard](/images/analytics/label-dist.png)

#### Access by user role
## Access by user role

| User role | Access restrictions |
|-----------|---------------------|
| **Owner** and **Admin** | Can view the **Label Distribution** dashboard and filter for all projects. |
| **Manager** | Can view the **Label Distribution** dashboard, but can only filter for and view projects in which they are a member. |
| **Reviewer** and **Annotator** | Cannot access the **Label Distribution** dashboard. <br/><br/>When they open **Analytics**, they are redirected to the [Member Performance dashboard](dashboard_annotator), where they can only see their own annotation and review history. |

## Supported control tags
## Label distribution by dimension

The label distribution dashboard supports control tags that have nested child tags.
The dashboard groups results by labeling dimension. A dimension can come from a traditional labeling config control tag or from an Interface output, such as ReactCode or CustomInterface fields.

For example, tags such as `<Choices>`, `<Labels>`, `<RectangleLabels>` are supported.
Each supported dimension has a card with:

But tags such as `<TextArea>`, `<Rating>`, and `<Number>` are not supported.
- **Average Agreement**: Agreement for the dimension when there is enough data to compare annotations, predictions, or ground truth.
- **Label Distribution chart**: A grouped horizontal bar chart comparing values from annotations and predictions.
- **Label breakdown table**: Counts from annotations and predictions for each value.

!!! note
Taxonomy tags with nested `Choice` tags are supported, but not [taxonomies defined with an external source](/templates/taxonomy#Taxonomies-defined-using-a-remote-source).
![Screenshot of Label distribution dashboard](/images/analytics/label-dist-charts.png)

## Label breakdown table

## Label summaries and charts
The label breakdown table shows each value for the selected dimension, plus a **Total** row.

At the top of the Label distribution dashboard, summary cards show how many annotations and predictions are in the project.
| Column | Description |
|--------|-------------|
| **Label** | Label or value name. The final row shows the total across all values. |
| **From Annotations** | Number of times the value appears in annotations. |
| **From Predictions** | Number of times the value appears in predictions. |

The charts show distribution of labels from annotations and from predictions.
Values in the **From Annotations** and **From Predictions** columns can link to the Data Manager with filters applied for the selected project, dimension, value, and source. Use these links to inspect the underlying tasks, annotations, or predictions.

![Screenshot of Label distribution dashboard](/images/analytics/label-dist-charts.png)
![Screenshot of Label distribution table](/images/analytics/label-dist-tables.png)

You can click the overflow menu next to each chart to view it as a pie chart or to export the data as a CSV file.
## Supported dimensions

![Screenshot of Label distribution dashboard](/images/analytics/label-dist-menu.png)
Label distribution supports dimensions with a finite set of values, including:

## Label distribution tables
- Label controls such as `<Labels>`, `<RectangleLabels>`, and other labels-based controls.
- Choice controls such as `<Choices>`.
- Taxonomy controls with finite, nested choices.
- Enum-backed ReactCode and CustomInterface dimensions.
- Other finite scalar dimensions that can be represented as a known set of values.

The label distribution table shows a detailed distribution of labels from annotations and from predictions, shown as total count and as a percentage relative to the total number of annotations/predictions.
Dimensions that do not have a finite set of values are not shown as label distributions. This includes free text, arbitrary numeric or date values, objects, arrays, and values that are not declared in the dimension schema.

![Screenshot of Label distribution table](/images/analytics/label-dist-tables.png)
!!! note
If a dimension is unsupported, the dashboard should show that it is unsupported rather than rendering an empty chart. This helps distinguish "no data" from data that cannot be safely summarized as a label distribution.

Loading
Loading