feat: add ECViT LTDETR backbone wrapper by gabrielfruet · Pull Request #772 · lightly-ai/lightly-train

gabrielfruet · 2026-06-10T17:25:54Z

What has changed and why?

Ports EdgeCrafter's ECViT backbone + ViTAdapter into Lightly as a standalone
LTDETR-compatible backbone wrapper. This is the backbone port only; wiring into
DINOv3LTDETRObjectDetection and end-to-end integration will be done in a
follow-up PR.

Highlights:

Adds ECViTWrapper at
src/lightly_train/_task_models/dinov3_ltdetr_object_detection/ecvit_wrapper.py.
Contract: forward(x) -> tuple[Tensor, Tensor, Tensor], matching the current
LTDETR backbone interface.
Preserves EdgeCrafter behavior: ECViT tokens -> selected layers -> average
/fuse -> reshape to spatial map -> interpolate to 3 levels -> project
channels -> return (P3, P4, P5).
Supports all EdgeCrafter detection presets: ecvitt, ecvittplus,
ecvits, ecvitsplus.
Loads original ECViT backbone checkpoints directly (no DINOv3 adaptation).
Does not use DINOv3STAs or any DINOv3 weight key conversion.

How has it been tested?

Added
tests/_task_models/dinov3_ltdetr_object_detection/test_ecvit_wrapper.py:

All four presets instantiate and forward returns (P3, P4, P5) with the
expected channels and spatial sizes.
A focused test asserts selected layers are averaged before being reshaped
to a spatial map.
weights_path loads backbone state dicts strictly, including
state_dict / model / backbone checkpoint containers.
Invalid name and num_levels != 3 raise the expected errors.
Optional real-checkpoint test (skipped by default) loads official
EdgeCrafter .pth files when ECVIT_CHECKPOINT_DIR is set.

Validated locally:

PYTHONPATH=src python -m ruff check \
  src/lightly_train/_task_models/dinov3_ltdetr_object_detection/ecvit_wrapper.py \
  tests/_task_models/dinov3_ltdetr_object_detection/test_ecvit_wrapper.py
python -m compileall \
  src/lightly_train/_task_models/dinov3_ltdetr_object_detection/ecvit_wrapper.py
PYTHONPATH=src pytest \
  tests/_task_models/dinov3_ltdetr_object_detection/test_ecvit_wrapper.py -q
# 11 passed, 4 skipped

Did you update CHANGELOG.md?

Yes
Not needed (internal change)

The wrapper is not yet wired into a task model and is not exposed to users.

Did you update the documentation?

Yes
Not needed (internal change without effects for user)

No user-facing surface changes in this PR. Documentation will be updated when
the wrapper is wired into DINOv3LTDETRObjectDetection in the follow-up PR.

gabrielfruet · 2026-06-11T12:30:15Z

@codex review

chatgpt-codex-connector · 2026-06-11T12:34:02Z

Codex Review: Didn't find any major issues. Swish!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gabrielfruet · 2026-06-11T13:00:49Z

/review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ddad28aae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-11T13:03:48Z

+    def forward(self, *, H: int, W: int) -> tuple[Tensor, Tensor]:
+        periods = cast(Tensor, self.periods)
+        device = periods.device
+        dtype = self.dtype if self.dtype is not None else torch.get_default_dtype()


Preserve RoPE dtype under mixed precision

When this backbone is run with AMP or after model.half(), the RoPE buffer is moved to fp16 but this line still creates coordinates, sin, and cos in torch.get_default_dtype() (usually fp32). apply_rope then converts only q/k to fp32 while v remains fp16, so scaled_dot_product_attention(q, k, v, ...) receives mixed dtypes and can fail in the mixed-precision LTDETR training/inference path. Derive the generated RoPE dtype from the buffer/input dtype instead of the global default.

Useful? React with 👍 / 👎.

Copilot

Pull request overview

Adds an EdgeCrafter ECViT backbone + ViTAdapter port as a standalone LTDETR-compatible backbone wrapper in lightly_train, along with licensing/NOTICE updates and unit tests. This sets up ECViT feature pyramid outputs (P3, P4, P5) for later integration into DINOv3LTDETRObjectDetection.

Changes:

Introduces ECViTWrapper implementing the LTDETR backbone interface and checkpoint loading/unwrapping.
Adds pytest coverage for presets, output shapes, layer-fusion behavior, and strict checkpoint loading.
Updates third-party licensing attribution (NOTICE + Apache 2.0 license header tooling + license text).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`src/lightly_train/_task_models/dinov3_ltdetr_object_detection/ecvit_wrapper.py`	New ECViT backbone wrapper + supporting modules + checkpoint loading utilities.
`tests/_task_models/dinov3_ltdetr_object_detection/test_ecvit_wrapper.py`	Unit tests for wrapper instantiation, forward output contract, fusion behavior, and weight loading.
`NOTICE`	Adds EdgeCrafter attribution and modification notes.
`Makefile`	Excludes the new EdgeCrafter-derived file from the default header step and applies an Apache 2.0 header template.
`licences/EDGECRAFTER_LICENSE`	Adds the Apache 2.0 license text for EdgeCrafter.
`dev_tools/edgecrafter_licenseheader.tmpl`	Adds the Apache 2.0 license header template for EdgeCrafter-derived files.

+            torch.Size([2, expected_channels, 2, 2]),
+            torch.Size([2, expected_channels, 1, 1]),
+        ]
+


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

liopeer

Looks clean and working. I would move it to the _models subpackage and properly implement the ModelWrapper in follow-up PRs.

liopeer · 2026-06-12T07:46:22Z

+
+# EdgeCrafter calls this module ViTAdapter. Keep the alias for familiarity while the
+# Lightly-facing name describes the LTDETR backbone wrapper contract.
+ViTAdapter = ECViTWrapper


Unless this is actually required somewhere I would discard it.

liopeer · 2026-06-12T08:07:13Z

@@ -0,0 +1,596 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");


Are you planning to move it to a package inside _models?

Yeah, I would write a package in a follow-up PR . I will move it to _models since is more suitable.

liopeer · 2026-06-12T08:15:30Z

+        state_dict = _unwrap_state_dict(state)
+        self.backbone.load_state_dict(state_dict, strict=True)
+
+    def forward(self, x: Tensor) -> tuple[Tensor, Tensor, Tensor]:


Note: If you're planning to move it to a package (which i would), you should implement the interfaces in the src/lightly_train/_models/model_wrapper.py module.

gabrielfruet added 5 commits June 10, 2026 14:22

feat: add ECViT LTDETR backbone wrapper

33e84a0

chore: add EdgeCrafter licensing and notice

759c292

feat: apply EdgeCrafter license header and type fixes

0f565e6

fix: use typing_extensions.TypeAlias and Union for 3.8

fca39f9

fix: typing errors

40e4e85

test: tighten ecvit wrapper coverage

a6dee97

gabrielfruet marked this pull request as ready for review June 11, 2026 13:00

Merge branch 'main' into gabriel-trn-2143-port-ecvit-to-lightlytrain

4ddad28

Copilot AI review requested due to automatic review settings June 11, 2026 13:00

Copilot started reviewing on behalf of gabrielfruet June 11, 2026 13:00 View session

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Copilot AI reviewed Jun 11, 2026

View reviewed changes

gabrielfruet and others added 2 commits June 11, 2026 10:36

Apply suggestions from code review

a1f94ae

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

refactor: reuse dinov3 ecvit layers

a0f0a29

liopeer approved these changes Jun 12, 2026

View reviewed changes

gabrielfruet and others added 2 commits June 12, 2026 09:40

feat: move ecvit wrapper to models

51d07fe

Merge branch 'main' into gabriel-trn-2143-port-ecvit-to-lightlytrain

5c97bce

gabrielfruet mentioned this pull request Jun 12, 2026

feat: incorporate ECViT backbones into DINOv3 LTDETR #778

Draft

liopeer mentioned this pull request Jun 12, 2026

Any backbone for linear segmentation #777

Open

4 tasks

		@@ -0,0 +1,596 @@
		#
		# Licensed under the Apache License, Version 2.0 (the "License");

Conversation

gabrielfruet commented Jun 10, 2026

What has changed and why?

How has it been tested?

Did you update CHANGELOG.md?

Did you update the documentation?

Uh oh!

gabrielfruet commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

gabrielfruet commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liopeer left a comment

Choose a reason for hiding this comment

Uh oh!

liopeer Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

liopeer Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

gabrielfruet Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

liopeer Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

gabrielfruet Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants