INT8 quantization of YOLOv8n costs 1.16 points of mAP@0.50 (1.9% relative) for a 3.51x smaller model — and on 94 evaluation images that difference is not statistically resolvable. The 95% bootstrap intervals overlap. The defensible claim is therefore the weaker and more useful one: INT8 costs this model little enough accuracy that the evaluation set cannot distinguish it from FP32.
This repo is deliberately narrow. It answers one question — what does INT8 quantization cost in mAP, and what could not be verified without a Jetson? — on CPU, using ONNX Runtime static quantization, on a 126-image subset of COCO128. It is not a Jetson benchmark, not a tracking implementation, and not a general detection framework. No Jetson hardware was available at any point during this work, so every deployment-side claim that would normally accompany edge-ML work is absent rather than estimated: there is no latency, power, or thermal figure anywhere in this repository, measured or quoted.
That gap is the second half of the deliverable, not an apology for it. See
configs/tensorrt/README.md for the specific
enumeration of what a Jetson would have been needed to answer.
| metric | FP32 | INT8 | drop |
|---|---|---|---|
| mAP@0.50 | 59.60 | 58.44 | 1.16 pp |
| mAP@[0.50:0.95] | 44.37 | 42.88 | 1.49 pp |
| model size | 12.82 MB | 3.65 MB | 3.51x smaller |
| model | mAP@0.50 | 95% bootstrap interval |
|---|---|---|
| FP32 | 59.60 | 56.23 – 68.56 |
| INT8 | 58.44 | 54.74 – 66.86 |
Full provenance — date, commit, model snapshot, opset, calibration split,
seed, and reproduce command — is in
results/int8-vs-fp32.md, which is generated by
scripts/generate_results.py. No number in
this README was typed by hand; CI regenerates the results and fails the
build if they drift.
The script also asserts its own claims and exits non-zero when they break — which is not decorative. It caught the one genuinely dangerous bug in this repo (below).
uv sync --extra dev
uv run pytest -q
uv run python scripts/fetch_data.py # 6.9 MB, COCO128
uv run python scripts/generate_results.py # ~10 min on 24 CPU coresThe pitch was Jetson edge deployment. There was no Jetson. The interesting question then becomes which half of the work survives that, and how to present the other half without implying it happened.
The accuracy half survives intact: quantization error is a property of the weights and activation distributions, not of the device, so measuring it on a CPU is legitimate. The deployment half does not survive at all, and the honest response is to ship its configuration clearly marked unexecuted and to enumerate precisely what remains unknown.
Both bugs below were found by the repo's own checks, which is the argument for having written them.
Whole-graph INT8 scored mAP exactly 0.000. The results script's sanity assertion refused to publish it. YOLOv8's classification branch ends in a Sigmoid, so its outputs are probabilities in [0, 1] — and on a real image the distribution is extremely sparse: the maximum class score was 0.90, but only 0.15% of 672,000 scores exceeded a single u8 step. One scale spanning [0, 1] rounds every score, true detections included, into the zero bucket. Box coordinates span 0–640 densely and quantize without trouble. Holding the head's activation-shaping ops in float is what production INT8 detectors do, and how TensorRT behaves via per-layer precision fallback.
Without that assertion this repo would have published "INT8 destroys this model" — dramatic, memorable, and wrong.
The head decoder silently mis-oriented small inputs. Writing post-processing tests surfaced a transpose heuristic correct for the real (84, 8400) tensor and wrong for any input with fewer candidates than channels, reading box coordinates out of the class-score rows. It never affected a measured number, but confident plausible output with no exception is the expensive failure mode.
ADR 0001 — ONNX INT8 and what I could not verify records the judgement: why the repo is split in half, why Jetson numbers were not quoted from datasheets, why renting cloud GPU time was rejected as producing numbers that look like the answer while describing different hardware, and why mAP was written from scratch rather than imported.
Things this repo does not establish:
- That this model is fast on an edge device. The original point of the work, and entirely unestablished. No Jetson was available.
- That the measured 1.16 pp is the true cost. The bootstrap intervals overlap; 94 images cannot resolve a difference this small. A different 94-image draw would give a different point estimate.
- That TensorRT's INT8 calibration agrees with ONNX Runtime's. Different calibrators over different layer-fusion boundaries. The number will differ; by how much is unmeasured.
- Anything about absolute accuracy. COCO128 comes from COCO train2017, which is YOLOv8n's training data, so the absolute mAP figures are optimistic and must not be compared to a published val2017 leaderboard. Only the FP32-to-INT8 difference is defensible, because both sides see identical images through identical code.
- That the calibration split is clean. COCO128 has no room for a properly held-out set. Calibration uses the last 32 images and evaluation the first 94 — disjoint, but from the same small pool. Any residual leakage biases INT8 accuracy upward, so the true cost is at least what is reported.
- That every operator survives INT8. 71 detection-head ops are held in float. The reported cost is the cost of quantizing the backbone and neck.
- Tracking, SLAM, sensor fusion, or camera calibration. None implemented. These exist as written notes — deliberately notes rather than stub code, because a plausible-looking SLAM module would imply a capability never demonstrated.
src/edge_detection_tracking_jetson_notes/
metrics.py IoU, greedy matching, all-points AP — pure numpy
dataset.py COCO128 loader, YOLO-format label conversion
infer.py letterbox, ONNX Runtime session, NMS, YOLOv8 head decode
export.py PyTorch -> ONNX, static shapes, pinned opset
quantize.py static QDQ INT8 + the head-exclusion rule
evaluate.py scoring both precisions through identical code
tests/ 70 tests; the mAP fixtures are hand-computed on paper
scripts/ fetch_data.py, generate_results.py
configs/tensorrt/ the Jetson path, shipped UNEXECUTED
notes/ SLAM, sensor fusion, camera calibration — written notes
docs/adr/ the judgement artifact
The mAP maths is pure numpy with no pycocotools dependency, and every fixture
in tests/test_metrics.py was computed by hand with the arithmetic shown in
the test docstring. The entire finding is a difference between two mAP
numbers, so a subtly wrong evaluator would not look wrong — it would quietly
move the answer. Capturing a first run's output as the expected value would
have produced a suite that passes forever while asserting nothing.
See docs/inventory-coverage.md. Detection,
IoU/mAP, NMS, anchor-free heads, ONNX export and INT8 calibration are
exercised in code; camera geometry, sensor fusion, SLAM and the TensorRT/
Jetson material are written notes and unexecuted configuration, labelled as
such throughout.
MIT