Aerial Perception Stack

An end-to-end, real-time aerial perception project focused on what happens when computer vision leaves a clean benchmark and becomes part of a deployed system.

Status: dataset preparation, validation, CUDA training, area-sliced detection analysis, the frozen detector candidate, and a first deployment benchmark matrix are implemented. MPS, ONNX, and Core ML plumbing smoke tests pass; the ONNX provenance/parity tooling is implemented but its strict parity criterion currently fails, while full ByteTrack integration, TensorRT validation, and end-to-end runtime benchmark results remain pending.

Objective

Build and evaluate a lightweight perception stack for UAV video:

UAV video
   │
   ▼
decode + input-quality signals
   │
   ▼
object detector (YOLO11n baseline)
   │ detections
   ▼
multi-object tracker (ByteTrack)
   │ tracks
   ▼
calibrated confidence + track health
   │
   ├── annotated video
   ├── runtime telemetry
   └── reproducible benchmark results

The central question is:

How much detection and tracking quality does a lightweight aerial perception stack retain when video quality degrades, and which interventions improve the robustness/accuracy/latency trade-off on edge hardware?

This is deliberately more than a detector demo. The project covers data validation, detection, multi-object tracking, robustness testing, confidence calibration, runtime monitoring, export, and failure analysis.

V1 scope

• Fine-tune one lightweight detector on VisDrone-DET.
• Track objects in VisDrone-MOT video with ByteTrack.
• Evaluate clean and corrupted inputs with fixed seeds and severities.
• Report detection, tracking, calibration, and runtime metrics.
• Export the detector to ONNX and Core ML.
• Run a sustained end-to-end benchmark on a MacBook Air M5.
• Produce an annotated demo video, an HTML benchmark report, and a failure gallery.

The first version intentionally excludes model ensembling, RT-DETR comparisons, multiple trackers, and a web platform. Native TensorRT execution remains optional and is only worth adding after the baseline is reproducible and NVIDIA hardware is available.

Target platform

Development and primary benchmarking are performed on a fanless MacBook Air M5 with 32 GB of unified memory.

An Ubuntu workstation with an RTX 3090 may also become available when additional compute is justified. It would be used for longer CUDA training runs and native TensorRT validation, not to replace the Mac as the primary edge-runtime target. Every published result must identify its hardware and runtime so that MPS and CUDA measurements are never mixed.

Purpose	Format/runtime
Training and reference evaluation	PyTorch with MPS
Optional accelerated experiments	PyTorch/CUDA on Ubuntu with RTX 3090, if available
Portable model artifact	ONNX
Native Apple Silicon deployment	Core ML
Optional cross-language runtime	C++ with OpenCV and ONNX
Optional NVIDIA deployment	TensorRT on Ubuntu/RTX 3090 or Linux/NVIDIA CI

TensorRT requires NVIDIA/CUDA hardware and is therefore not a local runtime target on the Mac. Core ML remains the relevant optimized deployment target for the primary machine; TensorRT results will only be reported if they are actually validated on compatible hardware or explicitly marked as CI-only.

Dataset

VisDrone is the primary dataset. The full benchmark contains 288 video clips, 261,908 video frames, 10,209 static images, and more than 2.6 million annotated bounding boxes. It includes aerial viewpoints, dense scenes, small objects, occlusion, and varied lighting and weather.

Required local splits

Split	Expected contents
VisDrone-DET train	6,471 images
VisDrone-DET val	548 images
VisDrone-DET test-dev	1,610 images
VisDrone-MOT val	7 sequences / 2,846 frames
VisDrone-MOT train	56 sequences / 24,201 frames

This repository does not download or redistribute VisDrone. The setup below assumes these five splits have already been downloaded and extracted.

Disk budget

Storage category	MVP estimate	Full-project estimate
Source archives, if retained	3.3 GB	10.8 GB
Extracted datasets and converted labels	5-8 GB	15-25 GB
Checkpoints, exports, logs, and metrics	3-10 GB	5-15 GB
Selected corruption caches and demo videos	5-15 GB	15-50 GB
Recommended free space	at least 25 GB	60-100 GB

The extracted and artifact figures are planning estimates, not official dataset sizes. Corruptions will normally be generated on demand. The project must not materialize every corruption and severity as a full copy of the dataset, which could multiply storage requirements by more than 20.

Storage configuration

Choose any writable directory with enough free space and expose it through AERIALTRACK_DATA_DIR. The repository, virtual environment, and package cache can remain separate from generated data:

$AERIALTRACK_DATA_DIR/
├── datasets/
├── cache/
├── runs/
├── models/
└── demos/

Dataset paths are not hardcoded in the source. Raw data, model weights, caches, and run outputs are excluded from Git.

Robustness benchmark

Input corruptions

Each corruption will have fixed parameters for mild, medium, and severe levels:

• motion blur;
• low light and sensor-like noise;
• abrupt exposure change;
• synthetic haze/fog;
• H.264 compression;
• dropped detector frames while preserving the video timeline;
• camera shake.

Real-data analysis slices

Small objects and occlusion are treated primarily as annotated data slices, not as synthetic corruptions:

• small and very small bounding boxes;
• partial and heavy occlusion;
• truncation;
• scene density;
• object class.

All benchmark configurations, severity parameters, sequence manifests, and random seeds must be versioned. Evaluation data must remain separate from data used to tune tracker thresholds or augmentation policies.

Metrics

Area	Primary metrics
Detection	mAP50-95, AP50, AP-small, recall
Tracking	HOTA, DetA, AssA, IDF1, ID switches, fragmentations
Calibration	calibration error, reliability diagram
Runtime	end-to-end FPS, p50/p95 latency, per-stage latency, dropped frames, memory
Robustness	corrupted/clean metric retention by severity

TrackEval will provide the reference implementation for HOTA and related tracking metrics.

The runtime benchmark will include a continuous ten-minute run so that reported performance reflects sustained behavior on a fanless laptop, not only a short warm benchmark.

Initial clean detection baseline

YOLO11n was fine-tuned for five epochs on all 6,471 VisDrone-DET train images and evaluated independently on the 548-image validation split with the versioned 640 px, confidence 0.001, NMS IoU 0.7, and max_det=500 protocol.

Precision	Recall	mAP50	mAP50-95	MPS inference only
29.09%	24.53%	20.17%	11.25%	2.85 ms/image

This is a reproducible initial baseline, not a converged final model. Car AP50-95 is 40.38%, while bicycle AP50-95 is only 0.41%; small and uncommon categories remain the dominant accuracy problem. The timing figure excludes decode, tracking, rendering, and queueing and is therefore not an end-to-end FPS claim.

See the detailed baseline report and machine-readable result.

RTX 3090 CUDA replication

The same five-epoch protocol was repeated on Ubuntu with the local RTX 3090, using configs/detector/yolo11n-cuda-baseline.yaml. The independent validation used the same 640 px, confidence 0.001, NMS IoU 0.7, and max_det=500 settings as the Mac baseline.

Run	Precision	Recall	mAP50	mAP50-95	Inference
Mac MPS baseline	29.09%	24.53%	20.17%	11.25%	2.85 ms/image
RTX 3090 CUDA	28.80%	24.65%	19.95%	11.15%	1.34 ms/image

The CUDA result is within 0.10 percentage point of the Mac baseline on mAP50-95, so the Linux environment is validated for the next, longer training candidate. This five-epoch run is still underfit and is kept separate from the original Mac baseline. See the CUDA run report and machine-readable result.

RTX 3090 longer candidate

The first longer candidate trained for 50 epochs with the same 640 px input, batch size, seed, and independent validation protocol. It is the current frozen detector candidate for the next pipeline stage.

Run	Precision	Recall	mAP50	mAP50-95	Inference
Mac MPS, 5 epochs	29.09%	24.53%	20.17%	11.25%	2.85 ms/image
RTX 3090 CUDA, 5 epochs	28.80%	24.65%	19.95%	11.15%	1.34 ms/image
RTX 3090 CUDA, 50 epochs	40.82%	31.68%	29.75%	16.61%	1.13 ms/image

The 50-epoch candidate improves mAP50-95 by 5.36 percentage points over the Mac baseline. Car reaches 47.56% mAP50-95, while bicycle remains at 2.15%, so small-object performance is still the main limitation. See the long candidate report and machine-readable result.

Area slices and frozen detector

The frozen 50-epoch checkpoint was evaluated with a COCO-style area-sliced evaluator. Small objects have an annotated box area below 32² pixels, medium objects are in [32², 96²), and large objects are at least 96² pixels.

Slice	AP	AR
Small	6.99%	13.01%
Medium	24.27%	—
Large	44.85%	—

There are 26,586 small boxes out of 38,759 validation boxes. The per-class small-object AP ranges from 1.27% for bicycle to 26.03% for car, confirming that small objects are the main remaining detector weakness. The area-sliced global AP is COCO-evaluator output and should not be read as a replacement for the Ultralytics mAP50-95 number above; the two evaluators use different aggregation details.

The selected checkpoint is copied and hash-verified at $AERIALTRACK_DATA_DIR/models/frozen/yolo11n-cuda-long50.pt with SHA-256 382020095c27a7038556bc48350aeaa5df27c5808cb6abf6155efa2b404677ac. Its configuration is configs/detector/yolo11n-cuda-frozen.yaml. The area report is in reports/2026-08-22-cuda-long50-area-slices.md.

The same frozen checkpoint was then evaluated once on VisDrone-DET test-dev:

Split	Precision	Recall	mAP50	mAP50-95	Inference
Test-dev, 1,610 images	37.20%	29.22%	25.27%	13.80%	0.91 ms/image

See the test-dev report and machine-readable result.

Deployment benchmark matrix

The project now records the deployment trade-off explicitly: artifact size, precision, measured latency, FPS, memory, and target platform. The current local pass uses 256 fixed validation images, 10 warm-up images, and three measured repetitions (768 warm samples per profile) on Ubuntu. These measurements include image loading plus detector preprocessing, inference, and postprocessing; they exclude camera/video decode, tracking, rendering, and queueing.

Profile	Artifact	Platform	p50	p95	FPS	Memory	Status
PyTorch FP32	5.2 MB	RTX 3090	8.45 ms	10.09 ms	118.4	GPU 56 MB	measured
PyTorch FP16	5.2 MB*	RTX 3090	8.94 ms	10.68 ms	111.9	GPU 44 MB	measured
ONNX Runtime FP32	10.1 MB	Linux CPU	34.19 ms	37.50 ms	29.2	RSS +7 MB	measured
TensorRT FP16	—	Jetson / NVIDIA GPU	—	—	—	—	target
TensorRT INT8	—	Jetson / NVIDIA GPU	—	—	—	—	target
Core ML FP16	—	Mac / Apple Silicon	—	—	—	—	target

* FP16 uses the same serialized PyTorch checkpoint but converts the runtime weights in memory, so the file size is unchanged. The current model accuracy reference remains 16.61% mAP50-95 on VisDrone-DET val for the PyTorch checkpoint. ONNX parity is measured separately and is not yet passing the strict criterion, so the ONNX row is a runtime measurement rather than an accuracy claim. The reported FPS is 1000 / p50 latency, not sustained video throughput. TensorRT and Core ML are targets, not claimed measurements, until they are run on compatible hardware.

The benchmark is reproducible with:

uv run aerialtrack benchmark-deployment \
  --config configs/detector/yolo11n-cuda-frozen.yaml \
  --model data/models/frozen/yolo11n-cuda-long50.pt \
  --split val \
  --count 256 \
  --warmup 10 \
  --repeats 3

It writes both a machine-readable JSON report and a Markdown table under $AERIALTRACK_DATA_DIR/runs/deployment/yolo11n-cuda-frozen/. See the versioned deployment report.

The fixed-sample ONNX parity checker is separate from the speed benchmark:

uv run aerialtrack check-export-parity \
  --config configs/detector/yolo11n-cuda-frozen.yaml \
  --model data/models/frozen/yolo11n-cuda-long50.pt \
  --onnx data/models/deployment/yolo11n-cuda-long50.onnx \
  --split val \
  --count 64 \
  --confidence 0.25

The current 64-image check is intentionally recorded as a failed strict parity check: 96.03% of detections were matched at IoU 0.5, with mean matched IoU 0.977. This is close, but not sufficient to claim exported-model equivalence. See the parity report.

Track health

V1 will avoid claiming Bayesian uncertainty. Instead, it will expose a measurable track-health score based on signals such as:

• calibrated detection confidence;
• track age;
• missed observations;
• bounding-box and motion stability;
• association ambiguity.

The demo will render stable, fragile, and at-risk tracks differently. This score will only be described as a probability if calibration against observed tracking failures supports that interpretation.

Definition of done

The project is complete when:

• the environment and data setup are documented and reproducible;
• dataset conversion is validated with automated sanity checks;
• clean detection and tracking baselines are frozen;
• at least five degraded conditions are evaluated at three severities;
• one robustness intervention is compared fairly against the baseline;
• PyTorch, ONNX, and Core ML outputs are checked for parity;
• an end-to-end video pipeline reports runtime telemetry;
• results include per-class and small-object analysis plus failure cases;
• the README contains measured results, limitations, and reproducible commands;
• the repository includes a short annotated demo video or GIF.

Provisional success targets, to be revisited only after measuring the baseline:

• at least 20 end-to-end FPS for the selected real-time profile;
• p95 end-to-end latency below 50 ms;
• less than one AP point between reference and exported-model evaluation;
• at least five percentage points of mean robustness-retention improvement;
• no more than one AP point of clean-performance regression.

Local setup

The repository uses Python 3.12 and uv. Start by copying the environment template and setting an absolute path to your existing data directory:

cp .env.example .env
# Edit .env:
# AERIALTRACK_DATA_DIR=/absolute/path/to/aerialtrack-data
uv sync

Before conversion, arrange the extracted data as follows:

$AERIALTRACK_DATA_DIR/
├── datasets/
│   ├── VisDrone/
│   │   ├── VisDrone2019-DET-train/{images,annotations}/
│   │   ├── VisDrone2019-DET-val/{images,annotations}/
│   │   └── VisDrone2019-DET-test-dev/{images,annotations}/
│   └── VisDrone-MOT/
│       ├── VisDrone2019-MOT-train/{sequences,annotations}/
│       └── VisDrone2019-MOT-val/{sequences,annotations}/
├── models/pretrained/yolo11n.pt
├── cache/
├── runs/
└── demos/

Prepare DET once, then validate the complete local setup:

uv run aerialtrack data prepare-det
uv run aerialtrack data validate-det
uv run aerialtrack data validate-mot --split train
uv run aerialtrack data validate-mot --split val
uv run aerialtrack doctor

prepare-det moves DET images into the Ultralytics layout and generates YOLO labels. Skip it if your DET data is already prepared under datasets/VisDrone/{images,labels}/{train,val,test}.

Implemented commands

uv run aerialtrack doctor
uv run aerialtrack data prepare-det
uv run aerialtrack data validate-det
uv run aerialtrack data validate-mot --split train
uv run aerialtrack data validate-mot --split val
uv run aerialtrack data report
uv run aerialtrack data render-samples
uv run aerialtrack train --config configs/detector/yolo11n-baseline.yaml
uv run aerialtrack evaluate \
  --config configs/detector/yolo11n-baseline.yaml \
  --model path/to/best.pt \
  --split val
uv run aerialtrack evaluate-areas \
  --config configs/detector/yolo11n-cuda-frozen.yaml \
  --model data/models/frozen/yolo11n-cuda-long50.pt \
  --split val
uv run aerialtrack benchmark-deployment \
  --config configs/detector/yolo11n-cuda-frozen.yaml \
  --model data/models/frozen/yolo11n-cuda-long50.pt \
  --split val \
  --count 256 \
  --warmup 10 \
  --repeats 3
uv run aerialtrack check-export-parity \
  --config configs/detector/yolo11n-cuda-frozen.yaml \
  --model data/models/frozen/yolo11n-cuda-long50.pt \
  --onnx data/models/deployment/yolo11n-cuda-long50.onnx \
  --split val \
  --count 64 \
  --confidence 0.25
uv run aerialtrack render-detections \
  --config configs/detector/yolo11n-baseline.yaml \
  --model path/to/best.pt \
  --split val
uv run aerialtrack plot-training \
  --results path/to/yolo11n-baseline/results.csv \
  --output reports/assets/yolo11n-baseline-training.png

The data report and data render-samples commands write a machine-readable dataset report and a five-split ground-truth contact sheet under $AERIALTRACK_DATA_DIR/demos/data-validation/. Generated artifacts are not committed to Git.

Training writes checkpoints, Ultralytics plots, an automatic four-panel training dashboard, and a provenance manifest. The custom dashboard is saved as $AERIALTRACK_DATA_DIR/demos/training/<experiment>-training.png and contains mAP, precision/recall, train/validation losses, and wall time per epoch. Evaluation uses the frozen parameters in the same YAML file and writes a machine-readable JSON report. The render command produces a deterministic six-image prediction montage. All generated artifacts stay under $AERIALTRACK_DATA_DIR.

Planned interface

The following higher-level commands describe the intended interface and remain to be implemented:

uv run aerialtrack benchmark --config configs/robustness.yaml
uv run aerialtrack export --format coreml
uv run aerialtrack demo --source path/to/video.mp4

See PLAN.md for implementation milestones and acceptance criteria, and docs/FAILURES.md for failed approaches and constraints that should not be rediscovered without new evidence.

References

• VisDrone official dataset repository
• Ultralytics VisDrone dataset guide
• Ultralytics multi-object tracking guide
• TrackEval reference implementation
• Apple Core ML overview
• NVIDIA TensorRT prerequisites

Dataset files and pretrained weights are not redistributed by this repository. Their respective terms must be reviewed before redistribution or commercial use.