TrackSense

TrackSense is a real-time desktop vision app for handheld object understanding. It detects prominent objects in the camera feed, tracks them smoothly, adds short Turkish labels on top of the boxes, and writes subtitle-style scene summaries at the bottom of the screen.

The project is built around a hybrid pipeline:

• Local object detection with ONNX YOLO
• Hand-aware ranking and multi-object tracking
• VLM-based labels and scene subtitles through llama-server

Highlights

• Startup onboarding inside the camera app instead of terminal setup prompts
• Automatic scan for valid GGUF + mmproj model pairs
• Runtime detection for llama-server
• Managed default model download from Hugging Face
• Local-only fallback mode when VLM is unavailable
• Stable multi-object tracking with low-noise overlays
• Modern glass-style UI with status pills and subtitle bar
• Remembered model/runtime selection across launches

Demo Behavior

TrackSense is designed for the following interaction:

• A user shows an object to the camera
• The object gets a stable rounded box
• A short label appears above the box
• A subtitle-style Turkish sentence describes the scene
• If two objects are shown, both can be tracked independently
• Background clutter is filtered so the screen does not fill with random boxes

Requirements

• Python 3.10+
• A working webcam
• llama-server installed and accessible through PATH or selected from the in-app settings
• Enough local disk space for the managed model download

Python dependencies are listed in requirements.txt.

Installation

1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

2. Install Python packages

pip install -r requirements.txt

3. Install llama-server

TrackSense does not auto-install the runtime. It detects whether llama-server exists and guides the user if it is missing.

Typical options:

• macOS: install llama.cpp with Homebrew or use a manually built binary
• Windows: use a prebuilt binary or package manager
• Linux: use a distro package or a manually built binary

You can also set the runtime path later from the in-app settings screen.

4. Start the app

python main.py

First Launch Flow

On startup the app silently checks:

1. Saved active model path
2. Managed model cache
3. Previously remembered model directories
4. LM Studio model directory if present
5. llama-server runtime availability

If both a valid model and runtime are found, the app goes straight into live mode.

If not, it opens an in-app onboarding overlay with options to:

• Download the default managed model
• Continue in local-only mode
• Select a discovered model
• Pick a model folder manually
• Pick a llama-server executable manually

Default Managed Model

The managed model used by onboarding is:

• Repository: bartowski/google_gemma-4-E2B-it-GGUF
• Text model: google_gemma-4-E2B-it-Q4_K_S.gguf
• Vision projector: mmproj-google_gemma-4-E2B-it-f16.gguf

The download is stored in a per-user cache directory, not inside the git repo.

Controls

• q: quit
• d: toggle debug HUD
• m: switch active model
• s: open settings
• j / k: move through model lists in overlays
• Esc: close onboarding/settings overlays or quit from startup screens

Modes

Ready mode

Full pipeline is active:

• local detection
• hand-aware selection
• tracking
• VLM labels
• VLM subtitles

Local-only mode

Used when the user skips VLM setup or runtime is missing:

• local detection and tracking continue
• object boxes still render
• fallback labels remain visible
• subtitle area stays in local status mode

Project Structure

• main.py: top-level app state machine and UI loop
• bootstrap.py: startup scan, onboarding helpers, settings, model discovery
• config.py: app constants, environment settings, paths
• vision.py: detector, hand analysis, tracking, scene state
• vlm.py: VLM worker, label/subtitle requests, runtime readiness
• model_server.py: llama-server launch and health checks
• ui.py: modern overlay rendering
• app_types.py: shared dataclasses and state types
• tests/test_smoke.py: smoke tests

Configuration

TrackSense supports environment variables for tuning runtime behavior:

• CAMERA_WIDTH
• CAMERA_HEIGHT
• CAMERA_DEVICE_INDEX
• CAMERA_BUFFER_SIZE
• SERVER_START_TIMEOUT_SECONDS
• REQUEST_TIMEOUT_SECONDS
• DETECTION_INTERVAL_SECONDS
• SUBTITLE_INTERVAL_SECONDS
• LABEL_INTERVAL_SECONDS
• TRACK_MAX_MISSES
• TRACK_MIN_HITS
• TRACK_ASSOCIATION_DISTANCE
• MIN_PRIORITY_SCORE
• DETECTOR_INPUT_SIZE
• DETECTOR_CONFIDENCE
• DETECTOR_NMS_IOU

See config.py for current defaults.

Storage Paths

TrackSense stores user settings and managed models outside the repo.

Settings

• macOS: ~/Library/Application Support/TrackSense/settings.json
• Windows: %APPDATA%/TrackSense/settings.json
• Linux: ~/.config/gemma-camera-prototype/settings.json

Managed model cache

• macOS: ~/Library/Caches/TrackSense/models
• Windows: %LOCALAPPDATA%/TrackSense/models
• Linux: ~/.cache/gemma-camera-prototype/models

The app also reads legacy settings written under the old GemmaCameraPrototype name.

Tests

Run the smoke tests with:

./.venv/bin/python -m unittest discover -s tests -v

Static syntax validation:

./.venv/bin/python -m py_compile main.py app_types.py bootstrap.py config.py model_server.py ui.py vlm.py vision.py tests/test_smoke.py

Troubleshooting

llama-server bulunamadi

• Install llama-server
• Or open settings with s and select the binary manually

Camera does not open

• Check webcam permissions for Terminal or Python
• Verify CAMERA_DEVICE_INDEX
• Make sure another app is not locking the camera

Model is not detected

• The selected folder must contain both:
  • a text .gguf model
  • a matching mmproj*.gguf

Hand detection fails

The app falls back to center-and-size prioritization. Tracking continues, but handheld-object emphasis becomes weaker.

VLM is slow

Reduce request load by tuning:

• LABEL_INTERVAL_SECONDS
• SUBTITLE_INTERVAL_SECONDS
• REQUEST_TIMEOUT_SECONDS

Architecture Notes

High-level technical notes are documented in docs/ARCHITECTURE.md.