Skip to content

feat: use LUT for numpy AEF dequantization#10

Open
Geethen wants to merge 1 commit into
jakenotjay:mainfrom
Geethen:feat/lut-dequantize
Open

feat: use LUT for numpy AEF dequantization#10
Geethen wants to merge 1 commit into
jakenotjay:mainfrom
Geethen:feat/lut-dequantize

Conversation

@Geethen

@Geethen Geethen commented Jul 8, 2026

Copy link
Copy Markdown

Problem

The plain numpy path in dequantize_aef recomputes the same int8-to-float32 formula for every pixel value. AEF stores only 256 possible raw values, so large materialized windows spend avoidable time repeating (v / 127.5) ** 2 * sign(v) and then applying a separate nodata mask.

Fix

This adds a 256-entry lookup table for the plain ndarray path and maps raw values through it with lut[raw + 128]. The -128 -> NaN nodata mapping is folded into the table. The xarray/DataArray path stays elementwise so dask-backed arrays remain lazy and blockwise.

In local checks on a 2048 x 2048 x 64 materialized AEF window, this changed numpy dequantization from about 2702 ms to 1582 ms, while preserving identical output.

Tests

Adds tests/test_utils.py coverage asserting the LUT path matches the reference formula for all 256 int8 values, for both int8 and int16 inputs.

Ran locally:

uv run pytest tests/test_utils.py
uv run ruff check aef_loader/utils.py tests/test_utils.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant