Skip to content

Speed up EM hot paths without changing results - #30

Open
olivierC80 wants to merge 2 commits into
dlinzer:masterfrom
olivierC80:codex/bitwise-performance
Open

olivierC80 wants to merge 2 commits into
dlinzer:masterfrom
olivierC80:codex/bitwise-performance

Conversation

@olivierC80

Copy link
Copy Markdown

Summary

This PR reduces allocation and native-call overhead in poLCA while preserving
the package's results exactly for the tested execution paths.

The implementation deliberately excludes algebraically equivalent changes that
altered floating-point results in the last bits. It also excludes the
poLCA.compress() optimization because that function is already being changed
in #28.

Changes

  • Compute as.integer(t(y)) once per fit and reuse it in native wrappers.
  • Add PACKAGE = "poLCA" to the existing .C() calls.
  • Fuse the existing postclass and probhat loops into one internal .Call()
    for models with and without covariates.
  • Keep separate complete-data and missing-data native paths without changing
    the order of floating-point operations.
  • Implement rmulti() with .Call() and R's RNG API while preserving both its
    numeric return type and RNG stream.
  • Preallocate known-size objects in poLCA.se() without replacing or
    reordering its matrix calculations.
  • Detect constant manifest columns directly.
  • Use max.col(..., ties.method = "first") for modal assignment.

Exact-result validation

The candidate was compared with commit
2ffbf2fd44fe707c54e3939fe929b8032b5db63f using separately installed source
packages. No rounding or numeric tolerance was used: results were checked with
identical() after removing only the nondeterministic elapsed-time field.

The strict comparison covered:

  • fits with and without standard errors;
  • models with and without covariates;
  • complete and missing manifest data;
  • one-, three-, and four-class models;
  • repeated estimation with nrep = 3;
  • constant manifest variables;
  • posterior probabilities, predicted cells, tables, entropy, coefficients,
    and variance-covariance matrices;
  • matrix and vector inputs to rmulti(), including the resulting RNG state.

All compared objects were strictly identical, including types, dimensions,
attributes, and floating-point values.

The new package test additionally compares the fused native helpers directly
with the historical postclass and probhat wrappers for shared/varying priors
and complete/missing data. It also checks exact rmulti() output and RNG state.

Performance

Mean elapsed time over 10 repetitions, original versus candidate installed
source packages:

Scenario Original Candidate Speedup
fit_basic 0.0222 s 0.0161 s 1.38x
fit_covariates 0.0253 s 0.0207 s 1.22x
fit_covariates_large 0.0783 s 0.0694 s 1.13x
fit_em_large 0.3207 s 0.1765 s 1.82x
fit_em_xlarge 0.8046 s 0.4302 s 1.87x
fit_heavy 0.1099 s 0.0579 s 1.90x
fit_heavy_se 0.1394 s 0.0896 s 1.56x
fit_missing_keep_na 0.0059 s 0.0036 s 1.64x
rmulti_large 0.4100 s 0.0778 s 5.27x
simdata 0.0056 s 0.0015 s 3.73x

Package check

R CMD check --no-manual --no-build-vignettes completes with no errors or
warnings. It reports the same existing NOTE as the unmodified baseline:

Unknown package 'flexmix' in Rd xrefs

Deliberate exclusions

The following previously explored changes are not included because they did
not satisfy strict bit-for-bit equality:

  • replacing t(s) %*% s with crossprod(s);
  • replacing covariance products with tcrossprod();
  • computing Jac.mix through an algebraically equivalent reordered formula;
  • fusing log-likelihood or prior accumulation into the native EM helper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant