Skip to content

Commit 3d43e1d

Browse files
committed
feat: add TDigest, every quantile of a stream from one sketch
P2QuantileEstimator tracks one quantile chosen in advance, in constant memory. A t-digest keeps a small list of centroids, each a mean and a weight, sorted by mean, and answers any quantile and the CDF afterwards from the same sketch. A centroid may absorb points only while it spans at most one unit of the scale function k(q) = d / (2 pi) * asin(2q - 1), and asin is steep at both ends, so centroids in the tails stay at one or a few points while those near the median are allowed to grow. The rank error is of the order of 1 / d around the median and shrinks towards q = 0 and q = 1, which is the trade latency percentiles want: p99.9 comes out nearly exact from a sketch of a few kilobytes. Insertions are buffered and folded into the centroid list in batches, so add is O(log d) amortised; quantile and cdf are O(d) and interpolate linearly between centroids, each treated as sitting at the centre of the weight it carries. The minimum and maximum are tracked separately and returned exactly for q = 0 and q = 1. Two digests merge the way partial sums do, so sketches built on separate shards combine into one. Non-finite samples, invalid weights, quantiles outside [0, 1] and a compression below 10 are refused; queries fold the pending buffer in first, so the class is documented as not thread-safe. Tests: on 100 000 uniform samples every tested quantile lands within 0.01 of the requested rank; on 200 000 Gaussian samples the rank error stays under a bound that narrows towards the tails, from 0.01 at the median to 0.0005 at q = 0.001 and q = 0.999; three shards merged answer within 0.01 of the true rank, with total weight, minimum and maximum exact; the centroid count stays at or below the compression after 200 000 samples for compressions of 20, 100 and 500; and weighted, sorted, constant and heavily duplicated input are covered. Signed-off-by: alxkm <19151554+alxkm@users.noreply.github.com>
1 parent 3f06708 commit 3d43e1d

2 files changed

Lines changed: 931 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)