You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Work on performance issues in summary statistics due to using ArrayBase::sum #35
The summary statistics methods use ArrayBase::sum (directly or indirectly) in anticipation of pairwise summation (rust-ndarray/ndarray#577), which provides improved accuracy over naive summation using fold. However, to do this, some of the methods have unnecessary allocations or other performance issues.
For example, harmonic_mean is implemented like this:
self.map(|x| x.recip()).mean().map(|x| x.recip())
It's implemented this way to take advantage of .mean() (which is implemented in terms of .sum()), but this approach requires a temporary allocation for the result of self.map.
summary_statistics::means::moments has a similar issue:
for k in2..=order {
moments.push(a.map(|x| x.powi(k)).sum() / n_elements)}
It's implemented this way to take advantage of .sum(). However, this implementation requires a temporary allocation for the result of a.map. Additionally, it would probably be faster to make the loop over k be the innermost loop to improve the locality of reference.
We should be able to resolve these issues with a lazy version of map combined with a pairwise summation method on that lazy map. Something like jturner314/nditer would work once it's stable.
[Edit: This issue also appears in the entropy methods.]
Activity
vks commentedon Apr 8, 2019
I'm not sure whether this helps, but the average crate provides constant-memory algorithms to calculate various statistics.