Skip to content
Snippets Groups Projects
  • Jeremy Freeman's avatar
    4bc3bb29
    StatCounter on NumPy arrays [PYSPARK][SPARK-2012] · 4bc3bb29
    Jeremy Freeman authored
    These changes allow StatCounters to work properly on NumPy arrays, to fix the issue reported here  (https://issues.apache.org/jira/browse/SPARK-2012).
    
    If NumPy is installed, the NumPy functions ``maximum``, ``minimum``, and ``sqrt``, which work on arrays, are used to merge statistics. If not, we fall back on scalar operators, so it will work on arrays with NumPy, but will also work without NumPy.
    
    New unit tests added, along with a check for NumPy in the tests.
    
    Author: Jeremy Freeman <the.freeman.lab@gmail.com>
    
    Closes #1725 from freeman-lab/numpy-max-statcounter and squashes the following commits:
    
    fe973b1 [Jeremy Freeman] Avoid duplicate array import in tests
    7f0e397 [Jeremy Freeman] Refactored check for numpy
    8e764dd [Jeremy Freeman] Explicit numpy imports
    875414c [Jeremy Freeman] Fixed indents
    1c8a832 [Jeremy Freeman] Unit tests for StatCounter with NumPy arrays
    176a127 [Jeremy Freeman] Use numpy arrays in StatCounter
    4bc3bb29
    History
    StatCounter on NumPy arrays [PYSPARK][SPARK-2012]
    Jeremy Freeman authored
    These changes allow StatCounters to work properly on NumPy arrays, to fix the issue reported here  (https://issues.apache.org/jira/browse/SPARK-2012).
    
    If NumPy is installed, the NumPy functions ``maximum``, ``minimum``, and ``sqrt``, which work on arrays, are used to merge statistics. If not, we fall back on scalar operators, so it will work on arrays with NumPy, but will also work without NumPy.
    
    New unit tests added, along with a check for NumPy in the tests.
    
    Author: Jeremy Freeman <the.freeman.lab@gmail.com>
    
    Closes #1725 from freeman-lab/numpy-max-statcounter and squashes the following commits:
    
    fe973b1 [Jeremy Freeman] Avoid duplicate array import in tests
    7f0e397 [Jeremy Freeman] Refactored check for numpy
    8e764dd [Jeremy Freeman] Explicit numpy imports
    875414c [Jeremy Freeman] Fixed indents
    1c8a832 [Jeremy Freeman] Unit tests for StatCounter with NumPy arrays
    176a127 [Jeremy Freeman] Use numpy arrays in StatCounter