Skip to content
Snippets Groups Projects
  • fidato's avatar
    6f369713
    [SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles · 6f369713
    fidato authored
    ## What changes were proposed in this pull request?
    
    This Pull request comprises of the critical bug SPARK-16575 changes. This change rectifies the issue with BinaryFileRDD partition calculations as  upon creating an RDD with sc.binaryFiles, the resulting RDD always just consisted of two partitions only.
    ## How was this patch tested?
    
    The original issue ie. getNumPartitions on binary Files RDD (always having two partitions) was first replicated and then tested upon the changes. Also the unit tests have been checked and passed.
    
    This contribution is my original work and I licence the work to the project under the project's open source license
    
    srowen hvanhovell rxin vanzin skyluc kmader zsxwing datafarmer Please have a look .
    
    Author: fidato <fidato.july13@gmail.com>
    
    Closes #15327 from fidato13/SPARK-16575.
    6f369713
    History
    [SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles
    fidato authored
    ## What changes were proposed in this pull request?
    
    This Pull request comprises of the critical bug SPARK-16575 changes. This change rectifies the issue with BinaryFileRDD partition calculations as  upon creating an RDD with sc.binaryFiles, the resulting RDD always just consisted of two partitions only.
    ## How was this patch tested?
    
    The original issue ie. getNumPartitions on binary Files RDD (always having two partitions) was first replicated and then tested upon the changes. Also the unit tests have been checked and passed.
    
    This contribution is my original work and I licence the work to the project under the project's open source license
    
    srowen hvanhovell rxin vanzin skyluc kmader zsxwing datafarmer Please have a look .
    
    Author: fidato <fidato.july13@gmail.com>
    
    Closes #15327 from fidato13/SPARK-16575.