-
- Downloads
SPARK-1240: handle the case of empty RDD when takeSample
https://spark-project.atlassian.net/browse/SPARK-1240 It seems that the current implementation does not handle the empty RDD case when run takeSample In this patch, before calling sample() inside takeSample API, I add a checker for this case and returns an empty Array when it's a empty RDD; also in sample(), I add a checker for the invalid fraction value In the test case, I also add several lines for this case Author: CodingCat <zhunansjtu@gmail.com> Closes #135 from CodingCat/SPARK-1240 and squashes the following commits: fef57d4 [CodingCat] fix the same problem in PySpark 36db06b [CodingCat] create new test cases for takeSample from an empty red 810948d [CodingCat] further fix a40e8fb [CodingCat] replace if with require ad483fd [CodingCat] handle the case with empty RDD when take sample
Showing
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 6 additions, 1 deletioncore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala 7 additions, 0 deletionscore/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
- python/pyspark/rdd.py 4 additions, 0 deletionspython/pyspark/rdd.py
Loading
Please register or sign in to comment