-
- Downloads
Remove Hadoop object cloning and warn users making Hadoop RDD's.
The code introduced in #359 used Hadoop's WritableUtils.clone() to duplicate objects when reading from Hadoop files. Some users have reported exceptions when cloning data in verious file formats, including Avro and another custom format. This patch removes that functionality to ensure stability for the 0.9 release. Instead, it puts a clear warning in the documentation that copying may be necessary for Hadoop data sets.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 77 additions, 43 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 48 additions, 117 deletions...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 5 additions, 20 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 3 additions, 19 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 1 addition, 22 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
Loading
Please register or sign in to comment