Skip to content
Snippets Groups Projects
Commit 52eb54d0 authored by Tathagata Das's avatar Tathagata Das
Browse files

[Spark 1877] ClassNotFoundException when loading RDD with serialized objects

Updated version of #821

Author: Tathagata Das <tathagata.das1565@gmail.com>
Author: Ghidireac <bogdang@u448a5b0a73d45358d94a.ant.amazon.com>

Closes #835 from tdas/SPARK-1877 and squashes the following commits:

f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with serialized objects
parent bcb9dce6
No related branches found
No related tags found
No related merge requests found
...@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging { ...@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging {
minPartitions: Int = defaultMinPartitions minPartitions: Int = defaultMinPartitions
): RDD[T] = { ): RDD[T] = {
sequenceFile(path, classOf[NullWritable], classOf[BytesWritable], minPartitions) sequenceFile(path, classOf[NullWritable], classOf[BytesWritable], minPartitions)
.flatMap(x => Utils.deserialize[Array[T]](x._2.getBytes)) .flatMap(x => Utils.deserialize[Array[T]](x._2.getBytes, Utils.getContextOrSparkClassLoader))
} }
protected[spark] def checkpointFile[T: ClassTag]( protected[spark] def checkpointFile[T: ClassTag](
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment