Skip to content
Snippets Groups Projects
Commit 43af2817 authored by Erik Erlandson's avatar Erik Erlandson Committed by Reynold Xin
Browse files

[SPARK-2911] apply parent[T](j) to clarify UnionRDD code

References to dependencies(j) for actually obtaining RDD parents are less common than I originally estimated.   It does clarify UnionRDD (also will clarify some of my other PRs)

Use of firstParent[T] is ubiquitous, but not as sure that benefits from being replaced with parent(0)[T].

Author: Erik Erlandson <eerlands@redhat.com>

Closes #1858 from erikerlandson/spark-2911-pr2 and squashes the following commits:

7ffea74 [Erik Erlandson] [SPARK-2911] apply parent[T](j) to clarify UnionRDD code
parent 1c84dba9
No related branches found
No related tags found
No related merge requests found
...@@ -83,8 +83,7 @@ class UnionRDD[T: ClassTag]( ...@@ -83,8 +83,7 @@ class UnionRDD[T: ClassTag](
override def compute(s: Partition, context: TaskContext): Iterator[T] = { override def compute(s: Partition, context: TaskContext): Iterator[T] = {
val part = s.asInstanceOf[UnionPartition[T]] val part = s.asInstanceOf[UnionPartition[T]]
val parentRdd = dependencies(part.parentRddIndex).rdd.asInstanceOf[RDD[T]] parent[T](part.parentRddIndex).iterator(part.parentPartition, context)
parentRdd.iterator(part.parentPartition, context)
} }
override def getPreferredLocations(s: Partition): Seq[String] = override def getPreferredLocations(s: Partition): Seq[String] =
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment