Skip to content
Snippets Groups Projects
Commit aceae029 authored by Giovanni Delussu's avatar Giovanni Delussu
Browse files

CoalescedRDD changed to work with a big number of partitions both in the...

CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD.
The limitation was in the range that Scala.Int can represent.
parent c1e9cdc4
No related branches found
No related tags found
No related merge requests found
...@@ -37,8 +37,8 @@ class CoalescedRDD[T: ClassManifest]( ...@@ -37,8 +37,8 @@ class CoalescedRDD[T: ClassManifest](
prevSplits.map(_.index).map{idx => new CoalescedRDDPartition(idx, prev, Array(idx)) } prevSplits.map(_.index).map{idx => new CoalescedRDDPartition(idx, prev, Array(idx)) }
} else { } else {
(0 until maxPartitions).map { i => (0 until maxPartitions).map { i =>
val rangeStart = (i * prevSplits.length) / maxPartitions val rangeStart = ((i.toLong * prevSplits.length) / maxPartitions).toInt
val rangeEnd = ((i + 1) * prevSplits.length) / maxPartitions val rangeEnd = (((i.toLong + 1) * prevSplits.length) / maxPartitions).toInt
new CoalescedRDDPartition(i, prev, (rangeStart until rangeEnd).toArray) new CoalescedRDDPartition(i, prev, (rangeStart until rangeEnd).toArray)
}.toArray }.toArray
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment