Skip to content
Snippets Groups Projects
Commit b1422cbd authored by Mark Hamstra's avatar Mark Hamstra
Browse files

added foldByKey

parent 4032beba
No related branches found
No related tags found
No related merge requests found
......@@ -88,6 +88,13 @@ class PairRDDFunctions[K: ClassManifest, V: ClassManifest](
combineByKey(createCombiner, mergeValue, mergeCombiners, new HashPartitioner(numPartitions))
}
/**
* Merge the values for each key using an associative function and a neutral "zero value".
*/
def foldByKey[V1 >: V](zeroValue: V1)(op: (V1, V1) => V1): RDD[(K, V1)] = {
groupByKey.mapValues(seq => seq.fold(zeroValue)(op))
}
/**
* Merge the values for each key using an associative reduce function. This will also perform
* the merging locally on each mapper before sending results to a reducer, similarly to a
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment