Skip to content
Snippets Groups Projects
Commit 2402b914 authored by WeichenXu's avatar WeichenXu Committed by Reynold Xin
Browse files

[SPARK-15702][DOCUMENTATION] Update document programming-guide accumulator section

## What changes were proposed in this pull request?

Update document programming-guide accumulator section (scala language)
java and python version, because the API haven't done, so I do not modify them.

## How was this patch tested?

N/A

Author: WeichenXu <WeichenXu123@outlook.com>

Closes #13441 from WeichenXu123/update_doc_accumulatorV2_clean.
parent 07a98ca4
No related branches found
No related tags found
No related merge requests found
...@@ -1352,41 +1352,42 @@ The code below shows an accumulator being used to add up the elements of an arra ...@@ -1352,41 +1352,42 @@ The code below shows an accumulator being used to add up the elements of an arra
<div data-lang="scala" markdown="1"> <div data-lang="scala" markdown="1">
{% highlight scala %} {% highlight scala %}
scala> val accum = sc.accumulator(0, "My Accumulator") scala> val accum = sc.longAccumulator("My Accumulator")
accum: org.apache.spark.Accumulator[Int] = 0 accum: org.apache.spark.util.LongAccumulator = LongAccumulator(id: 0, name: Some(My Accumulator), value: 0)
scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x) scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum.add(x))
... ...
10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s
scala> accum.value scala> accum.value
res2: Int = 10 res2: Long = 10
{% endhighlight %} {% endhighlight %}
While this code used the built-in support for accumulators of type Int, programmers can also While this code used the built-in support for accumulators of type Long, programmers can also
create their own types by subclassing [AccumulatorParam](api/scala/index.html#org.apache.spark.AccumulatorParam). create their own types by subclassing [AccumulatorV2](api/scala/index.html#org.apache.spark.AccumulatorV2).
The AccumulatorParam interface has two methods: `zero` for providing a "zero value" for your data The AccumulatorV2 abstract class has several methods which need to override:
type, and `addInPlace` for adding two values together. For example, supposing we had a `Vector` class `reset` for resetting the accumulator to zero, and `add` for add anothor value into the accumulator, `merge` for merging another same-type accumulator into this one. Other methods need to override can refer to scala API document. For example, supposing we had a `MyVector` class
representing mathematical vectors, we could write: representing mathematical vectors, we could write:
{% highlight scala %} {% highlight scala %}
object VectorAccumulatorParam extends AccumulatorParam[Vector] { object VectorAccumulatorV2 extends AccumulatorV2[MyVector, MyVector] {
def zero(initialValue: Vector): Vector = { val vec_ : MyVector = MyVector.createZeroVector
Vector.zeros(initialValue.size) def reset(): MyVector = {
vec_.reset()
} }
def addInPlace(v1: Vector, v2: Vector): Vector = { def add(v1: MyVector, v2: MyVector): MyVector = {
v1 += v2 vec_.add(v2)
} }
...
} }
// Then, create an Accumulator of this type: // Then, create an Accumulator of this type:
val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam) val myVectorAcc = new VectorAccumulatorV2
// Then, register it into spark context:
sc.register(myVectorAcc, "MyVectorAcc1")
{% endhighlight %} {% endhighlight %}
In Scala, Spark also supports the more general [Accumulable](api/scala/index.html#org.apache.spark.Accumulable) Note that, when programmers define their own type of AccumulatorV2, the resulting type can be same or not same with the elements added.
interface to accumulate data where the resulting type is not the same as the elements added (e.g. build
a list by collecting together elements), and the `SparkContext.accumulableCollection` method for accumulating
common Scala collection types.
</div> </div>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment