Skip to content
Snippets Groups Projects
Commit cf5963c9 authored by 郭小龙 10207633's avatar 郭小龙 10207633 Committed by Sean Owen
Browse files

[SPARK-20177] Document about compression way has some little detail ch…

…anges.

## What changes were proposed in this pull request?

Document compression way little detail changes.
1.spark.eventLog.compress add 'Compression will use spark.io.compression.codec.'
2.spark.broadcast.compress add 'Compression will use spark.io.compression.codec.'
3,spark.rdd.compress add 'Compression will use spark.io.compression.codec.'
4.spark.io.compression.codec add 'event log describe'.

eg
Through the documents, I don't know  what is compression mode about 'event log'.

## How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn>

Closes #17498 from guoxiaolongzte/SPARK-20177.
parent 567a50ac
No related branches found
No related tags found
No related merge requests found
......@@ -639,6 +639,7 @@ Apart from these, the following properties are also available, and may be useful
<td>false</td>
<td>
Whether to compress logged events, if <code>spark.eventLog.enabled</code> is true.
Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>
......@@ -773,14 +774,15 @@ Apart from these, the following properties are also available, and may be useful
<td>true</td>
<td>
Whether to compress broadcast variables before sending them. Generally a good idea.
Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>
<td><code>spark.io.compression.codec</code></td>
<td>lz4</td>
<td>
The codec used to compress internal data such as RDD partitions, broadcast variables and
shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>,
The codec used to compress internal data such as RDD partitions, event log, broadcast variables
and shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>,
and <code>snappy</code>. You can also use fully qualified class names to specify the codec,
e.g.
<code>org.apache.spark.io.LZ4CompressionCodec</code>,
......@@ -881,6 +883,7 @@ Apart from these, the following properties are also available, and may be useful
<code>StorageLevel.MEMORY_ONLY_SER</code> in Java
and Scala or <code>StorageLevel.MEMORY_ONLY</code> in Python).
Can save substantial space at the cost of some extra CPU time.
Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment