From cf5963c961e7eba37bdd58658ed4dfff66ce3c72 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=83=AD=E5=B0=8F=E9=BE=99=2010207633?= <guo.xiaolong1@zte.com.cn> Date: Sat, 1 Apr 2017 11:48:58 +0100 Subject: [PATCH] =?UTF-8?q?[SPARK-20177]=20Document=20about=20compression?= =?UTF-8?q?=20way=20has=20some=20little=20detail=20ch=E2=80=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit …anges. ## What changes were proposed in this pull request? Document compression way little detail changes. 1.spark.eventLog.compress add 'Compression will use spark.io.compression.codec.' 2.spark.broadcast.compress add 'Compression will use spark.io.compression.codec.' 3,spark.rdd.compress add 'Compression will use spark.io.compression.codec.' 4.spark.io.compression.codec add 'event log describe'. eg Through the documents, I don't know what is compression mode about 'event log'. ## How was this patch tested? manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. Author: éƒå°é¾™ 10207633 <guo.xiaolong1@zte.com.cn> Closes #17498 from guoxiaolongzte/SPARK-20177. --- docs/configuration.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index a975392540..2687f542b8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -639,6 +639,7 @@ Apart from these, the following properties are also available, and may be useful <td>false</td> <td> Whether to compress logged events, if <code>spark.eventLog.enabled</code> is true. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> @@ -773,14 +774,15 @@ Apart from these, the following properties are also available, and may be useful <td>true</td> <td> Whether to compress broadcast variables before sending them. Generally a good idea. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> <td><code>spark.io.compression.codec</code></td> <td>lz4</td> <td> - The codec used to compress internal data such as RDD partitions, broadcast variables and - shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>, + The codec used to compress internal data such as RDD partitions, event log, broadcast variables + and shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>, and <code>snappy</code>. You can also use fully qualified class names to specify the codec, e.g. <code>org.apache.spark.io.LZ4CompressionCodec</code>, @@ -881,6 +883,7 @@ Apart from these, the following properties are also available, and may be useful <code>StorageLevel.MEMORY_ONLY_SER</code> in Java and Scala or <code>StorageLevel.MEMORY_ONLY</code> in Python). Can save substantial space at the cost of some extra CPU time. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> -- GitLab