Skip to content
Snippets Groups Projects
Commit 39e4e7e4 authored by Steve Lindemann's avatar Steve Lindemann Committed by Cheng Lian
Browse files

[SPARK-8841] [SQL] Fix partition pruning percentage log message

When pruning partitions for a query plan, a message is logged indicating what how many partitions were selected based on predicate criteria, and what percent were pruned.

The current release erroneously uses `1 - total/selected` to compute this quantity, leading to nonsense messages like "pruned -1000% partitions". The fix is simple and obvious.

Author: Steve Lindemann <steve.lindemann@engineersgatelp.com>

Closes #7227 from srlindemann/master and squashes the following commits:

c788061 [Steve Lindemann] fix percentPruned log message
parent 86768b7b
No related branches found
No related tags found
No related merge requests found
...@@ -65,7 +65,7 @@ private[sql] object DataSourceStrategy extends Strategy with Logging { ...@@ -65,7 +65,7 @@ private[sql] object DataSourceStrategy extends Strategy with Logging {
logInfo { logInfo {
val total = t.partitionSpec.partitions.length val total = t.partitionSpec.partitions.length
val selected = selectedPartitions.length val selected = selectedPartitions.length
val percentPruned = (1 - total.toDouble / selected.toDouble) * 100 val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
s"Selected $selected partitions out of $total, pruned $percentPruned% partitions." s"Selected $selected partitions out of $total, pruned $percentPruned% partitions."
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment