-
- Downloads
[SPARK-21051][SQL] Add hash map metrics to aggregate
## What changes were proposed in this pull request? This adds the average hash map probe metrics to hash aggregate. `BytesToBytesMap` already has API to get the metrics, this PR adds an API to `UnsafeFixedWidthAggregationMap` to access it. Preparing a test for this metrics seems tricky, because we don't know what collision keys are. For now, the test case generates random data large enough to have desired probe. TODO in later PR: add hash map metrics to join. ## How was this patch tested? Added test to SQLMetricsSuite. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #18258 from viirya/SPARK-20953.
Showing
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 7 additions, 16 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala 19 additions, 8 deletions...che/spark/sql/execution/aggregate/HashAggregateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 7 additions, 3 deletions...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 31 additions, 1 deletion...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 8 additions, 16 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 63 additions, 16 deletions...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
Loading
Please register or sign in to comment