-
- Downloads
[SPARK-12887] Do not expose var's in TaskMetrics
This is a step in implementing SPARK-10620, which migrates TaskMetrics to accumulators. TaskMetrics has a bunch of var's, some are fully public, some are `private[spark]`. This is bad coding style that makes it easy to accidentally overwrite previously set metrics. This has happened a few times in the past and caused bugs that were difficult to debug. Instead, we should have get-or-create semantics, which are more readily understandable. This makes sense in the case of TaskMetrics because these are just aggregated metrics that we want to collect throughout the task, so it doesn't matter who's incrementing them. Parent PR: #10717 Author: Andrew Or <andrew@databricks.com> Author: Josh Rosen <joshrosen@databricks.com> Author: andrewor14 <andrew@databricks.com> Closes #10815 from andrewor14/get-or-create-metrics.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java 1 addition, 2 deletions...ache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 1 addition, 2 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 1 addition, 3 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/scala/org/apache/spark/CacheManager.scala 1 addition, 5 deletionscore/src/main/scala/org/apache/spark/CacheManager.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 120 additions, 60 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala 29 additions, 17 deletions...rc/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
- core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala 2 additions, 2 deletions...la/org/apache/spark/shuffle/BlockStoreShuffleReader.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala 1 addition, 2 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 2 additions, 4 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 4 additions, 8 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 1 addition, 1 deletion...rg/apache/spark/storage/ShuffleBlockFetcherIterator.scala
- core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala 1 addition, 1 deletion...cala/org/apache/spark/storage/StorageStatusListener.scala
- core/src/main/scala/org/apache/spark/ui/storage/StorageTab.scala 2 additions, 2 deletions...c/main/scala/org/apache/spark/ui/storage/StorageTab.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 75 additions, 90 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 6 additions, 4 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/CacheManagerSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/CacheManagerSuite.scala
- core/src/test/scala/org/apache/spark/executor/TaskMetricsSuite.scala 1 addition, 1 deletion...st/scala/org/apache/spark/executor/TaskMetricsSuite.scala
Loading
Please register or sign in to comment