-
- Downloads
[SPARK-1010] Clean up uses of System.setProperty in unit tests
Several of our tests call System.setProperty (or test code which implicitly sets system properties) and don't always reset/clear the modified properties, which can create ordering dependencies between tests and cause hard-to-diagnose failures. This patch removes most uses of System.setProperty from our tests, since in most cases we can use SparkConf to set these configurations (there are a few exceptions, including the tests of SparkConf itself). For the cases where we continue to use System.setProperty, this patch introduces a `ResetSystemProperties` ScalaTest mixin class which snapshots the system properties before individual tests and to automatically restores them on test completion / failure. See the block comment at the top of the ResetSystemProperties class for more details. Author: Josh Rosen <joshrosen@databricks.com> Closes #3739 from JoshRosen/cleanup-system-properties-in-tests and squashes the following commits: 0236d66 [Josh Rosen] Replace setProperty uses in two example programs / tools 3888fe3 [Josh Rosen] Remove setProperty use in LocalJavaStreamingContext 4f4031d [Josh Rosen] Add note on why SparkSubmitSuite needs ResetSystemProperties 4742a5b [Josh Rosen] Clarify ResetSystemProperties trait inheritance ordering. 0eaf0b6 [Josh Rosen] Remove setProperty call in TaskResultGetterSuite. 7a3d224 [Josh Rosen] Fix trait ordering 3fdb554 [Josh Rosen] Remove setProperty call in TaskSchedulerImplSuite bee20df [Josh Rosen] Remove setProperty calls in SparkContextSchedulerCreationSuite 655587c [Josh Rosen] Remove setProperty calls in JobCancellationSuite 3f2f955 [Josh Rosen] Remove System.setProperty calls in DistributedSuite cfe9cce [Josh Rosen] Remove use of system properties in SparkContextSuite 8783ab0 [Josh Rosen] Remove TestUtils.setSystemProperty, since it is subsumed by the ResetSystemProperties trait. 633a84a [Josh Rosen] Remove use of system properties in FileServerSuite 25bfce2 [Josh Rosen] Use ResetSystemProperties in UtilsSuite 1d1aa5a [Josh Rosen] Use ResetSystemProperties in SizeEstimatorSuite dd9492b [Josh Rosen] Use ResetSystemProperties in AkkaUtilsSuite b0daff2 [Josh Rosen] Use ResetSystemProperties in BlockManagerSuite e9ded62 [Josh Rosen] Use ResetSystemProperties in TaskSchedulerImplSuite 5b3cb54 [Josh Rosen] Use ResetSystemProperties in SparkListenerSuite 0995c4b [Josh Rosen] Use ResetSystemProperties in SparkContextSchedulerCreationSuite c83ded8 [Josh Rosen] Use ResetSystemProperties in SparkConfSuite 51aa870 [Josh Rosen] Use withSystemProperty in ShuffleSuite 60a63a1 [Josh Rosen] Use ResetSystemProperties in JobCancellationSuite 14a92e4 [Josh Rosen] Use withSystemProperty in FileServerSuite 628f46c [Josh Rosen] Use ResetSystemProperties in DistributedSuite 9e3e0dd [Josh Rosen] Add ResetSystemProperties test fixture mixin; use it in SparkSubmitSuite. 4dcea38 [Josh Rosen] Move withSystemProperty to TestUtils class.
Showing
- core/src/test/scala/org/apache/spark/DistributedSuite.scala 5 additions, 16 deletionscore/src/test/scala/org/apache/spark/DistributedSuite.scala
- core/src/test/scala/org/apache/spark/FileServerSuite.scala 8 additions, 8 deletionscore/src/test/scala/org/apache/spark/FileServerSuite.scala
- core/src/test/scala/org/apache/spark/JobCancellationSuite.scala 10 additions, 11 deletions...rc/test/scala/org/apache/spark/JobCancellationSuite.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 9 additions, 13 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- core/src/test/scala/org/apache/spark/SparkConfSuite.scala 19 additions, 32 deletionscore/src/test/scala/org/apache/spark/SparkConfSuite.scala
- core/src/test/scala/org/apache/spark/SparkContextSchedulerCreationSuite.scala 13 additions, 18 deletions...org/apache/spark/SparkContextSchedulerCreationSuite.scala
- core/src/test/scala/org/apache/spark/SparkContextSuite.scala 22 additions, 40 deletionscore/src/test/scala/org/apache/spark/SparkContextSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 4 additions, 2 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala 3 additions, 6 deletions...scala/org/apache/spark/scheduler/SparkListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala 8 additions, 15 deletions...la/org/apache/spark/scheduler/TaskResultGetterSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 3 additions, 3 deletions...a/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 7 additions, 16 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala 1 addition, 1 deletion...src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala
- core/src/test/scala/org/apache/spark/util/ResetSystemProperties.scala 57 additions, 0 deletions...t/scala/org/apache/spark/util/ResetSystemProperties.scala
- core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala 8 additions, 30 deletions...test/scala/org/apache/spark/util/SizeEstimatorSuite.scala
- core/src/test/scala/org/apache/spark/util/UtilsSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/util/UtilsSuite.scala
- examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala 2 additions, 4 deletions.../main/scala/org/apache/spark/examples/BroadcastTest.scala
- external/flume/src/test/java/org/apache/spark/streaming/LocalJavaStreamingContext.java 6 additions, 2 deletions...org/apache/spark/streaming/LocalJavaStreamingContext.java
- external/mqtt/src/test/java/org/apache/spark/streaming/LocalJavaStreamingContext.java 6 additions, 2 deletions...org/apache/spark/streaming/LocalJavaStreamingContext.java
- external/twitter/src/test/java/org/apache/spark/streaming/LocalJavaStreamingContext.java 6 additions, 2 deletions...org/apache/spark/streaming/LocalJavaStreamingContext.java
Loading
Please register or sign in to comment