-
- Downloads
[Spark-3490] Disable SparkUI for tests
We currently open many ephemeral ports during the tests, and as a result we occasionally can't bind to new ones. This has caused the `DriverSuite` and the `SparkSubmitSuite` to fail intermittently. By disabling the `SparkUI` when it's not needed, we already cut down on the number of ports opened significantly, on the order of the number of `SparkContexts` ever created. We must keep it enabled for a few tests for the UI itself, however. Author: Andrew Or <andrewor14@gmail.com> Closes #2363 from andrewor14/disable-ui-for-tests and squashes the following commits: 332a7d5 [Andrew Or] No need to set spark.ui.port to 0 anymore 30c93a2 [Andrew Or] Simplify streaming UISuite a431b84 [Andrew Or] Fix streaming test failures 8f5ae53 [Andrew Or] Fix no new line at the end 29c9b5b [Andrew Or] Disable SparkUI for tests
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 9 additions, 3 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 1 addition, 1 deletion...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/SimrSchedulerBackend.scala 3 additions, 3 deletions...apache/spark/scheduler/cluster/SimrSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala 3 additions, 1 deletion...spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
- core/src/test/scala/org/apache/spark/ui/UISuite.scala 31 additions, 13 deletionscore/src/test/scala/org/apache/spark/ui/UISuite.scala
- pom.xml 1 addition, 1 deletionpom.xml
- project/SparkBuild.scala 1 addition, 1 deletionproject/SparkBuild.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 9 additions, 2 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala 1 addition, 1 deletion...in/scala/org/apache/spark/streaming/StreamingSource.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingTab.scala 19 additions, 6 deletions...in/scala/org/apache/spark/streaming/ui/StreamingTab.scala
- streaming/src/test/scala/org/apache/spark/streaming/UISuite.scala 12 additions, 4 deletions...g/src/test/scala/org/apache/spark/streaming/UISuite.scala
- yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 1 addition, 1 deletion...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala 1 addition, 1 deletion.../spark/scheduler/cluster/YarnClientSchedulerBackend.scala
Loading
Please register or sign in to comment