Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    6324eb7b
    [Spark-3490] Disable SparkUI for tests · 6324eb7b
    Andrew Or authored
    We currently open many ephemeral ports during the tests, and as a result we occasionally can't bind to new ones. This has caused the `DriverSuite` and the `SparkSubmitSuite` to fail intermittently.
    
    By disabling the `SparkUI` when it's not needed, we already cut down on the number of ports opened significantly, on the order of the number of `SparkContexts` ever created. We must keep it enabled for a few tests for the UI itself, however.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #2363 from andrewor14/disable-ui-for-tests and squashes the following commits:
    
    332a7d5 [Andrew Or] No need to set spark.ui.port to 0 anymore
    30c93a2 [Andrew Or] Simplify streaming UISuite
    a431b84 [Andrew Or] Fix streaming test failures
    8f5ae53 [Andrew Or] Fix no new line at the end
    29c9b5b [Andrew Or] Disable SparkUI for tests
    6324eb7b
    History
    [Spark-3490] Disable SparkUI for tests
    Andrew Or authored
    We currently open many ephemeral ports during the tests, and as a result we occasionally can't bind to new ones. This has caused the `DriverSuite` and the `SparkSubmitSuite` to fail intermittently.
    
    By disabling the `SparkUI` when it's not needed, we already cut down on the number of ports opened significantly, on the order of the number of `SparkContexts` ever created. We must keep it enabled for a few tests for the UI itself, however.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #2363 from andrewor14/disable-ui-for-tests and squashes the following commits:
    
    332a7d5 [Andrew Or] No need to set spark.ui.port to 0 anymore
    30c93a2 [Andrew Or] Simplify streaming UISuite
    a431b84 [Andrew Or] Fix streaming test failures
    8f5ae53 [Andrew Or] Fix no new line at the end
    29c9b5b [Andrew Or] Disable SparkUI for tests