Skip to content
  • Sean Owen's avatar
    7120a297
    SPARK-1798. Tests should clean up temp files · 7120a297
    Sean Owen authored
    Three issues related to temp files that tests generate – these should be touched up for hygiene but are not urgent.
    
    Modules have a log4j.properties which directs the unit-test.log output file to a directory like `[module]/target/unit-test.log`. But this ends up creating `[module]/[module]/target/unit-test.log` instead of former.
    
    The `work/` directory is not deleted by "mvn clean", in the parent and in modules. Neither is the `checkpoint/` directory created under the various external modules.
    
    Many tests create a temp directory, which is not usually deleted. This can be largely resolved by calling `deleteOnExit()` at creation and trying to call `Utils.deleteRecursively` consistently to clean up, sometimes in an `@After` method.
    
    _If anyone seconds the motion, I can create a more significant change that introduces a new test trait along the lines of `LocalSparkContext`, which provides management of temp directories for subclasses to take advantage of._
    
    Author: Sean Owen <sowen@cloudera.com>
    
    Closes #732 from srowen/SPARK-1798 and squashes the following commits:
    
    5af578e [Sean Owen] Try to consistently delete test temp dirs and files, and set deleteOnExit() for each
    b21b356 [Sean Owen] Remove work/ and checkpoint/ dirs with mvn clean
    bdd0f41 [Sean Owen] Remove duplicate module dir in log4j.properties output path for tests
    7120a297
    SPARK-1798. Tests should clean up temp files
    Sean Owen authored
    Three issues related to temp files that tests generate – these should be touched up for hygiene but are not urgent.
    
    Modules have a log4j.properties which directs the unit-test.log output file to a directory like `[module]/target/unit-test.log`. But this ends up creating `[module]/[module]/target/unit-test.log` instead of former.
    
    The `work/` directory is not deleted by "mvn clean", in the parent and in modules. Neither is the `checkpoint/` directory created under the various external modules.
    
    Many tests create a temp directory, which is not usually deleted. This can be largely resolved by calling `deleteOnExit()` at creation and trying to call `Utils.deleteRecursively` consistently to clean up, sometimes in an `@After` method.
    
    _If anyone seconds the motion, I can create a more significant change that introduces a new test trait along the lines of `LocalSparkContext`, which provides management of temp directories for subclasses to take advantage of._
    
    Author: Sean Owen <sowen@cloudera.com>
    
    Closes #732 from srowen/SPARK-1798 and squashes the following commits:
    
    5af578e [Sean Owen] Try to consistently delete test temp dirs and files, and set deleteOnExit() for each
    b21b356 [Sean Owen] Remove work/ and checkpoint/ dirs with mvn clean
    bdd0f41 [Sean Owen] Remove duplicate module dir in log4j.properties output path for tests
Loading