Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    378b2315
    [SPARK-3061] Fix Maven build under Windows · 378b2315
    Josh Rosen authored
    The Maven build was failing on Windows because it tried to call the unix `unzip` utility to extract the Py4J files into core's build directory.  I've fixed this issue by using the `maven-antrun-plugin` to perform the unzipping.
    
    I also fixed an issue that prevented tests from running under Windows:
    
    In the Maven ScalaTest plugin, the filename listed in <filereports> is placed under the <reportsDirectory>; the current code places it in a subdirectory of reportsDirectory, e.g.
    
    ```
    ${project.build.directory}/surefire-reports/${project.build.directory}/SparkTestSuite.txt
    ```
    
    This caused problems under Windows because it would try to create a subdirectory named "c:\\".
    
    Note that the tests still fail under Windows (for other reasons); this PR just allows them to run and fail rather than crash when trying to create the test reports directory.
    
    Author: Josh Rosen <joshrosen@apache.org>
    Author: Josh Rosen <rosenville@gmail.com>
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #2165 from JoshRosen/windows-support and squashes the following commits:
    
    651d210 [Josh Rosen] Unzip to python/build instead of core/build
    fbf3e61 [Josh Rosen] 4 spaces -> 2 spaces
    e347668 [Josh Rosen] Fix Maven scalatest filereports path:
    4994af1 [Josh Rosen] [SPARK-3061] Use maven-antrun-plugin to unzip Py4J.
    378b2315
    History
    [SPARK-3061] Fix Maven build under Windows
    Josh Rosen authored
    The Maven build was failing on Windows because it tried to call the unix `unzip` utility to extract the Py4J files into core's build directory.  I've fixed this issue by using the `maven-antrun-plugin` to perform the unzipping.
    
    I also fixed an issue that prevented tests from running under Windows:
    
    In the Maven ScalaTest plugin, the filename listed in <filereports> is placed under the <reportsDirectory>; the current code places it in a subdirectory of reportsDirectory, e.g.
    
    ```
    ${project.build.directory}/surefire-reports/${project.build.directory}/SparkTestSuite.txt
    ```
    
    This caused problems under Windows because it would try to create a subdirectory named "c:\\".
    
    Note that the tests still fail under Windows (for other reasons); this PR just allows them to run and fail rather than crash when trying to create the test reports directory.
    
    Author: Josh Rosen <joshrosen@apache.org>
    Author: Josh Rosen <rosenville@gmail.com>
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #2165 from JoshRosen/windows-support and squashes the following commits:
    
    651d210 [Josh Rosen] Unzip to python/build instead of core/build
    fbf3e61 [Josh Rosen] 4 spaces -> 2 spaces
    e347668 [Josh Rosen] Fix Maven scalatest filereports path:
    4994af1 [Josh Rosen] [SPARK-3061] Use maven-antrun-plugin to unzip Py4J.