Skip to content
  • hyukjinkwon's avatar
    05f83c53
    [SPARK-21128][R] Remove both "spark-warehouse" and "metastore_db" before listing files in R tests · 05f83c53
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR proposes to list the files in test _after_ removing both "spark-warehouse" and "metastore_db" so that the next run of R tests pass fine. This is sometimes a bit annoying.
    
    ## How was this patch tested?
    
    Manually running multiple times R tests via `./R/run-tests.sh`.
    
    **Before**
    
    Second run:
    
    ```
    SparkSQL functions: Spark package found in SPARK_HOME: .../spark
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ....................................................................................................1234.......................
    
    Failed -------------------------------------------------------------------------
    1. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
    length(list1) not equal to length(list2).
    1/1 mismatches
    [1] 25 - 23 == 2
    
    2. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
    sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
    10/25 mismatches
    x[16]: "metastore_db"
    y[16]: "pkg"
    
    x[17]: "pkg"
    y[17]: "R"
    
    x[18]: "R"
    y[18]: "README.md"
    
    x[19]: "README.md"
    y[19]: "run-tests.sh"
    
    x[20]: "run-tests.sh"
    y[20]: "SparkR_2.2.0.tar.gz"
    
    x[21]: "metastore_db"
    y[21]: "pkg"
    
    x[22]: "pkg"
    y[22]: "R"
    
    x[23]: "R"
    y[23]: "README.md"
    
    x[24]: "README.md"
    y[24]: "run-tests.sh"
    
    x[25]: "run-tests.sh"
    y[25]: "SparkR_2.2.0.tar.gz"
    
    3. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
    length(list1) not equal to length(list2).
    1/1 mismatches
    [1] 25 - 23 == 2
    
    4. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
    sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
    10/25 mismatches
    x[16]: "metastore_db"
    y[16]: "pkg"
    
    x[17]: "pkg"
    y[17]: "R"
    
    x[18]: "R"
    y[18]: "README.md"
    
    x[19]: "README.md"
    y[19]: "run-tests.sh"
    
    x[20]: "run-tests.sh"
    y[20]: "SparkR_2.2.0.tar.gz"
    
    x[21]: "metastore_db"
    y[21]: "pkg"
    
    x[22]: "pkg"
    y[22]: "R"
    
    x[23]: "R"
    y[23]: "README.md"
    
    x[24]: "README.md"
    y[24]: "run-tests.sh"
    
    x[25]: "run-tests.sh"
    y[25]: "SparkR_2.2.0.tar.gz"
    
    DONE ===========================================================================
    ```
    
    **After**
    
    Second run:
    
    ```
    SparkSQL functions: Spark package found in SPARK_HOME: .../spark
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................
    ```
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #18335 from HyukjinKwon/SPARK-21128.
    05f83c53
    [SPARK-21128][R] Remove both "spark-warehouse" and "metastore_db" before listing files in R tests
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR proposes to list the files in test _after_ removing both "spark-warehouse" and "metastore_db" so that the next run of R tests pass fine. This is sometimes a bit annoying.
    
    ## How was this patch tested?
    
    Manually running multiple times R tests via `./R/run-tests.sh`.
    
    **Before**
    
    Second run:
    
    ```
    SparkSQL functions: Spark package found in SPARK_HOME: .../spark
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ....................................................................................................1234.......................
    
    Failed -------------------------------------------------------------------------
    1. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
    length(list1) not equal to length(list2).
    1/1 mismatches
    [1] 25 - 23 == 2
    
    2. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3384)
    sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
    10/25 mismatches
    x[16]: "metastore_db"
    y[16]: "pkg"
    
    x[17]: "pkg"
    y[17]: "R"
    
    x[18]: "R"
    y[18]: "README.md"
    
    x[19]: "README.md"
    y[19]: "run-tests.sh"
    
    x[20]: "run-tests.sh"
    y[20]: "SparkR_2.2.0.tar.gz"
    
    x[21]: "metastore_db"
    y[21]: "pkg"
    
    x[22]: "pkg"
    y[22]: "R"
    
    x[23]: "R"
    y[23]: "README.md"
    
    x[24]: "README.md"
    y[24]: "run-tests.sh"
    
    x[25]: "run-tests.sh"
    y[25]: "SparkR_2.2.0.tar.gz"
    
    3. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
    length(list1) not equal to length(list2).
    1/1 mismatches
    [1] 25 - 23 == 2
    
    4. Failure: No extra files are created in SPARK_HOME by starting session and making calls (test_sparkSQL.R#3388)
    sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
    10/25 mismatches
    x[16]: "metastore_db"
    y[16]: "pkg"
    
    x[17]: "pkg"
    y[17]: "R"
    
    x[18]: "R"
    y[18]: "README.md"
    
    x[19]: "README.md"
    y[19]: "run-tests.sh"
    
    x[20]: "run-tests.sh"
    y[20]: "SparkR_2.2.0.tar.gz"
    
    x[21]: "metastore_db"
    y[21]: "pkg"
    
    x[22]: "pkg"
    y[22]: "R"
    
    x[23]: "R"
    y[23]: "README.md"
    
    x[24]: "README.md"
    y[24]: "run-tests.sh"
    
    x[25]: "run-tests.sh"
    y[25]: "SparkR_2.2.0.tar.gz"
    
    DONE ===========================================================================
    ```
    
    **After**
    
    Second run:
    
    ```
    SparkSQL functions: Spark package found in SPARK_HOME: .../spark
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................................................
    ...............................................................................................................................
    ```
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #18335 from HyukjinKwon/SPARK-21128.
Loading