Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    cf6cbe9f
    [SPARK-1824] Remove <master> from Python examples · cf6cbe9f
    Andrew Or authored
    A recent PR (#552) fixed this for all Scala / Java examples. We need to do it for python too.
    
    Note that this blocks on #799, which makes `bin/pyspark` go through Spark submit. With only the changes in this PR, the only way to run these examples is through Spark submit. Once #799 goes in, you can use `bin/pyspark` to run them too. For example,
    
    ```
    bin/pyspark examples/src/main/python/pi.py 100 --master local-cluster[4,1,512]
    ```
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #802 from andrewor14/python-examples and squashes the following commits:
    
    cf50b9f [Andrew Or] De-indent python comments (minor)
    50f80b1 [Andrew Or] Remove pyFiles from SparkContext construction
    c362f69 [Andrew Or] Update docs to use spark-submit for python applications
    7072c6a [Andrew Or] Merge branch 'master' of github.com:apache/spark into python-examples
    427a5f0 [Andrew Or] Update docs
    d32072c [Andrew Or] Remove <master> from examples + update usages
    cf6cbe9f
    History
    [SPARK-1824] Remove <master> from Python examples
    Andrew Or authored
    A recent PR (#552) fixed this for all Scala / Java examples. We need to do it for python too.
    
    Note that this blocks on #799, which makes `bin/pyspark` go through Spark submit. With only the changes in this PR, the only way to run these examples is through Spark submit. Once #799 goes in, you can use `bin/pyspark` to run them too. For example,
    
    ```
    bin/pyspark examples/src/main/python/pi.py 100 --master local-cluster[4,1,512]
    ```
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #802 from andrewor14/python-examples and squashes the following commits:
    
    cf50b9f [Andrew Or] De-indent python comments (minor)
    50f80b1 [Andrew Or] Remove pyFiles from SparkContext construction
    c362f69 [Andrew Or] Update docs to use spark-submit for python applications
    7072c6a [Andrew Or] Merge branch 'master' of github.com:apache/spark into python-examples
    427a5f0 [Andrew Or] Update docs
    d32072c [Andrew Or] Remove <master> from examples + update usages