Skip to content
Snippets Groups Projects
  • zsxwing's avatar
    f9ed2b66
    [SPARK-4608][Streaming] Reorganize StreamingContext implicit to improve API convenience · f9ed2b66
    zsxwing authored
    There is only one implicit function `toPairDStreamFunctions` in `StreamingContext`. This PR did similar reorganization like [SPARK-4397](https://issues.apache.org/jira/browse/SPARK-4397).
    
    Compiled the following codes with Spark Streaming 1.1.0 and ran it with this PR. Everything is fine.
    ```Scala
    import org.apache.spark._
    import org.apache.spark.streaming._
    import org.apache.spark.streaming.StreamingContext._
    
    object StreamingApp {
    
      def main(args: Array[String]) {
        val conf = new SparkConf().setMaster("local[2]").setAppName("FileWordCount")
        val ssc = new StreamingContext(conf, Seconds(10))
        val lines = ssc.textFileStream("/some/path")
        val words = lines.flatMap(_.split(" "))
        val pairs = words.map(word => (word, 1))
        val wordCounts = pairs.reduceByKey(_ + _)
        wordCounts.print()
    
        ssc.start()
        ssc.awaitTermination()
      }
    }
    ```
    
    Author: zsxwing <zsxwing@gmail.com>
    
    Closes #3464 from zsxwing/SPARK-4608 and squashes the following commits:
    
    aa6d44a [zsxwing] Fix a copy-paste error
    f74c190 [zsxwing] Merge branch 'master' into SPARK-4608
    e6f9cc9 [zsxwing] Update the docs
    27833bb [zsxwing] Remove `import StreamingContext._`
    c15162c [zsxwing] Reorganize StreamingContext implicit to improve API convenience
    f9ed2b66
    History
    [SPARK-4608][Streaming] Reorganize StreamingContext implicit to improve API convenience
    zsxwing authored
    There is only one implicit function `toPairDStreamFunctions` in `StreamingContext`. This PR did similar reorganization like [SPARK-4397](https://issues.apache.org/jira/browse/SPARK-4397).
    
    Compiled the following codes with Spark Streaming 1.1.0 and ran it with this PR. Everything is fine.
    ```Scala
    import org.apache.spark._
    import org.apache.spark.streaming._
    import org.apache.spark.streaming.StreamingContext._
    
    object StreamingApp {
    
      def main(args: Array[String]) {
        val conf = new SparkConf().setMaster("local[2]").setAppName("FileWordCount")
        val ssc = new StreamingContext(conf, Seconds(10))
        val lines = ssc.textFileStream("/some/path")
        val words = lines.flatMap(_.split(" "))
        val pairs = words.map(word => (word, 1))
        val wordCounts = pairs.reduceByKey(_ + _)
        wordCounts.print()
    
        ssc.start()
        ssc.awaitTermination()
      }
    }
    ```
    
    Author: zsxwing <zsxwing@gmail.com>
    
    Closes #3464 from zsxwing/SPARK-4608 and squashes the following commits:
    
    aa6d44a [zsxwing] Fix a copy-paste error
    f74c190 [zsxwing] Merge branch 'master' into SPARK-4608
    e6f9cc9 [zsxwing] Update the docs
    27833bb [zsxwing] Remove `import StreamingContext._`
    c15162c [zsxwing] Reorganize StreamingContext implicit to improve API convenience