Skip to content
Snippets Groups Projects
  • Sean Owen's avatar
    11d54941
    SPARK-1663. Corrections for several compile errors in streaming code examples,... · 11d54941
    Sean Owen authored
    SPARK-1663. Corrections for several compile errors in streaming code examples, and updates to follow API changes
    
    I gave the Streaming code examples, both Scala and Java, a test run today. I turned up a number of small errors, mostly compile errors in the Java examples. There were a few typos in the Scala too.
    
    I also took the liberty of adding things like imports, since in several cases they are not obvious. Feel free to push back on some changes.
    
    There's one thing I haven't quite addressed in the changes. `JavaPairDStream` uses the Java API version of `Function2` in almost all cases, as `JFunction2`. However it uses `scala.Function2` in:
    
    ```
      def reduceByKeyAndWindow(reduceFunc: Function2[V, V, V], windowDuration: Duration)
      :JavaPairDStream[K, V] = {
        dstream.reduceByKeyAndWindow(reduceFunc, windowDuration)
      }
    ```
    
    Is that a typo?
    
    Also, in Scala, I could not get this to compile:
    ```
    val windowedWordCounts = pairs.reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))
    error: missing parameter type for expanded function ((x$1, x$2) => x$1.$plus(x$2))
    ```
    
    You can see my fix below but am I missing something?
    
    Otherwise I can say these all worked for me!
    
    Author: Sean Owen <sowen@cloudera.com>
    
    Closes #589 from srowen/SPARK-1663 and squashes the following commits:
    
    65a906b [Sean Owen] Corrections for several compile errors in streaming code examples, and updates to follow API changes
    11d54941
    History
    SPARK-1663. Corrections for several compile errors in streaming code examples,...
    Sean Owen authored
    SPARK-1663. Corrections for several compile errors in streaming code examples, and updates to follow API changes
    
    I gave the Streaming code examples, both Scala and Java, a test run today. I turned up a number of small errors, mostly compile errors in the Java examples. There were a few typos in the Scala too.
    
    I also took the liberty of adding things like imports, since in several cases they are not obvious. Feel free to push back on some changes.
    
    There's one thing I haven't quite addressed in the changes. `JavaPairDStream` uses the Java API version of `Function2` in almost all cases, as `JFunction2`. However it uses `scala.Function2` in:
    
    ```
      def reduceByKeyAndWindow(reduceFunc: Function2[V, V, V], windowDuration: Duration)
      :JavaPairDStream[K, V] = {
        dstream.reduceByKeyAndWindow(reduceFunc, windowDuration)
      }
    ```
    
    Is that a typo?
    
    Also, in Scala, I could not get this to compile:
    ```
    val windowedWordCounts = pairs.reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))
    error: missing parameter type for expanded function ((x$1, x$2) => x$1.$plus(x$2))
    ```
    
    You can see my fix below but am I missing something?
    
    Otherwise I can say these all worked for me!
    
    Author: Sean Owen <sowen@cloudera.com>
    
    Closes #589 from srowen/SPARK-1663 and squashes the following commits:
    
    65a906b [Sean Owen] Corrections for several compile errors in streaming code examples, and updates to follow API changes