-
- Downloads
Merge pull request #277 from tdas/scheduler-update
Refactored the streaming scheduler and added StreamingListener interface - Refactored the streaming scheduler for cleaner code. Specifically, the JobManager was renamed to JobScheduler, as it does the actual scheduling of Spark jobs to the SparkContext. The earlier Scheduler was renamed to JobGenerator, as it actually generates the jobs from the DStreams. The JobScheduler starts the JobGenerator. Also, moved all the scheduler related code from spark.streaming to spark.streaming.scheduler package. - Implemented the StreamingListener interface, similar to SparkListener. The streaming version of StatusReportListener prints the batch processing time statistics (for now). Added StreamingListernerSuite to test it. - Refactored streaming TestSuiteBase for deduping code in the other streaming testsuites.
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 2 additions, 5 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 0 additions, 1 deletion...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/DStream.scala 3 additions, 8 deletions...g/src/main/scala/org/apache/spark/streaming/DStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala 1 addition, 0 deletions.../main/scala/org/apache/spark/streaming/DStreamGraph.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 14 additions, 12 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala 8 additions, 0 deletions...pache/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/ForEachDStream.scala 2 additions, 1 deletion...a/org/apache/spark/streaming/dstream/ForEachDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/NetworkInputDStream.scala 1 addition, 0 deletions.../apache/spark/streaming/dstream/NetworkInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala 55 additions, 0 deletions...cala/org/apache/spark/streaming/scheduler/BatchInfo.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/Job.scala 12 additions, 12 deletions...main/scala/org/apache/spark/streaming/scheduler/Job.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala 26 additions, 26 deletions...a/org/apache/spark/streaming/scheduler/JobGenerator.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala 108 additions, 0 deletions...a/org/apache/spark/streaming/scheduler/JobScheduler.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobSet.scala 68 additions, 0 deletions...n/scala/org/apache/spark/streaming/scheduler/JobSet.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/NetworkInputTracker.scala 2 additions, 1 deletion...pache/spark/streaming/scheduler/NetworkInputTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListener.scala 75 additions, 0 deletions.../apache/spark/streaming/scheduler/StreamingListener.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListenerBus.scala 81 additions, 0 deletions...ache/spark/streaming/scheduler/StreamingListenerBus.scala
- streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala 0 additions, 12 deletions...ala/org/apache/spark/streaming/BasicOperationsSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala 10 additions, 16 deletions...st/scala/org/apache/spark/streaming/CheckpointSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/FailureSuite.scala 9 additions, 4 deletions.../test/scala/org/apache/spark/streaming/FailureSuite.scala
Loading
Please register or sign in to comment